Looking for Constructive Feedback

Sorry if this gets long…I’m genuinely looking for feedback on my views and the way I’m handling visitation of my daughter.
We do not have a court ordered custody agreement for our 1 year old daughter. It’s clear that we need one in the near future but for now our agreement is that he picks her up from daycare every other Friday and returns her to daycare on Monday morning. Most every time he has an upcoming weekend with her I receive a text or phone call asking to rearrange the schedule. Apparently he’s been traveling for work quite a bit so it isn’t uncommon for him to contact me to say he’ll be out of town for the next month. Up to this point I’ve voiced my opinion but have always accommodated his schedule change requests in an effort to try and get him to like me. (Ha! Ha!)
My real feeling is that his weekends with his daughter are his responsibility. Out of every 2 weeks he is responsible for 2 days. I cover 12, he covers 2. If he is required to work or travel then he needs to arrange for child care (he lives with his mother so it’s her he has to arrange it with) just as I would. I don’t get to pick and choose when it’s convenient to be a parent and I don’t believe he should either. I’m not saying he can’t ask me if I’m available or willing to help him out, I’m saying he cannot just “dump” it on me, think it’s ok, and become abusive if I’m unable to accommodate him.
I made a decision and a promise to myself that being abusive will no longer serve him in dealing with me. Punctuating his sentences with “bitch”, calling me a horrible mother and accusing me of using our daughter to get to him will no longer win my cooperation in visitation schedule changes. I’ve taught him his behavior works by allowing it to manipulate me and so I’m going to have to teach him that I will only respond to civil and respectful communication regarding our child.
When I received a text at 12:45am last night saying he’s working Friday night and will pick her up at my house Saturday after work -my response was that he can just take her next weekend instead. It doesn’t appear his work schedule permits this weekend with his daughter and I’m not available on Saturday to wait around all day for him to pick her up. (He’s always at least an hour early or late whenever he’s picking her up from my home). He responded with the usual name calling and telling me how much I’m going to regret keeping his daughter from him, he even followed it up with a pretty vial statement about how if we were f*/%&ing it wouldn’t be a problem but he’s so sexually repulsed by me he’d rather f$#% 100 women before he’d ever put his d*&% in me again…..(Ahhhh such a class act!)
My question is whether I’m wrong in my feelings about his weekends being his responsibility and requests for schedule changes needing to be made in a respectful manner or they will not be accommodated.