You just have to suck it up and take the time to put up a boundary and enforce it. They will balk and do whatever they can to get you to fold. You have to do it.
Guess what? I will be in the trenches with you. There is an adult in my life right now, at work, whom I see every day. I refer to him in my mind as my 8th special needs kiddo. When I understand he is this way it is a bit easier to set the boundary.
Will you set your boundary? Remember I am there with you each moment.