You’re completely in the right and I am so happy you are doing this for yourself. If he’s not respecting set boundaries, that’s an issue and you either need to address or end things. It might be something that some people don’t think is a big deal, but honestly when getting to know someone, the small stuff just shows what it could be like in the long run.