You were completely fine sometimes people say that as a joke reply and it’s generally not a rude thing but I’m sorry he was rude about it. You did the right thing it would’ve been rude if you didn’t ask and it turned out someone was sitting there. Honestly sometime you just have to ignore people being rude for the sake of enjoying yourself