The last part is funny! :)

I don't think you're off base. It's strange that he would feel free to use your office without asking, especially if you weren't close or he wasn't your subordinate.
It's a space that's designated to you by the company for the betterment of the company. You have a job to perform and that's one of your tools for doing so. It's your privilege and your responsibility, so when he uses it without permission, that can reflect poorly on your responsibility (if he does something bad).
Good for you for talking to him about it. It's too bad that he couldn't take responsibility, apologize, and use the learning opportunity to build a relationship with you. Never hurts to have higher level contacts.