I don't know that it really has anything to do with "patriarchy". There would likely be social rules for style and grooming that people were judged for not adhering to even if men didn't exist. It's easy to blame "the patriarchy" but the truth is that people are assholes and were taught to be as such by those around them.