Also, he might not have been very good at what he did (by some accounts at least). He effed things up with France at one point, confused his constituents with apparently conflicting principles guiding his position on various political or social topics.
Evidently, he was hesitant to sign the bill that became his legacy, he refused to sign the constitution because it did not enumerate individual rights (hadn't yet included a bill of rights), and took a hard stand against slavery. Maybe he was just an opportunist, or he had the good fortune of family connections or whatnot?
In any case I'm sure there are other things he could have been remembered for were it not for a stunning and memorable critique in a newspaper with a clever take on current events.