"to chat with whomever happened to stop by"
would be okay if it were "to chat (with whomever) happened to stop by", but it's not. It's "to chat with (whomever happened to stop by)", so it should be "whoever", the subject of the clause "... happened to stop by."