Horrible fuckin' thing, however just few bananas is not a tragedy. It also depends if they produce pollen or not, cause if they're sterile you can pluck them and go about your day. But if they do, then you're in for a pollinated crop!

Also, NEVER EVER grow hermie seeds, even if they came from a nice-yielding plant, cause stress trait is passed onto the next generation, which means you have 50% chance on average to grow a bunch of hermies and sometimes fully intersex individuals!