what's stopping big investors from owning a small fortune of one crypto and have a large influence in its performance

Lack of regulation is what is stopping big investors, too much money is simply tied up in markets which is making it impossible for it to grow into cryptocurrencies. Those with billions of dollars have their hands tied and unable to enter crypto markets without the regulations.