More than 50% of BTC is in the hands of individuals. WHales can manipulate the market, but the influence is not too huge.
It's bigger than you think because of leverage. People who use leverage hurt themselves, but the urge to gamble is often too strong.
When a whale sells a large amount of bitcoin, levered traders are the first to go down. Bitcoin would be much less volatile if leveraged trading was allowed only to a certain amount like 5x, but that's wishful thinking.