It doesn't necessarily mean you're late though. The number of people who've mined/bought BTC when it was worth almost nothing and held up to this day is extremely small. Nobody knew it'd ever reach this price point.
It's almost like 10 years from now we'd have a Bitcoin worth a billion bucks. Before it gets there, almost everyone who's purchased today would have sold at $100k, $1M or $10M. Those who'd still hold either have a shitload of money they don't really need to sell Bitcoin in the future anyway, were lucky enough to forget the password to their wallets for many years and became rich or so.
On the other hand, the opposite could happen. You never know how, all of a sudden, people start not giving a crap about Bitcoin anymore and the markets'd crash like never before.
The main idea is, you have to assume the risks and, if you can do it, then join it. It's a fun ride anyway.

Exactly. Most people who saved up their bitcoins from day one are the ones that are really late because a lot of them jumped ship years ago when bitcoins won't even buy you a can of beer. You don't have to worry about being late because bitcoin will provide. And with the current publixity bitciin is facing, we can only go upwards from here, meaning there's no way bitcoin will go back from what it was years ago, even if you argue it's high volatility rate.