Not true at all....
A lot of rich people started small and they worked hard to grow a business and then expand that business into a franchise and they eventually end up with a chain of shops and empire to be proud of...
I started with Bitcoin sucking on faucets, because I did not trust the technology and I doubted it's future... then I researched the technology and understood the risk and gradually bought small amounts of bitcoin.. until I accumulated a few coins and those coins became valuable over time.

It all matters with perseverance and effort because you couldn't really achieve something if you dont really set up goals or targets on your own and if you do intently to make your life a bit
ahead or have that financial freedom then it is just right that you would really be putting up some goals but ofcourse it would really be needing that extreme effort.
It doesnt really need to start up big and same as you said that you could build up yourself gradually until you do able to reach the state where
everything becomes big and scattered.