I think the main reason is the lack of funds. After all, if you have enough money, you will not be afraid of losing it because of the fear of a future price drop. You just try it!

There are also many people who don't hold bitcoin, simply because they don't have enough knowledge. They just don't understand what bitcoin is and why it is needed.