It is advisable to always have your savings at your place, not trusting the banks or anyone else. Because in which case, you have the opportunity to lose all your money, and literally stay with nothing. But in what exactly to store, everyone already decides for himself.
You can do that, but it is safe?
Not all of the people think like you since most of them when it comes to money, they overthink. Fiat is not like Bitcoin and your Bitcoin wallet since you can secure that with a lot of ways and people won't know, they won't know that you have a Bitcoin either. But in terms of fiat, it can show either with your job or your state in life and if you will just save that in your house and not in a bank account, you are risking that to be robbed, while in banks, it is not just sitting there it is also giving you some interest.