If you Know your family well then i don't think this will ever takes place not unless you grown from different places so you know nothing about the family members then they might have that Idea of taking your funds out.
but as long as the people you are sharing are someone that you know well? I am sure they will never take your funds(but still need to stay silence about your holding) because like me, Only Wife and my parents have idea how much I am Holding now because they are the only person i trusted the most.
When it comes to money is really hard to trust someone even if the person is very close to you, like you said, is your wife and family who knows how much you're holding. I believe you can as well say out your seed phrase to them right?
All I'm saying is money has a way of changing people, in some cases if they know how much you're holding they might try to know your phrase and even try to get hold of your funds, things might get messy