In the end it always boils down to knowing the private key for each address you have funds in. The address and this private key is the only information which matters. Everyone able to access those is able to spend the funds.

Please recall, your coins aren't stored "in" the wallet. They are in the blockchain!