Having more applications installed on my device means the more chance of getting hacked. Though I'm New into the crypto space, I don't know but I've read somewhere that without someone having access to my secret code that is my passphrase he wouldn't be able to hack or get access to my wallet.
My question here is "Do you mean my wallet can be hacked even without the person getting access to my Passphrase"?
You're right, and yes there is a tendency that you'll still get hacked if you innocently connect your wallet to some services that you'd be required to just press your consent. That's why don't just connect your wallets if a website asks you to do so if you are not aware of the website and the service that it has. Because there are websites that upon visiting, you'll be asked to do that automatically so you just have to decline it and leave that website or simply ignore and leave it.