They are not wrong, but you are taking it way to litteraly. Purpose of the VPN is to hide your location. So if someone sees you're doing big transactions, they won't know your physical location and pay you a visit with a hammer and tweezers. You know, the old $5 wrench attack
OP specifically mentioned hackers, so that should be online attackers and not physical attackers like a $5 wrench attack. That being said, it is very, very difficult, and almost impossible for just anyone to get your exact
locationaddress with only your IP address, so even if you are not using a VPN, you might never suffer a physical attack, but if you are using a VPN and your wallet is online, chances are higher that you might suffer an online attack.