I keep hearing about how if you're not using Tor or or VPN you can be identified as a bitcoin user by your IP address. How is that a thing?

If I have bitcoin on a wallet, what about my IP could identify me? Does it matter if you use Tor to look at transactions but just didn't have all internet traffic through Tor when you activate your wallet on your phone or computer?