I don't even know how much is in my cold storage as I haven't accessed it in a number of years.
Smart. Attackers can't know how much you are holding if you don't know how much you are holding. Perhaps this is the best way to protect against $5 wrench attacks.

But if my leather wallet is empty, it's not like I'm going to get shot because they think I have $1000 stored somewhere else on my person. The mugger would run away.
Sure, but physical attacks with the aim of stealing bitcoin aren't happening by approaching random people on the street in the spur of the moment. The attack is targeted and planned, and the attacker usually knows the victim is holding a significant amount of bitcoin. Often they are home invasions, or fake meetups under the pretense of trading. If you initially refuse to hand over any bitcoin or wallet details, the attacker is unlikely to just say "Oh well" and leave.