No, that can't lead to loss because I have safely saved my private keys. Even if I lose my smartphone, I have Electrum wallet hidden, I have a very strong password on it and I believe I'm able to recover my wallet quickly once I get home. I even have plan B.
-snip-
I don't know if I am lucky or not but I have never had any problem.
It is always better to be safe than sorry. I'm not sure what your plan B is, but I feel like being very sure nothing bad can happen because you are never visiting a strange website is a terrible idea. I've been hearing so many stories related to how people lost money even though they use a HW, claiming they never store their seed phrase online, and so on. Heck, who knows maybe you're drunk and accidentally click on a phishing website one day. It is always better to assume the worst thing can happen to you. I believe you're lucky to some extent, which is extremely helpful if you plan on improving your security practices.