I ran into such a situation once and I wrote down the seed phase of my trust wallet in a diary. But when I set off on a long journey with that diary, I fell into the hands of robbers. A group of robbers stole my diary along with my luggage, mobile phone and wallet. Despite my best efforts, I could not keep the diary safe from the bandits. So that was the first experience in my life and I have never forgotten that experience till now.
You made two mistakes at the same time.
First, you did only one back up in your diary. It is bad because back up must be done in some forms, you can do it in a diary but must have other back ups. When you lose your diary, you can have other back ups to use.
Second, you brought your back up around with you and ignored risk that you will lose it. Back ups must be at your home, safely and secretly. Of course you can store your back ups at different locations, not only at home, or only one location at home. However, put your backup in your luggage and bring it around is very risky, not recommended.