If security is your goal, I think this hassle you mentioned is an acceptable trade-off. I've seen users lose money because they only use a single hardware wallet/air-gapped device, and when a keylogger/malware attacks them, their money got drained easily. This should help you avoid that at least.

What's the purpose of this multi-sig wallet? Did you use it to store your trading money, so you need to make transfers regularly, or did you use it for something else? Knowing the purpose might also help others to give you suggestions for a better setup in terms of flexibility and security.