Obviously, I don't think it is worth buying another house just for that, but if you already have them, if you have two houses, why not take advantage of that to store the HW with their corresponding seeds separately?
The second biggest consideration when storing a back up off site (second obviously to the security of that location), is how you would be notified of compromise, and how long that would take. If I buried a back up somewhere under cover of darkness (for example), then the only way I'm going to know if someone has dug it up is if I visit that site to check. That in itself might lead someone to my back up, or I might go weeks or months without realizing that someone has accessed by seed phrase and is currently searching for or trying to brute force my passphrase.
What is the set up of your vacation home? Does anyone else use it? Do you rent it out? Does anyone check on it when you are not there? Does it have a security system you can log in to remotely to check? If someone breaks in, will you be notified immediately, and can take immediate steps to move any coins to a new wallet?