Thank you for the comprehensive reply, @Quickseller. I do like your idea of using a Electrum server. However, I believe it needs lots of RAM - not usually available in a VPS plan.
Keeping the VPS idea plus your insights, maybe I can create unsigned transactions on the VPS from watch only addresses and sign them locally. It would be nice to automate that flow.