Presumably GPU VPS's are the same to use as a CPU VPS e.g. the same Ubuntu OS. It's just that the hardware specs "behind the scenes" are different. Yes?
That heavily depends on VPS provider you use, few might provide OS with installed tools (GPU Driver, CUDA Toolkit, ML library, etc.) so users can use the VPS immediately.
For example,
GPU Rental mention they specifically create image for deep learning, which means GPU driver and it's toolkit/library should be installed already.