So it would seem that the CPU needs enough virtual memory to match all the GPU memory in use. I can see a preallocation
strategy being better for speed but if the CPU is swapping to disk it would be slower than dynamic memory allocation.
this memory isn't used anymore once it has been allocated to the vram
While this is technically correct I can tell from experience some drivers will still keep the address range as reserved, apparently this has some benefits for driver mangling (I can see how assuming different range <--> different resource can help).
Beware, CUDA is way more than your GPU or CPU, sometimes it goes through some heavy magic.
How is the memory consumption being measured?