Never had memory issues. Didnt do anything fancy.

That said im using raspbian server edition, and allocated the lowest possible memory for GPU... so maybe without the added bloat of GUI it just doesnt encounter any limits.

Also, you can load up raspbian in qemu, but thats even slower than building on the device...