can i build it against cuda9 and copy the binary to rigs with only cuda8 installed? does it run slower than cuda9 rig?
All you need is recent Nvidia driver with CUDA9 support, you won't need any toolkits installed.
Great, thanks. Does the CPU of build machine matter? I have G1840, g3900 ,g4560, e3 1231, which one is better to build the binary and copy to other rig with different CPU?
I use ubuntu 16, nvOC.
For *nix it will be a problem (running without reboot), if you already have recent drivers installed, you can ask someone to build binary for you and just update, but if you need to update driver or want to install toolkit and build from scratch - as I know you'll need to reboot at least 1 time. If access is only remote it's quite risky, especially regarding driver module update.
CPU matters, so if you want to build it for all variety of hardware you need to remove "-march=native" flag from configure.sh file.
P.S: I'm not so experienced in *nix, it's just what I remember
I checked the document of gcc, removing arch=native can lead performance losses, so I think build against the weakest g1840 is fine.