It's not a good idea to use a static IP on my opinion.

The main reason to use load balancing is to distribute the load and failover to the remaining live servers in case of problems, which you will loose in this case.
Lowest latency does not mean 'best' ... what if the server is close but overloaded?

If you insist on fixed IP - it's recommended to at least add the hostname for failover