Still, I strongly advice you to add another layer (compression whatever) instead of using a binary format.
I would agree that using, say, zlib style compression would be a better idea as it would allow for more TPS than the current JSON but would not be as tricky (i.e. stuff like "endian") as binary.
I think 1000+ TPS is the eventual goal - but hey Bitcoin doesn't even do its currently max. 7 TPS at the moment (so no great hurry to get there). Even 100 TPS would put Nxt well ahead of the pack.
Actually using ZLIB might not be the best idea, better would be something that's
faster.
I'd go with
LZ4. There's also lzturbo, but, lz4 has format that might be nicer for "small devices" (raspis & Co.)
And ChuckOne is right, binary protocol equals problems