Is there a big difference in performance between pre-compiled binaries and self-compiled?

I'm thinking about only compiling libgmp from source and using the precompiled binaries for primecoind.

Thanks a lot for your guide btw!