Hi colinistheman,
it's very good that people have concerns about the security of code, or the process used to assure it. I hope your concerns have been addressed by now.
Your post made me realize one thing though: you probably haven't seen gmaxwell's reddit post (
http://www.reddit.com/r/Bitcoin/comments/2rrxq7/on_why_010s_release_notes_say_we_have_reason_to/). This explains the reason for the at the time somewhat cryptic "we have reason to believe it is better tested". I encourage you to read the details there, but in short: we found a very tricky (but most likely harmless) bug in OpenSSL itself while writing this library - because the tests did comparisons with OpenSSL and failed once. It's by no means a proof that libsecp256k1 is bug free (more review is always welcome), but it does show that its testing practices pay off.
We should probably change the language in the release notes, now that the OpenSSL bug it was referring to has been disclosed.
I've been looking at the code, and theres quite a few
magic numbers in there

Most of the constants are taken directly from the secp256k1 standard, or computed using algorithms explained in code. But more comments to explain where they come from would not be a bad idea. We'll add some.