I've been looking at the code, and theres quite a few
magic numbers in there

Can you clarify this? If you mean "32" that is the byte size of the field elements used for all of the arithmetic. If you mean the 32-bit hex numbers, those are parameters of the curve as defined by NIST. If you mean "0", "-1", "-2" error return values, I'd guess that a PR to #define better names for these would be welcome.
Edit: I've been reminded that the secp256k1 curve is actually defined by SECG, not NIST. My bad.