I myself wouldn't trust referer headers as they could be fabricated.
While I didn't trust them too for FaucetBOX.com, how could they be fabricated in context of CSRF? If I were to attack you using CSRF I wouldn't be able to force your browser to fake the referrer.
You are right. Referer check seems to be good enough to protect against CSRF. However there's ways to get in control of someone's browser and then spoof the headers(is XSS + CSRF possible?). Also what about HTTPS or if someone's browser doesn't send the referer headers (guess 99% do but still)? That would be considered as an attack.
BTW Do you guys support p2sh for litecoin yet?