Well try several ones:
- Koblitz: Number Theory and Cryptography - get the crucial understanding of elliptic curves math from complete zero,
- then try a great "Introduction to Modern Cryptography" by Jonathan Katz and Yehuda Lindell
and you good to go.
Both are good, Koblitz is really a must and also very interesting for other things besides crytography. Additionally, Arto Salomaa's Public-Key Cryptography is a classic text regarding, surprsise, public-key cryptography and I found it was particularly clear and well-written.