Did you choose the correct elliptic curve to make the calculations ? Bitcoin is using secp256k1.
To generate a private key using openssl:
openssl ecparam -genkey -name secp256k1 -rand /dev/urandom -out $PRIVATE_KEY
To generate the public key related to the previously generated private key:
openssl ec -in $PRIVATE_KEY -pubout -out $PUBLIC_KEY
I'm using the OpenSSL headers and linking against the shared library, not generating stuff from the command line, but as I said for like maybe 1 in 5 signatures everything is fine (because I am lucky enough that the produced signature's S value has not its first byte >= 128).
I read some other threads which said that if that happens then you need to prepend an additional 0x00 byte in front of S but after doing that now I'm getting SCRIPT_ERR_SIG_HIGH_S heheh... so now I'm trying to get S = N-S if S >= N/2 where N is secp256k1's order... any idea if that's the way to go?