I did convert the 32 bytes into integer and I believe the curve point is generated using the order of the base point, now can we remove the order of the base point and look at only the values on the curve?
No you can't. On elliptic curve you can multiply a point by a number, but you can't divide a point by anything.
If you could, we had know k and thus we can retrieve private keys from every p2pk signatures on the bitcoin network.
Edit: Or if you want to substract the point (not divide it), you can add the point G'=(Gx, -Gy).
Then, you will have P-G=P+G'