Yes.  Absolutely. 

The Public Key is mathematically related to the Private Key by an algorithm.  If you get the Private Key, you can steal the Bitcoin.  Just work a little bit to discover the algorithm which converts the Public Key to the Private Key and you are totally golden.  First one to discover it is a millionaire!!!  But don't be greedy.   If others find out you did it, the whole system will get less valuable fast.