Exactly it is that simple.  The Bitcoin network has no concept of "ownership" only authentication.  If one can sign a transaction with a valid private key they can spend the coins.  If someone generates an address which produces the same public key as your address then they can spend your coins.