Even someone with brief access to your laptop could simply run ssh-copy-id to some remote server they control.
ssh-copy-id transfers the public key. That is fine. Your public key can be public. It's the private key that you have to protect and often have encrypted.
You're right - I got turn around. It's someone adding a public key to your authorized_keys file that you would need to be wary of.