I think they would have to bruteforce your PIN..

AFAIK Ledger Nano S do factory reset (remove private key & seed as well) after 3 invalid PIN attempt.

Can you bypass the user interface by connecting/mounting the disk (somehow) then have access to the file system and private keys stored there?