I think I have solved it, but it still confuses me a lot.
When I first got this ledger, I have now remembered that I saved a photo of the seeds to my secure folder(mistake #1). I think there must be some malware on the phone, because I have never entered my seeds onto my computer or anywhere else and I don't ever remember opening the image. They must have gotten the 4 digit passcode from monitoring my phone when I used it with other apps, for example some apps are unlocked with the same 4 digit code I use to unlock my ledger(mistake #2). This is the only way I can see that they have accessed it. I've run anti virus etc on there but it's found nothing. I also checked all the installed apps and there is nothing strange here. In fact this phone is rarely used and only really for whatsapp and google authenticator. For this to be true, it would mean someone has had access to my phone for over a month which is very worrying
Physically, the seeds are hidden in my house. I live alone and no one else has access, certainly no one I know knows my passcode or understands bitcoin, or even knows I own a ledger.
$5000 is a nice expensive lesson in security I guess. Thanks all for any replies and help

I'm off to buy a new phone
Ouch! Quite expensive indeed.
Your logic has some mistakes. As soon as anyone had access to your seed, they don't need your pin or anything: with the seed they can re-create the wallet on any other computer or smartphone, or ledger or ... (other options may exist).
The wallet doesn't contain the coins, it only handles the private keys (simplified explanation) and the private keys are based on your seed.
The private keys are the most important thing: they are the only thing that allows spending the coins.
So whoever got your seed, it was all he needed to steal your money.