but, if the recovery seed uses common words aren't they very easy for some one to guess and 'recover' the keys even without touching your device? they could just guess all combos?
There are 4,000,000,000,000,000,000,000,000,000,000,000,000,000 combinations of common words in BIP39 with 12 words (least secure choice) You can take your guess.
Also another problem is that you can't effectively check if the combination is correct. You need to derive the master public key, then you have to generate first account, then you have to generate first address, then you need to check it's transaction history. And still you might have missed a correct combination if the first address is unused for some reason (i.e. the user is deliberately avoiding the first address of first account because of this theoretical possibility of attack).