The problem is the double-spending. An SPV could set you up, if say, another miner paid them so.