I use for mining only 1-2 threads, diff = 5000 is OK. Higher diff (20 000 or more) would be the end for me (probably)..
Please notice that higher difficulty means only that the shares are sent less frequently. Your balance on pool would be updating slower, but mining profitability would stay on the same level.