2-3 days ago I was mining solo and getting more than half the blocks when difficulty was like 25k.
At some point I took some core off mining because I was affecting difficulty for myself

That would be me. Testing Xcz.mypool.name starting yesterday w/GPU rig. Net hash went from 150k to 620k then I set up my own pool and switched from mypool.name to my private one. Gonna grab 3 million or so coins then move on. Expect Network diff to climb fast over the next few days.
Actually what I meant was that difficulty was so low there was no point using all my cpu to mine. I could save power and get almost as many coins with fewer cores because difficulty was so low
But then difficulty rose fast for the reason you explained. Pity I thought the coin was gaining in popularity but it turned to be just one person
3 million coins is a lot to mine whatever the difficulty. You need almost 100000 solo blocks so it would take you months even if you are the only miner