Sometimes the progress gets so bad I have to restart Bitcoin Core many times.
I don't think that helps. If anything, it makes it worse.
"Being slow" can have 2 main causes: slow internet, or slow hardware. If the problem is your internet, restarting Bitcoin Core won't help. If it's your hardware, all you can do is shut down other programs. Restarting a web browser can (temporarily) help to reduce it's memory consumption.
Mmm. But when I do this, I normally see little changes though. And it’s tempting to do when I see progress increase of 0.02 per hour. But based on a reply I saw here, could it be the block it’s handling at the time? Because I do use speed test (used this so much the previous night) and I can get 3MBS or like last night’s 15MBS. As for my CPU usage it’s currently below 10% (and the increase per hour is 1.30%) but if I get the slow progress again, I’ll check.