When you say "from scratch", do you mean that you changed assumeValid to be 0 (instead of defaultAssumeValid) ?
With "from scratch" I meant no blocks have been stored, folders
blocks and
chainstate were empty. I didn't touch or use
assumeValid in my
bitcoin.conf.
In
this post I listed a few other configuration details, not entirely sure if I had
blocksonly=1 in my config. I would use that too for an IBD.