If you put the number of hashes a miner does in a second into perspective it becomes easier to figure things out. A single S19 antminer can compute 95 TH per second. That is 95,000,000,000,000 hashes per second. A block header has a nonce that can go from 0 to 4,294,967,295 that means each time the miner goes through all those nonces it has to change something else (extra nonce in coinbase transaction to run from 0 again). S19 would change extra nonce 22,118 times per second.
So basically at least a transaction in the block is changing which requires changing the merkle root hash and witness merkle root hash. Adding or changing more transactions in that block doesn't add any extra times.
Small like... empty? Probably empty blocks could be solved more easily, due to the fact that there's less information to be hashed each time.
Technically not correct since the miner hashes the fixed length block header (80 bytes) and it won't change whether the block has 1 transaction or thousands.