So if I mine using a mining pool (ie with my own server, not using a paid-for remote mining service where its all done on the server run by the provider) then I don't need to download the blockchain?
Correct. Only the mining pool has to download the full blockchain. When you mine with a mining pool, all the heavy load has to be done by the mining pool operator. Your local mining software just has to send shares to the pool as soon as possible. No blockchain has to be download locally.