The limiting factor on the CPU is not memory bandwidth, rather memory latency which is less than 1 GB per second because the memory latency of main memory (if outside of L3) is several hundred clock cycles. With only 8 hardware hyperthreads, that latency isn't entirely masked away.
Whereas, the GPU can run 1000s of hardware threads (not software threads!) which masks away the latency and hits the memory bandwidth as the limit.
This is why the GPU blows away the CPU.
A hardware thread has its own copy of registers so there is nearly no cost to blocking the thread on memory access, so another thread can run which was blocked and is ready to proceed.
I think an optimized algorithm, either for CPU or GPU is going to need really detailed knowledge of memory latency issues - more knowledge than I have. You might be onto something with this - I don't have enough knowledge in this area to know if or what level of improvement managing memory latency could offer. You'll have to wait for BM's response.
I have that knowledge and have done all the research.