The GPU can compute 1000s of lookups on that memory in parallel, thus obviating the random memory latency to be bound at the superior memory bandwidth of the GPU.
I'm not an expert in this area, but as far as I know the GPU has a wider bus, so can do more memory access (x4), but is slower so can do less (x0.25). So I think you could get comparable speeds out of a high end GPU as for a high end processor.
The limiting factor on the CPU is not memory bandwidth, rather memory latency which is less than 1 GB per second because the memory latency of main memory (if outside of L3) is several hundred clock cycles. With only 8 hardware hyperthreads, that latency isn't entirely masked away.
Whereas, the GPU can run 1000s of hardware threads (not software threads!) which masks away the latency and hits the memory bandwidth as the limit.
This is why the GPU blows away the CPU.
A hardware thread has its own copy of registers so there is nearly no cost to blocking the thread on memory access, so another thread can run which was blocked and is ready to proceed.