Turns out this wasn't as big of an obstacle as people thought though, GDDR5 chips are expensive but there are mountains of DDR3 chips laying around for a cheap price. Since the algorithm scales perfectly in parallel, there is no reason why these slow DDR3 chips can't be used en masse to provide the required memory for ASIC's.
But putting chips in parallel is complicated and expensive to manufacture. It's a major reason why video card manufacturers have almost always gone fast but narrow vs wide but slow when given the choice. Even the most integrated wide and slow method, HBM, remains expensive and rare.