VHDL from scratch. I write embedded software (ASM, C, C++) for a living, so the language syntax is easy enough. The bit I'm struggling with at the moment is exactly what makomk has said - tailoring the HDL to the FPGA. I still think in terms of high level code that has the correct behaviour in the simulator, not how best to utilise the available slices, flip flops, block rams etc. With any luck this will come with time. It seems that getting the VHDL working would be an interesting challenge when I start understanding things.
I sincerely wish you good luck, but this project isn't a good starting assignment for a beginner. The competitive motivation element (bounties, etc.) is already almost gone. What you have now is just a combination of workarounds for the deficiences in the Xilinx toolchain; e.g. the use of 512-bit vectors where 16-element array of 32-bit vectors would produce much cleaner code. This skill has a value now, but Xilinx will eventually fix it in some future release and the skill would start to look cargo-cult-ey.
Also, in your past experience, how often have you faced a problem where you could drop half of the valid results and the project would still appear to work and be valuable?
I'm not trying to discourage you at all from working on a miner, just set yourself appropriate goals; e.g. use a comm protocol with CRC so you'll know the actual BERT of your transport.
You can also restore the competitive element for yourself and make a first Litecoin FPGA hasher.
Again: good luck.