[...]- The (mother) board has [...] one Molex 8981 connector for power supply.
- Only the 12V pins (+12V:1, GND:2+3) on the Molex are used (for simplicity).
[...] I just had a look at different ATX power supplies: while they had 9A of +12V their 5V rail was above 20A! So maybe I picked the wrong voltage?
The reason why I picked one voltage at all instead of allowing the use of both 5V and 12V: some may not want to power their FPGAs from an ATX power supply but use something dedicated. There, a single supply is strongly preferrable.
On that note: these people may not like the Molex 8981 as much. What about the Tyco 284513-2 from the Buchanan series:
It's basically a question what there are more of: ATX power supply users or dedicated supply users?
I would still stick with Molex 8981. People using bench supplies can easily stick a Molex on their wiring, and people using ATX are already set. If you use a different connector, then everyone has to make custom cables.
And I would go with a 240 pin DIMM socket for the cards. Figure on about a third of the pins being grounds. Add a few for SPD, a few more for JTAG, a few more for whatever serial bus you come up with. Still leaves you with plenty of open pins for power, even if you go with 8 or 10 pins for each likely voltage level.