OS: Ubuntu 16.04
RAM: G.SKILL TridentZ 16GB (8GBx2) DDR4 3200MHz
Two GPUs work no matter how I connect them to motherboard, be it via risers or on 16x slots. If M.2 installed, only one GPU works. I'm currently using one 850W Thermaltake toughpower RGB PSU to run two GPUs, 850W should easily be able run three GPUs, right? Risers are SATA powered (instead of Molex).
Yeah, your 850W power supply should be fine.
I'm not too familiar with the new Ryzen series, but the chipset/CPU should have enough PCIe lanes for more than 2 cards, albeit not with maximum bandwidth, but that doesn't matter for this type of use.
When you say that 2 GPUs work no matter how you connect them, have you also tried putting one in the primary x16 slot (through an x1 riser) and then others on risers only through the physical x1 slots (and not the other x16 slot)?
Have you tried opening the BIOS and looking for settings regarding the PCIe link speed? It should be somewhere under chipset settings. If you find it, set the link speed to 1.