VCU1525 has 2 x QSFP28 connectors, so you link 2 boards with 2 x 100 gigabit ethernet cables, and two boards is the max that can be daisy chained. With the Bittware XUPP3R board, since it has 4 x QSFP28 connectors, there is no limit to the daisy chain length. These specialized ethernet cables have nothing to do with internet, they are solely so data can flow from one FPGA board to the next board. The cables are around $40 each.
I was wondering about other ways of setting up the communication between the boards.
Looking at the VCU1525 user's guide they have 8 transceivers hooked to Ethernet ports and 16 transceivers hooked to PCIe, remaining transceiver blocks aren't connected. Also, it looks like the required intellectual property block for the PCIe support.
Have you considered plugging those boards into a PCIe passive backplane type of interface, not to the actual PCIe computer motherboard? This should be very simple to interface, even to the point of not really obeying the official PCIe protocol rules if the passive backplane contains only the boards running our project.
Can you write a paragraph or two about the relative logic overhead required to implement inter-board communication over Ethernet versus PCIe?
I currently have all my development hardware in storage, so I can't even make a dry-run with an evaluation version of the software.