A proper fix would be to check if we had a carry at the end, instead of ignoring it, and add "r" (2**32 + 977) to the result in that case. Ofcourse, with proper carry propagation again (so one check e.g. an addition + 4 more additions = 5 more 64-bit additions). Only then we can be guaranteed the result is correct, during the computations, not after we collected lots of invalid outputs.
That shouldn't incur much overhead, in fact we might be able to do it with 4 lines of code for each occurrence:
// I think we just discard the carry from the previous mod reduce
UADDO(r[0],r[0],0x0000000F000003D1); // 2**32 + 977
UADDC(r[1],r[1],0);
UADDC(r[2],r[2],0);
UADDC(r[3],r[3],0);
// And this carry too
I *think* we are keeping everything mod 256 bits so the highest carries can just be discarded. Or maybe it's actually mod N and it's making assurances above that the product is always less than N.