What exactly is the maximum message length in OP_RETURN?
It is defined by the maximum size of the block. See:
https://mempool.space/tx/516c63376556d87c4779033327184ee00a08c4c14498e14673357ce4a791406bSo, in practice, it is something slightly below 1 MB. And if OP_RETURN is placed in unexecuted OP_IF branch, behind P2WSH, then it can take
slightly less than 4 MB
(edit: only up to 10000 bytes, because of MAX_SCRIPT_SIZE).
but what is the maximum amount where any block containing it would be considered simply invalid?
It would happen only when transaction size would exceed the maximum block size, or when OP_RETURN will be executed in witness space.
because I remembered that in the bitcoin developer guide, the maximum was 80 bytes, however some sources say that it's 75 or even 40
There are standardness rules. And it is set by default to 83 bytes, where one byte is taken by OP_RETURN, and two bytes are taken to declare the size of the stack push. And if you ignore Script decoding, then in practice, you can use one byte for OP_RETURN, and 82 bytes for anything.
Edit: Some people say 75, because this is the limit for which you can declare stack push size with a single byte, and not with two bytes. Others say 40, because it was historically set to this value. And in the latest version, it is unlimited by default (and only limited by other limits, like block size limit), and can be limited only by each node operator individually.