M-of-N signatures are checked in scripts by OP_CHECKMULTISIG

It is limited to 20 public keys, so the limit of M and N is 20.
Source: https://github.com/bitcoin/bitcoin/blob/master/src/script.cpp#L871

The limit exists out of concern for limting the size and computational complexity of transactions. Private keys are not divided in M-of-N transactions. Rather, there are N private keys, of which M need to sign the transaction for it to be valid.