I'm not so sure if you got it finally right. Let's stay with your 2-of-4 multisig wallet scenario.

To generate addresses of this wallet, ALL four extended public keys are needed. You can't afford to loose any of the four! (I think this is what many people don't get right when for signing not all signers are needed, only two in this example are sufficient). So any of the four possible signers needs the extended public keys of all other signers besides his own one to be able to setup the wallet and generate the wallet's addresses.

To sign a transaction two distinct signers of the four are sufficient. This 2-of-4 multisig setup can afford to loose two extended private keys, but, again, all four extended public keys are still required!