The raw binary is not the seed, and you can verify that by simply changing the mnemonic length; raw binary remains the same, but BIP39 seed changes.

Raw binary is represented by the mnemonic in a human readable format. The seed is derived from the mnemonic, not the opposite. This is probably what confuses you. When you have the mnemonic sentence, you pass it through 2048 PBKDF2 rounds, and you get the seed. This explains why increasing the words (12 -> 15) does not change the sentence (only adds new words), while the seed is completely different.