Your observation is correct, however B4 and B9 are two sides of the same card, and flipping this card causes a slight change in the position of the through-hole, so moving the two cards will produce different combinations.
The index letters on the silver card, which correspond to the plaintext, and the letters on the black card are the results of substitutions, and for each definite combination there are four possible substitution results for each plaintext letter.
Following the simple rules of this puzzle setup, once one side of the silver card has been selected to be used as a mask, and a particular combination placed on the black card has been determined, there is no longer any need to consider the other side of the card, or any other possible combination. The combination determined by this picture can then be used to translate between plaintext and ciphertext.
As for the issue of special characters, as mentioned earlier, this card/set was originally designed to manage strong passwords, so as many characters as possible were retained that could be entered directly via the keyboard.
From my observations.
OP uses two silver cards. One of them is ended on B4 while the other on 89.
B4 card holds letters in qwerty layout as on typical English keyboard while 89 card keeps letters according to their alphabetic order.
Characters on silver cards feed message intended for encoding while characters on black card are relevant to digest.
It is very likely that SEED word coming for encoding is split into two half and each of two silver card is design to encode its own half. Or those cards are used separately to encode even-numbered and uneven words.
But, it is still unclear for me why silver cards hold special characters and numbers. SEED words don't have any of them.
It is highly likely that two characters (a few option for this) in digest serves as decoy as OP encrypts only first 4 letters from the SEED.
Unplugged is a better option for seed phrases, I mean, of course there are other options to generate seed phrases, but when saving them, write them down and don't give them to any electronic device, you should know that even just taking a picture of a note with a seed phrase written on it with your cell phone is extremely risky behavior. Not to mention that in certain circumstances, you can't decrypt/unzip an electronic copy of the seed phrases on a device you can trust.
.
You could even keep a picture of the CipherCard in Gmail's drafts folder, the rules in Outlook's drafts folder, and the ciphertext of the seed phrases in the drafts folder of all your email services, and then use another CipherCard to manage the passwords for all your mailboxes.
.
Hm, wouldn't be better to keep their the one of SSS blobs, encrypted over & above for security with the hardware pgp key? Or even encrypt with such key the whole SEED.