however to brute force we would still need to go for the full 2^264 route since we cannot know if a phrase would result in a valid checksum, correct?
You would still have to calculate the checksum for all 2
264 combinations, which simply involves a single SHA256. After checking the checksum you will be able to immediately exclude 255 out of every 256 seed phrases (on average).
For the one seed phrase which does pass the checksum, you must then perform 2048 rounds of HMAC-SHA512 to calculate the root seed number, then various more rounds of HMAC-SHA512 alongside elliptic curve multiplication to work down the derivation path, then three SHA256s, one RIPEMD160, and a Base58 conversion to turn that final public key in to an address to check to see if it matches the one you are looking for.
This is obviously far more resource intensive and time consuming that performing a single SHA256 in order to calculate the checksum.