An address is absolutely required if you want this process to be as efficient as possible... any "passphrase" generated and used in conjunction with a valid BIP39 seed will generate a valid wallet and addresses... You don't get an "invalid passphrase" error or anything like that like you would with encryption passphrases etc.

So, unless you have some additional method for checking if the wallet/addresses are they correct ones, you have no way of knowing if the generated wallet is correct or not.

The only other option would be to search for the addresses using a blockexplorer API and see if they have any transaction history etc... but this is a LOT slower (and generally rate limited) that simply comparing a "known" address to the first X addresses that would be generated by a given wallet.