If the wallet really encrypted, you won't able to find private key/master private key (xprv) on the raw file.
There are several wallet.dat format, but the one i know use PBKDF2 where PBKDF2 private key used to encrypt master private key & PBKDF2 private key is encrypted with user's password.

There are few tools to brute-force Bitcoin core (such as https://github.com/glv2/bruteforce-wallet), but i've no idea if it's works with broken/corrupt wallet.
Personally i'd recommend professional recovery services (such as https://walletrecoveryservices.com/) in this case.