Sorry, I'm a bit confused by your statement. please help by writing where should I add or even replace the code in my code above? 
Start by looking at the existing projects on github that are already doing this (recovering WIF with missing characters). You'll get a good idea.As for what I meant, I'll give you an example.
Instead of using the line I mentioned above (key = Key(wif=test_wif)) you'd use these 4 lines and under the if branch instead of throwing an exception you'd just skip when the checksum is invalid. And in the else branch (you add) you perform the conversion to address and the rest of the checks.
Sorry,,, can you replace the code above? because I also confused where put your code.