I've read about this (a couple of years ago): some block-box AI that created better diagnoses on specific conditions than the doctors, and they didn't know why.
...
Wolfram Alpha has been doing that for decades.
I find it quite shocking that a guy as obviously intelligent as you are so reluctant to embrace this new technology. Artificial intelligence is already being used in several areas and improving productivity a lot, even if you don't see apps you can download from Google Play or Apple Store as useful. For me it is your emotional mind that is working there making you reluctant to accept it.
AI is a disruption as was the internet itself in its day or cell phones or bitcoin. It is not a question of whether it will be useful or not, it is being used massively today in many fields and medicine is just one of them. I can understand that you don't want a forum full of AIs and not humans (just wait until you see the AI agents in full operation, and see if you can distinguish them from humans) but that vision you have of AI is quite reminiscent of the one scribes had when the printing press was invented.
People destroyed printing presses out of fear. What will we do to AI?Just like with AI, people worried about job security and the spread of disinformation. Machines were destroyed and book merchants were chased out of town.