I've read about this (a couple of years ago): some block-box AI that created better diagnoses on specific conditions than the doctors, and they didn't know why.

Wolfram Alpha has been doing that for decades.