if given a public key, how would you use AI to tackle the algorithm and solve for the appropriate private key?
You wouldn't. You'd use a quantum computer running Shor's algorithm.
AI might help you to derive a more efficient algorithm, and improve your solution time
slightly, but it does next to nothing to address the fundamental issue, which is the sheer number of potential solutions. Whereas a quantum computer
does address this, because its processing power scales differently.
A conventional computer can solve a problem 'x' in 'y' seconds, taking 'z' number of steps.
If you use AI to improve your algorithm, then perhaps it can solve problem 'x' in 'y/2' seconds, so twice as fast - but it will still take 'z' number of computational steps to do so.
The advantage of a quantum computer is that it can drastically reduce 'z', the number of steps required. This is why they are 'faster'.
Where a classical computer with 'n' bits can represent 'n' states, a quantum computer can represent 2^n states. This is because the potential outcomes are superposed.
So as we increase complexity, the number of states that can be represented are as follows:
Classical: 1,2,3,4,5,6,7,8 etc.
Quantum: 1,2,4,8,16,32,64,128 etc.
The upshot is that whilst a classical computer takes an unimaginably huge 2^128 operations to derive the bitcoin private key, a QC running Shor takes a mere 128^3.
It doesn't matter how great your algorithm is, there is always the limit that a classical computer still faces that huge number of processing steps.