because of the black-box part that I have mentioned above, we can not provide a consensus algorithm that works fully by ML,
Your perspective is interesting, and even as a non-expert I understand the problem that we maybe do not really understand what's going on inside the "black box". So machine learning may not be (in its present state) the way to "create" a totally new algorithm.
But it could be interesting not to let the NN "create" the algorithm but to use it to select an algorithm from a group of "humanly constructed" algos, creating simulations in the same way AlphaGo or AlphaZero did, only using Bitcoin as the "game" and the attacker(s) as one party and the honest participants as the other one.
The idea would be to simulate different complex attacks, and then compare the values of the different algorithms. This way, one could create inputs based on the incentive structure one wants to achieve, and play around with the values. One of the NNs, for example, could simulate to be an attacker using a "51% + short-sell" attack with different algo configurations, trying to profit from it, with several other NNs playing the roles of different market participants.