Nope, Inoue already has established a good name and he has been mentioned by several boxing legends and some active boxers as well. Whoever Inoue is facing it will always going to generate a good money because the boxing fans would come to see Inoue's performance.
A boxer who have an excellent skill sets can be easily become popular that will attract boxing fans.
While he could generate a good amount of money, it doesn't compare to what he could make in the higher division. We can take a cue from the experience of Pacquiao. He was already popular in the lower division, but he didn't earn the same kind of money until he moved up to the higher division.