You forget about Nery, as Inoue for the first time was sent into the canvass. He doesn't look hurt though, but it was the first time that he was caught with a counter and went down with a 8 count.
Against Tapales it was a back and forth fight, but it was obvious that Tapales doesn't have any knockout power and that's why even if he hit Inoue openly, Inoue just brushed that shot and was not affected. While his total accumulation of volume on Tapales was too much specially those body shots that took it's damage overtime.
Nery, Fulton and Tapales were all very good opponents, so it is not like Inoue is only fighting cab drivers. It is just that everything looks like a mismatch when you are so superior. Even if he moved to featherweight, we would see more of the same. He already beat Fulton and Nick Ball looked very average against Doheny, who Inoue already beat easily. Espinoza and Leo aren’t at the elite level either, so even if he fought those champions there will still be critics who say he is cherry picking.