I'm a brown man living in Japan. Originally raised in the US. I find the Japanese reputation of being racist kinda overblown, but I can see that there are some systematic issues and it becomes more evident while living here for a long time. I'm not sure if it's a bit different in Tokyo as well, because I don't have any issues making friends with Japanese people here, even with a huge language barrier. Osaka people are always trying to talk to me.
Personally, Japan's had the most tolerable racism out of all the countries I've lived in. I'm just kinda an animal at the zoo here, whereas in the US there's a shit ton of stereotypes from terrorist to socially inept nerd and in Europe I've had to deal with "Are you a Muslim? I'm watching you."