Personally I think working remotely is bad, you decrease your chance to meet new people which could increase your opportunity to get better jobs.
That is one way to look at it and you could call that a downside, another one is you'd have less physical activity, which isn't so healthy, except of course if you prioritize taking a walk and going to the gym. That said, nowadays people who work remotely are also able to network online, with the people who are in the same field as them, they might not meet physically, but they are able to get better opportunities through networking online and through recommendations.
Health part is true, I have been a remote worker for 8 years give or take, and I have gained a lot of weight, and I have been sick more frequently than I have been any other time in my life, I keep getting sick, one day it something on my leg, sometimes it's allergies, sometimes blood pressure, it keeps happening, even my teeth are worse lol.
However "meeting" new people is very viable for future references. I had the same job for many years because I love my job, but I had met with a bunch of people, like dozens, maybe more than 100 by now, in the same business as I am, and I had other business offers before too. Usually they were stuff I wasn't interested in, and I didn't want to risk my current job I love, I worked a few times for some projects in the crypto world, but that's it. So, it's possible to make a network online.