Personally I think working remotely is bad, you decrease your chance to meet new people which could increase your opportunity to get better jobs.
You have a point there except for the fact that, this doesn’t seem to acknowledge the fact that, people could be met anywhere these days. The web has made the world a global village they say, you don’t need to step out of your house to meet new people or find new engagements. When you surf the web just enough and try to connect with people, you might just find them readily available.
That is one way to look at it and you could call that a downside, another one is you'd have less physical activity, which isn't so healthy, except of course if you prioritize taking a walk and going to the gym. That said, nowadays people who work remotely are also able to network online, with the people who are in the same field as them, they might not meet physically, but they are able to get better opportunities through networking online and through recommendations.
It all comes down to timing, just how well do you use your time. People have some workout equipments at home and just like you create time for food, you could always create time for just any exercise, recreational activities or meetups. That’s why it’s a remote job, you do it at your convenience.