I'm curious how much of your bad experiences with players, as you have described, boil down to their own fundamental inexperience playing with a band.
As a producer myself, I find I'm brutally harsh on myself when it comes down to final mix - making sure everything ebbs and flows properly across the piece. Making sure no one element overpowers another unless I choose to accent a particular segment.
Shit like this takes at least a decade to hone.
Much of the guitar problem playing out live is immaturity, insecurity, or lack of direction.
If a guitar player is immature, overplaying is sometimes just a lack of experience. If the player is insecure (no matter their age, especially if there are other good players present) sometimes they overplay as if it's a pissing match, and showing off. A jam, with a rotating cast of many is bad for encouraging this.
A gigging band should be tight, know exactly what is coming up next. No extra times around for an extra solo unless it is really happening, and the crowd is reacting accordingly Strong band leadership and rehearsal makes this happen. In a recording session, the producer is king.
One of the best guitar players I have ever seen has traveled the country several times gigging. He used to be in a band with Sheryl Crowe before she hit ("she wasn't that good, at the time", he says). Now he is a painter with a mortgage, wife and kids. He has a room of amps, another room of guitars. He says it is his 401k, but he loves them too much to sell any. He is a white guy, plays in an all-black soul band (and in several other bands), and in the soul band, he is magnificent. He plays a lot of our jams, and he is overplaying, arrogant, and way too loud on stage. The soul band has strong leaderhip, and he keeps it in line. In the jam, there is lax leadership. He is trying to be the biggest fish in the pond.