I'm sure there could be a mechanism that allows you to swap fully charged batteries for the empties as well so charging times wouldn't be an issue if you have spares.
The problem is that you cannot simply swap your batteries as fast as you can fill your gas tank to full. Also, anyone would not want to go through the hassle of opening up your hood(or anywhere your battery is stored) and the hassle of removing and reattaching some things when you are in a long distance travel because you could have had used that time to do other more important things or rest.
Free energy is free whereas gas costs a lot of money. Doesn't really matter if it takes a little longer to 'fill up' and seems a small price to pay for free energy.
Still, the time it would take for it to be fully charged is a long time and typically charging happens at the busiest time of the day(which is in the morning). You would not need to drive to work at night, do you? (unless you are in a night shift or travels for only a kilometer or two, the long charging time is fine).
Having to swap batteries is a small price to pay. I'm sure changing them will become as easy as swapping them in your remote or laptop at some point in the future as well. These things always become more efficient. The time it takes to charge cars will become faster as well. Look how quick phones charge now. I remember having to charge my very first few phones for hours back in the day and now they are super fast. There will be more and more places available to charge them also. I'm seeing charging points all over the place, and as electric cars become more popular so will access points to charge them. You can also buy hybrid cars that take both gas and electricity. If you ever run out of juice then you can stop off at a place to fill up gas, but I imagine people with electric cars become accustomed to knowing how far they can go on a charge and factor that into the places they go to.