JFYI...  we currently own two single family homes, right next door to each other.  We are trying to sell the smaller of the two, but renting it out has crossed our minds.  Many of our friends are landlords, too:  young, a little bit of financial headroom, own 1-2 properties.

The main thing that stops us?  Being landlords to our neighbors.  Two data points:

1) My story:  About a decade ago, when I was a renter, I lived next door to my landlord, who seemed a very reasonable fellow.  Late one night, I heard banging on our door.  It was the landlord's girlfriend, who had just been beaten by the landlord.  We protected her, called the police, who came and carted the landlord off to jail.  The next day, the landlord was out... and served us immediately with eviction papers.  It was a fraudulent eviction, I called our sexy attack dog lawyer, and things got legally nasty for a while.

2) Friends' anecdotes:  As a landlord, they say, you will eventually evict someone.  It is inevitable.  Therefore,

• Follow eviction procedures in the lease to the letter.  Do not give second chances, accept partial payments, etc. or they will bite you in court.
• If you rent to friends, remember you may have to evict that friend.
• If you live near the property, you have to live near someone who hates you, for kicking you out of "their home."

So go for it... as long as you are not next door neighbors to your renters.