I am seeing some banning of peers in my logs for various reasons. If you drop down to a low connection count, it is probably due to various peer issues stemming from masternodes not being setup yet, and the peers repeatedly requesting masternode information.

You can set your debug to not ban so easy: (also delete your peers.dat so that you start with fresh scores for each peer)

I have 57 connections.