I wouldn't say there is anything wrong with it at all. Everyone needs time to themselves, just because you have a family doesn't change that. I finish work around midnight and I enjoy being able to come home and do my own thing for a couple of hours before going to sleep.
It's essential to have 'me time' in a relationship, not having so risks co-dependency (or symptoms thereof). What OP is describing seems very healthy to me, as does your comment (: