It won't be impossible for you, that's for sure. You have an additional source of income besides your job, which is good. However, if you aim to retire early and spend more time with your family, having your own business might get you there faster than working an 8 or 9-hour job daily. The latter may take a lifetime to achieve your goal.
Working for someone else for 8 or 9 hours every day is actually not bad for those who are still unemployed and unable to run their own business through their lives. Because it can also be better than someone who doesn't work at all in their life and those who really need income for their living, but I think the option you gave is also a very good option. Because in the end, when everyone is no longer able to work for other people, they will also choose this option as their main job in life to make money.