1. I noticed that rigs without Teamviewer are more stable or "responsive"
2. On the farm, where I service 27 rigs, I have 1 cheap PC with Teamviewer and SSH or VNC to the rigs.
It is much more convenient than connect by Teamviewer to each Rig. So users just don't know how

I never liked Teamviewer and don't think ever will,
I have an RPi with forwarded ssh port on router, with firewall, fail2ban, ... and do all my stuff through that.
But I'm willing to work with everyone interested in creating a full headless image.
What are the main problems on this way?
No idea.
I think 2unix starts after lightdm, and it calls to run 3main, but not sure.
You can start striping current image from desktop component and see how it goes.