For me, it is either I keep holding onto that job or I'll find another job that I like. Though, my passive income pays slightly above my salary I'll still working for a job. It is safe to resign if you already have multiple sources of income, aside from your job, and those incomes double or triple your salary.
When you cannot take it anymore, you're free to resign and you won't be problematic with what you're going to face because you've got other source of income. That will support you in finding another job if ever you're too decisive to leave the company as soon as possible.
This generation is making everyone realize that having multiple source of income is a norm. With how tough life is financially, supporting ourselves and our families through many source of income is truly the key.
If someone can stay on that day job and you're just receiving some unpleasant feedbacks and hearsays, it's your determination whether to stay or leave.