I completely agree with your points regarding freedom of speech, but freedom of speech doesn't include freedom to disrupt and derail. I'd never dream of trying to have posts deleted or users banned simply because they are expressing unpopular opinions. The whole point of freedom of speech is protect unpopular opinions; popular opinions don't need protecting. I'm sure you've seen the relevant XKCD which explains it well. However, CH/TOAA (as an example), has literally been posting the same nonsense regarding trust/merit/gangs/inner circles/etc. for over a year now. Any thread remotely related to merit or trust, he shows up with the same wall of nonsense, and the thread is rapidly derailed. I don't know of anyone who has reasonably defended that, or possibly could, as "not trolling".

Furthermore, the most commonly broken rule is subjective too. Who decides what is low value or uninteresting? Why enforce the spam rule but not the trolling rule?