Honestly, it doesn't matter, because I used to smoke myself and I understand both sides.
Everyone with there own opinion and smokes are not supposed to be seen like an alien. They are humans too.
Occasional smokers are better but immediately you become a smoker addict, problem set in here. If you are an occasional smoker people around you may not even know you smoke. I know some people who smoke but doesn't smoke in an open place.
Smoke is not the only causative of cancer and other diseases which we don't need to assume that smokers could be the only ones that could die young.
The word only here sound somehow funny. This shows people knows it could have significant effect on their body. Like you said other things can cause people death as well but people always see smoker as people buying and doing what can easily cause their death. Although I don't against smoking but I hate being addicted.