No one is suggesting that there aren't other factors such as mining pools/miners rejecting Txs, certain blocks hitting the 1MB limit, users not including proper Tx fees, dynamically changing difficulty rate within the retarget, ect...

This doesn't negate the reality that the Poisson distribution still represents a very useful probability curve which matches most real world confirmation times within the bitcoin ecosystem. One could create a more sophisticated algo which anticipated other variables like difficulty adjustments but the probabilistic accuracy wouldn't be much more accurate than the Poisson distribution.