Sure it is all fine and dandy to make various idle threats to withdraw poole mining power - but it is a very risky move to actually attempt some kind of politically divisive action, and then potentially lose mining power when miners move to other pools because they don't want to support such unnecessary and wreckless attempts at blackmailing (that even under decent scenarios seems likely to fail)
Antpool does indeed seem to be reinterpreting the agreement (to assume that those devs magically spoke for core as a whole and could make decisions for the community) with the assumption that was very carefully and clearly refuted before they signed the agreement.
I don't think their threats to withhold segwit will last long and hold as much weight as the threat to switch PoW algo's or dump BTC for alts.
I frequently hear a lot of bullshit being circulated around these interwebs regarding some kind of "clear, unambiguous and binding commitment" to implement a hardfork increase in the blocksize limit.
I have not been studying the HK agreement in detail, but I do recall reading it around the time that it was agreed to and circulated, and I remember a lot of the then complaints from XT/Classic supporting folks that the agreement doesn't really bind anything... which is actually the case, even though they are now trying to change the agreement to cause it to be more binding than it is.
In essence, even taken the language of the agreement, it was sufficiently broad concerning what they were agreeing to being fairly open concerning time lines and that there would be attempts to suss out or at least to measure the extent of consensus regarding what to do.
So anyhow, even if the persons involved in the agreement were sufficient agents to bind action, the action does not bind any kind of specific outcome beyond intending to take reasonable measures to figure out consensus in respect to whether a hardfork may be practical after some testing and release of seg wit.