While segregated witnesses is a soft-fork, because it adds new data blocks that old nodes don't relay segwit nodes can't sync from non-segwit nodes and still be fully validating; once the segwit softfork has activated full nodes need witness data to function. This poses a major problem during deployment: if full node adoption lags miner adoption, the segwit-supporting P2P network can partition and lose consensus.

While Pieter Wuille's segwit branch(1) doesn't yet implement a fix for the above problem, the obvious thing to do is to add a new service bit such as NODE_SEGWIT, and/or bump the protocol version, and for outgoing peers only connect to peers with segwit support. Interestingly, a closely related problem already exists in Bitcoin Core: neither addrman nor the outgoing connection thread takes what service bits a peer advertises into account. So if a large number of non-block-relaying nodes joined the network and advertised their addresses the network could, in theory, partition even without an explicit attack. (My own full-RBF fork of Bitcoin Core does fix(2) this issue, though by accident!)

So your security assumption goes from not being sybilled, and no miner collusion, goes to "and I am not censored from other nodes which altogether do 100% validation" (for receiving fraud proofs). This is a far-more scalable full-node or partial-full-node model that we could evolve to. It's a security tradeoff. It's certainly not one that everyone would want to make, but it doesn't effect those who wouldn't want that.