Maybe we are conflicted about what you mean by 'working set size'? When one is validating, the witness data is part of the data you are working with, right? (duh - otherwise there would be nothing to validate against). Accordingly, is it not part of the 'working set'? Are are we dealing with an arcane ephemeral definition-of-passion?
So again I ask: How, pray tell, does one validate a transaction, if one does not possess the signature data?
Maybe you noticed when you sync full node from zero, it goes fast first and only last months take so long to finish. The reason for this is until last checkpoint, no signature data for transactions is ever validated. Thus you dont need old witness data to sync until last checkpoint. Only after the last checkpoint is reached, signature data for transactions is validated and so witness data is required as well. Basically SegWit helps syncing new full nodes, althought MUCH better performance would be if you just downloaded latest UTXO set and started syncing from this point.