There are many improvements in 0.13 and 0.14 that are totally unrelated to segwit and if segwit never gets activated the segwit code goes untouched, so forking off 0.12 is just leaving us with ancient code.
If segwit does not get activated, then the segwit code should be removed. Sounds like a horrible and potentially dangerous maintenance issue to have unused code lying around.
So then the question is, regarding the non segwit performance improvements in 0.13 and 0.14, is it easier to remove the segwit code, or implement the performance improvements to a pre-segwit version? I suppose only those that are really familiar with the code will know, and it might be a combination of doing both depending on the area of code that needs to be modified.