With segwit, now the transaction id is not malleable. However, I could still modify the witness part during relaying and make the modified transaction still valid. Am I missing something here?

A potential attack is to make the block size larger by making the witness script larger?