by the second party not signing. you cant have them back. because it requires dual signing.
by this i agree the second party cannot simply "take it from you" without your signature, but you cant take it without theirs
IIRC there is a timeout period after one party initiates the closing of a payment channel. Please take this with a grain of salt, as I'm unsure and this would be a *stupid flaw* to enable the possibility of indefinite state of irresoluteness. I'll do some research on this once I find more time.