5. I double spend the payment in another instant X transaction which also happens to hit my masternodes
Impossible. In order for an InstantX transaction to be locked, a majority of the eligible masternodes for that UTXO must sign the lock. Thus there are not enough remaining eligible masternodes to sign the same UTXO again. (If Dash doesn't actually work this way I described, then it should be fixed to work this way.)
What defines eligibility?
Indeed, why wouldn't the same set of masternodes be eligible to sign the same UTXO?
If they've already authorized a transaction spending some UTXO output, then that output can't be spent again. It is already spent. The next transaction will be mathematically invalid.
You could claim that it is unprovable which authorization was first, but it is provable that the masternodes who sign more than once (regardless which one they signed first) are lying and thus must be penalized.
You are correct though that even though the masternodes could be penalized, the damage would already be done in terms of the ambiguity over which of the liar InstantX transactions is the first one and which is the invalid one. So what protocol rule could be employed at the PoW block chain level to avoid a fork?
You raise one of the issues that I had to solve in my design and which any "segregated witness" design must solve. That is why I stated yesterday that you were correct. That is how to order when lying occurs, because per above even detecting that lying occurred and proving who was involved, is not necessarily enough to prevent ambiguity and forking of the block chain. The protocol must be specific and deal with this case. I am just now recovering my logic that was fresh in my mind yesterday (experiencing a dull mind today).
So the protocol rule would have to be something that attempted to discard both of the InstantX transactions that haven't yet been confirmed in the block chain, because we know it is impossible prove anything consistent for consensus about propagation ordering until there is a PoW confirmation. Which is exactly the rule Dash employes:
Clients would be tasked with clearing out conflicting locks and possibly reversing attacker transactions. This would only happen in a case where an attacker submitted multiple locks to the network at once and the network formed consensus on one but not the other.
If no consensus is reached, standard confirmation will be required to assure that a transaction is valid.
But if you discard transactions, that is an attack on payees (and perhaps even on payers that were not colluding with the masternodes). And there is ambiguity for such a rule because of orphaned chains and because timing of propagation is orthogonal to perhaps multiple simultaneous realities of multiple chains (all but one of which will eventually be orphaned but we don't know which one yet). Thus there is no objectivity. Consensus would be what ever the longest chain decided to do. But if other honest mining nodes disagree with the ambiguous decision of the longest chain, then there is a fork.
So yes the lack of ordering in masternode announcements adds to my assertion "
Dash has more holes than Swiss cheese" that I wrote yesterday.
Masternodes could indeed wreck havoc. The InstantX white paper shows some math that claims an adversary needs 2/3 of the masternodes to attain 1.72% chance of controlling the majority of each InstantX authorization. I think this math may be flawed. Can you whip up the correct probability math quickly or should I?
https://www.dash.org/instantx/