Let's say a person drops a few million bucks and now has 51% of the network hashing power, and he wants to run the 51% attack we hear so much about. What does he do?

My understanding is that he makes a payment to a person for goods, receive the goods, and then quickly make a second payment to an address he owns. If he happens to mine the next block (which he will with 51% probability), he includes the payment to his own address in the block but not the payment to the person who he received goods from. Then what? Will all further blocks reflect that the first transaction to the defrauded person never happened and the second transaction did happen?

The problem I have with my understanding of this is that there is only marginal benefit to having 51% of the network - having 51% of the network only allows you to double spend 51% of the time. But if you had 40% of the network, you would be able to double spend 40% of the time, which is still a pretty serious problem. There's nothing really special about getting 51%, right?

Is my understanding of this wrong? Thanks.