The more, the better?

Should I spend some more and get a 8GB (or even 11GB) memory card? Is it worth the extra money? Or is 4GB enough?