The problem is mainly your wood base. It can flex and absorb the majority of your impact force. A heavier hammer is also advisable. A 500g hammer isn't beefy enough for my taste. Get something real and not a toy.

fillippone used an anvil as base for a reason. You want something heavy and sturdy with significant inertia to move it and it should not flex at all. All your impact force should drive the stamp into the washer, no energy should be wasted in flexing or moving the base!

You don't need necessarily to buy an anvil. Get something heavy and beefy from the metal scrapyard, something compact upwards a few kilos with a flat surface is fine. Avoid flat stuff, you want something chunky, blocky, more or less cube-shaped, the heavier the better.