Yeah, I did try that, didn't work. I think I actually figured it out though:
What did work:
sendtoaddress 0.17291295 paytxfee .0005
Edit:
Actually I lied. Yes, the transaction went through, but it used the recommended .002 fee, not what I had specified.