I had a similar problem recently[1], after following the advice quoted below[2] it runs stable at ~900 MB memory usage.

See the part about transaction flooding:

https://bitcoin.org/en/release/v0.11.0

On reddit someone set minrelaytxfee to 0.0001 (double what is suggested above) and his node is using very little RAM.

Interesting. Thank you.