Having a look at the code, the effects of the flag are as follows:

• Version message sets the relay flag to zero
• Prints a warning if an inv message is received (and ignores the inv)
• Prints a warning if a tx message is received (and ignores the tx)

The warnings are disabled for whitelisted peers if the whitelistrelay flag is set.

It doesn't count as misbehavior error though.  A peer could send lots of transactions and inv messages and not get banned.

Transactions that are received from whitelisted peers are still relayed fully.