No, I already promised them I would remove the flags so I better stick to my promises.
Well, you can't

You can withdraw your support, but that doesn't mean the Flag becomes inactive.
You may want to read up on
trust flags:
Creating or supporting a scammer flag is actively affirming a set of pretty clear fact-statements.