If we didn't have the Notes then I agree, Fast mode would have the same disadvantages as any other mixer. But since outputs from Notes and Fast look exactly the same there is no way for any outside observer to know which mode you used. So Fast mode is as secure as the Notes from a privacy set standpoint. I would still recommend using Notes regardless because they offer the end-user full control over the process.
That's a good point, and one I did not consider. For an external observer using blockchain analysis, then a fast mix appears identical to someone using notes. They can see the deposit being made, but since they don't know if the user is using fast or notes, they are unable to reach any conclusions about the time frame of when the withdrawal will be made. The fast user benefits from being in the larger anonymity set provided by notes users.