Perhaps, your thinking was right as you are gaining attention from the public and yes, it makes sense for the bad guys but on the other side, this will also build confidence in the others who are also a Bitcoin holder in your place. It is a sort of spreading information but still, the best option is to remain silent from the people around just to be safe.
Because people assume that once you hold Bitcoin, you are already rich --perhaps not all. Therefore, to prevent such a scenario, we don't have to give them any idea who we are and what we are doing. We never know what is in their mind...