The mask does not protect YOU. It is meant to reduce spread in case you got it but don't know it because its at an early stage or you are asymptomatic like a quarter of cases are.
akram143 is correct that keeping your mouth shut does achieve the same thing. But can you do it? Can you not TALK at all while outside? That's the reason for the fabric masks. Also coughs and sneezes...
Indeed your hands should not touch anything. If they do, don't touch yourself until you can wash them with soap, etc. Especially important to not touch your face or "adjust" your mask once you are outside, simply don't touch it again until you return and washed your hands. And then wash it / dispose it properly.
You musn't underestimate the virus ability to spread from inanimate objects you may casually touch with your hands. Wearing gloves merely moves the danger to the gloves, but its still there.
But can you tell how I can infect other people just by walking near by them without having any symptoms like cough or sneeze? If you don't have any symptoms, longer contact is nneded to infect him. And at current stage, at least when you're walking outside, masks isn't needed - there is very few new cases of Covid-19 in my country, for example just 5 today in whole country and almost all of it are in hospitals. So, risk to get virus is small.
I wash hands when return from shop, but I had to admit that I touch my face so many times not intentionally and I can't anything about it. When I was wearing mask, I also touched it many times to adjust it.