If the authorities are the ones who will ask you your phone, I guess you can refuse to give it to them. But if they have a search warrant or a court order for you to surrender your phone, then you cannot refuse it. Refusal means something like obstruction of justice or not cooperating to a lawful search or investigation. So the best way to safe keep your important information is not to store them in your phone.