Fees varies from time to time so if you set the fee quite low and there's a sudden surge in the fee then your transaction might get some delay due to it. The miners are choosing those transactions that pays for quite higher and they're processing it even there are earlier transactions that has been sent to the network but the fees are quite low. It is correct to adjust the fee but make sure that the fee you're setting depends on how quick you want it to be or else, you might be surprised if it gets delayed a lot.