Your multimeter will have a few Ohms settings.
Choose one and hold the two leads together.
The display should show 0 ohm.
Now hold the two leads to each end of the fuse, the display should show 0 again.
If it doesn't, the fuse is blown and has to be replaced.

This might help: https://www.youtube.com/watch?v=iG-AsEemqD4