I think I've solved the reason why the fan rpm doesn't sound stable even when the fan is manually set to some specific speed and have a solution to fix it.
The fan is being controlled with software generated pwm as there doesn't appear to be any hw pwm capable pins unused on the board. The software solution needs to keep a stable timing with the pwm in order for the fan to keep the same rpm. At the same time, the MCU can have a frequency of several values between 240 MHz and 1.20 GHz. By default, this works so that if there's zero load then the frequency will eventually drop to 240 MHz and when there's any load spike then the frequency jumps directly to 1.20 GHz no matter how small the actual cpu demand was. I used the following command to monitor the frequency and it's clear that it's constantly jumping between minimum and maximum even when nobody is using the dashboard:
while true ; do echo "$(date) - $(sudo cat /sys/devices/system/cpu/cpufreq/policy0/cpuinfo_cur_freq)" ; sleep 1 ; done
The constant cpu frequency variation makes it difficult for the software pwm to maintain exact rpm. It's close but still the variation is audible. Looking at the frequency usage stats with the "cpufreq-info" command, we can also see that the cpu is spending much time at maximum frequency:
240 MHz:42.72%, 480 MHz:24.41%, 648 MHz:0.00%, 816 MHz:0.00%, 912 MHz:0.00%, 960 MHz:0.00%, 1.01 GHz:0.01%, 1.10 GHz:0.01%, 1.20 GHz:32.85% (2229569)
The solution itself is rather simple. The default behaviour is to react instantly to any processing power demand and bump the frequency to maximum. However, other behaviours are also available and switching to the one called "conservative" results in slower changes in the frequency is processing power is needed. It turns out that the demand with the current software content is so low in reality that with the "conservative" setting the frequency rarely increases resulting in the software pwm staying stable and the audible changes in the manually set fan rpm going away.
First, without saving the change, the behaviour (frequency governor) can by changed with:
sudo cpufreq-set -g conservative
If that doesn't help then the change can be reverted back with:
sudo cpufreq-set -g ondemand
Making the change permanent requires modifying /etc/default/cpufrequtils and replacing "GOVERNOR=ondemand" with "GOVERNOR=conservative". The other positive side effect is that since the higher frequencies are no longer used that often, the MCU temperature also stays a little bit lower.
With the changed governor, "cpufreq-info" now shows (after a stats reset):
cpufreq stats: 240 MHz:99.21%, 480 MHz:0.30%, 648 MHz:0.07%, 816 MHz:0.08%, 912 MHz:0.05%, 960 MHz:0.05%, 1.01 GHz:0.03%, 1.10 GHz:0.03%, 1.20 GHz:0.17% (18182)