Does this mean that version number actually is not part of the consensus rule, since miners could set whatever version number they want?
There are consensus rules regarding what the version number can be. But there is no consensus rule saying that the version number must be a particular value.
E.g. there is a rule saying that the number cannot be less than 3. But there is no rule saying that it must be 4, so it can be 4, 5, or any other number so long as it is greater than 3.