The same principle applies if more people start taking Ordinals seriously; as long as my coins can be exchanged for equal value, it doesn't matter what others do.
First, if Ordinals is taken seriously, then you may
not be able to exchange your coins for their nominal value.
Second, it does matter what others think or do. The value of the satoshis you have are directly and indirectly affected by what others think about them. Try to buy a U.S. 1943 steel penny for its nominal value of $0.01. You can't because there is a small number of people in the world who would pay $0.20 for it.