I suggest a penalty/debuff for clothing that does not fit the biome.
The penalty could be
- you should not go into the desert with Fur clothing
- and not strive with with Cloth Armor into the cold north
The penalty could be
- more thirst when overheated
- slower movement or reduced aiming ability when not well protected from cold