There needs to be a survival tier of clothing, as even just the common cloth rags require the loom. Survival needs a caveman level of clothes using simple leather/fur wraps. Primitive Humanoids did not need a weaving loom to cloth themselves, and maybe a more primitive version of the spinning wheel can be used for cloth wraps. The loom is in the metalworking tech tier and not a survival tool - so the survival part of the game is spent running around naked - which is not fun with the rainstorms.
Since you get basic clothing tech at the start of the game just change the loom to primitive bench since those look like survival rags anyways and you are really just wearing the cloth you can make with primitive bench. Make the loom only needed for the medevial clothing, since once you have the loom and it rewards better clothes why would you waste time making the rag clothing?