The meaning or origin of many day-to-day phrases is often lost in the mists of time (what does that mean?), but still are in regular use. Why' losing one's rag'?
I have no idea but it indicates 'losing one's temper', something hopefully everyone would understand.
Tempers flare up.
Tempers are lost.
Tempers have tantrums.
Tempers get hot.
Now why don't you leave the bl***y subject alone!!!!! You are making me mad