Then it could go back up again.  I know it's only cold for a few days at a time but again, it's like a roller coaster.  If it would get cold and stay cold, maybe it wouldn't be so bad.
I see people go to work in short sleeves on those cold days then wonder why they come down with a cough/cold/flu.  I bundle up in layers and "peal" as need be to match the temp.