We had a "sports day" in our elementary school in Connecticut. Only it was called something else - can't remember what, exactly. At the end of the school year, the entire school was split into 4 teams including all the grades, and there were many events with points accruing toward the overall competition. It was a full day outside, and I remember it being fun. Even for someone like me, who detested sports of every kind.
Except swimming. I kept up competitive swimming all the way through grade 12. And the early morning before-school workouts got me out of gym for the entirety of my high school career. That alone made it worth it.