Heh, this comes up often. It's actually funny because it only happens when there are major problems such as multiple crashes, steam connection issues, broken quickplay, etc... Since our servers are usually pretty busy, they don't go below 22 players very often. Everybody freaks out when that happnes
Fear not! The Payload/Votation servers are set to 32 players, and I don't see it changing anytime soon. I don't like 24 players, not enough heads to snipe.