Yes, you can with the Server Browser like DrQuantum said. Or you can connect via the console:
connect serverip:serverport
In our Payload server case:
connect 66.151.138.182:27015
Or simply use this link:
steam://connect/66.151.138.182:27015(You can click the link above the servers gametracker banner on our main page as well, it will launch TF2 and connect to the servers)