In order to survey a universe of major histocompatibility complex (MHC)-presented peptide antigens whose numbers greatly exceed the diversity of the T cell repertoire, T cell receptors (TCRs) are thought to be cross-reactive. However, the nature and extent of TCR cross-reactivity has not been conclusively measured experimentally. We developed a system to identify MHC-presented peptide ligands by combining TCR selection of highly diverse yeast-displayed peptide-MHC libraries with deep sequencing. Although we identified hundreds of peptides reactive with each of five different mouse and human TCRs, the selected peptides possessed TCR recognition motifs that bore a close resemblance to their known antigens. This structural conservation of the TCR interaction surface allowed us to exploit deep-sequencing information to computationally identify activating microbial and self-ligands for human autoimmune TCRs. The mechanistic basis of TCR cross-reactivity described here enables effective surveillance of diverse self and foreign antigens without necessitating degenerate recognition of nonhomologous peptides.