Development of testes in the mammalian embryo requires the formation and assembly of several cell types that allow these organs to achieve their roles in male reproduction and endocrine regulation. Testis development is unusual in that several cell types such as Sertoli, Leydig, and spermatogonial cells arise from bipotential precursors present in the precursor tissue, the genital ridge. These cell types do not differentiate independently but depend on signals from Sertoli cells that differentiate under the influence of transcription factors SRY and SOX9. While these steps are becoming better understood, the origins and roles of many testicular cell types and structures—including peritubular myoid cells, the tunica albuginea, the arterial and venous blood vasculature, lymphatic vessels, macrophages, and nerve cells—have remained unclear. This review synthesizes current knowledge of how the architecture of the testis unfolds and highlights the questions that remain to be explored, thus providing a roadmap for future studies that may help illuminate the causes of XY disorders of sex development, infertility, and testicular cancers.