Evolution has resulted in profound differences between males and females that extend to non-reproductive organs and are reflected in the susceptibility and progression of diseases. However, the cellular and molecular basis for these differences remains largely unknown. Here we report that adrenal gland tissue renewal is highly active and sexually dimorphic, with female mice showing a 3-fold higher turnover than males. Moreover, in males, homeostasis relies on proliferation of cells within the steroidogenic zone, but females employ an additional stem and/or progenitor compartment situated in the adrenal capsule. Using lineage tracing, sex reversal models, gonadectomy, and dihydrotestosterone treatments, we further show that sex-specific stem cell activity is driven by male hormones that repress recruitment of Gli1⁺ stem cells from the capsule and cell proliferation. Taken together, our findings provide a molecular and cellular basis for adrenal sex dimorphism that may contribute to the increased incidence of adrenal diseases in females.