The brains of large mammals have lower rates of metabolism than those of small mammals, but the functional consequences of this scaling are not well understood. An attractive target for analysis is axons, whose size, speed and energy consumption are straightforwardly related. Here we show that from shrews to whales, the composition of white matter shifts from compact, slow-conducting, and energetically expensive unmyelinated axons to large, fast-conducting, and energetically inexpensive myelinated axons. The fastest axons have conduction times of 1–5 ms across the neocortex and <1 ms from the eye to the brain, suggesting that in select sets of communicating fibers, large brains reduce transmission delays and metabolic firing costs at the expense of increased volume. Delays and potential imprecision in cross-brain conduction times are especially great in unmyelinated axons, which may transmit information via firing rate rather than precise spike timing. In neocortex, axon size distributions can account for the scaling of per-volume metabolic rate and suggest a maximum supportable firing rate, averaged across all axons, of 7 ± 2 Hz. Axon size distributions also account for the scaling of white matter volume with respect to brain size. The heterogeneous white matter composition found in large brains thus reflects a metabolically constrained trade-off that reduces both volume and conduction time.