The ability of adipose tissue to adapt to a changing nutrient environment is critical to the maintenance of metabolic control. Nutrient excess and deficiency alter the shape of adipose tissue drastically and trigger many events that are collectively known as adipose tissue remodeling. Remodeling of adipose tissue involves more than adipocytes and is controlled by an extensive network of stromal cells and extracellular matrix proteins. Prominent players in this process are adipose tissue macrophages, which are a specialized leukocyte present in lean and obese states that contributes to adipose tissue inflammation. The interest in adipose tissue remodeling has been accelerated by the current epidemic of obesity and the chronic generation of signals that lead to expansion of adipose tissue. It is clear that evidence of dysfunctional remodeling events is a hallmark of obesity associated with metabolic disease. This review summarizes and highlights the recent work in this area and provides a framework in which to consider how adipose tissue macrophages contribute to the remodeling events in lean and obese states. Advancing our understanding of the involvement of macrophages in adipose tissue remodeling will promote one aspect of the new field of “immunometabolism,” which connects control systems developed for regulation of immunity with those that control metabolism. It will also provide insight into how physiologic and pathophysiologic remodeling differs in adipose tissue and identify potential nodes for intervention to break the link between obesity and disease.