Despite the unprecedented tumor regression and long-term survival benefit observed with anti-programmed death (PD) [anti-PD-1 or anti-B7-homolog 1 (B7-H1)] therapy in patients with advanced cancers, a large portion of patients do not benefit from such treatment and a fraction of responders relapse. Current efforts to overcome resistance and improve efficacy of anti-PD therapy require a clear understanding of resistance and should precede current avenues using random combinations with available treatment regimens. Here, we categorized three types of resistance, namely target-missing, primary, and acquired resistance. This categorization requires reliable, accurate tissue sampling and appropriate interpretation of results based on the four classifications of tumor immunity in the microenvironment (TIME). We believe that fundamental understanding of these complex tumor-immune interactions and of the cellular and molecular mechanisms underlying these types of true resistance is the key for targeting the right targets in combination with or beyond anti-PD therapy in the future.