Histone modifiers like acetyltransferases, methyltransferases, and demethylases are critical regulators of most DNA-based nuclear processes, de facto controlling cell cycle progression and cell fate. These enzymes perform very precise post-translational modifications on specific histone residues, which in turn are recognized by different effector modules/proteins. We now have a better understanding of how these enzymes exhibit such specificity. As they often reside in multisubunit complexes, they use associated factors to target their substrates within chromatin structure and select specific histone mark-bearing nucleosomes. In this review, we cover the current understanding of how histone modifiers select their histone targets. We also explain how different experimental approaches can lead to conflicting results about the histone specificity and function of these enzymes.