The immune system and its orchestrated response are affected by a multitude of endogenous and exogenous factors, modulators and challenges. One of the most frequent differences described in the immune response is its vigor and activity in females compared to males, leading to the consequent increase in autoimmune conditions seen in the female population as well as differences in the immune response to pathogens and viruses. The following review summarizes our present knowledge on sex differences in the immune response, detailing the hormonal and genetic effects that have been proposed as explanatory mechanisms. Sexual hormones, mostly estrogen but also progesterone and testosterone, affect immune cells quantitatively and qualitatively. Relevant research has focused on the impact of hormones on cytokine production by the different effector cells, as well as impact on immunoglobulin production by B lymphocytes and activity of granulocytes and NK cells. The biological aspects are complemented by research data on the possible modulatory role of the X chromosome. In addition to biological differences, the frequently neglected role of gender as an immunomodulator is introduced and explored. Gender affects all areas of human life and consequently affects the different steps of an immune response. Exposure to various types of antigens, access to health promotion programs and health care, as well as prioritization of health needs and household resource allocation all affect the different response of females and males to immunologic challenges.