Purpose: Following a case–control design, as a primary objective, this study aimed to explore the relationship between quality of life (QoL) scores and gray matter (GM) volumes in patients with Huntington’s disease (HD). As a secondary objective, we assessed the relationship between QoL scores and other important behavioral, clinical and demographical variables in patients with HD and HD patients’ caregivers. Methods: We recruited 75 participants (25 HD patients, 25 caregivers, and 25 controls) and assessed their QoL using the World Health Organization Quality of Life scale-Brief Version (WHOQOL-BREF). Participants were also assessed with general cognitive functioning tests and clinical scales. In addition, we acquired MRI scans from all participants. Results: Our results showed that patients exhibited significantly lower scores in all four QoL domains (physical health, psychological wellbeing, social relationships, and relationship with the environment) compared to caregivers and controls. Caregivers showed lower scores than controls in the physical health and the environmental domains. In HD patients, lower scores in QoL domains were associated with lower GM volumes, mainly in the precuneus and the cerebellum. Moreover, in HD patients, physical disability and GM volume reduction were significant predictors of QoL decrease in all domains. For caregivers, years of formal education was the most important predictor of QoL. Conclusions: HD patients exhibit greater GM volume loss as well as lower QoL scores compared to caregivers and controls. However, caregivers displayed lower scores in QoL scores than controls, with years of education being a significant predictor. Our results reflect a first attempt to investigate the relationships among QoL, GM volumes, and other important factors in an HD and HD caregiver sample.