Pancreatic ductal adenocarcinoma (PDA) develops predominantly through pancreatic intraepithelial neoplasia (PanIN) and intraductal papillary mucinous neoplasm (IPMN) precursor lesions. Pancreatic acinar cells are reprogrammed to a “ductal-like” state during PanIN-PDA formation. Here, we demonstrate a parallel mechanism operative in mature duct cells during which functional cells undergo “ductal retrogression” to form IPMN-PDA. We further identify critical antagonistic roles for Brahma-related gene 1 (Brg1), a catalytic subunit of the SWI/SNF complexes, during IPMN-PDA development. In mature duct cells, Brg1 inhibits the dedifferentiation that precedes neoplastic transformation, thus attenuating tumor initiation. In contrast, Brg1 promotes tumorigenesis in full-blown PDA by supporting a mesenchymal-like transcriptional landscape. We further show that JQ1, a drug that is currently being tested in clinical trials for hematological malignancies, impairs PDA tumorigenesis by both mimicking some and inhibiting other Brg1-mediated functions. In summary, our study demonstrates the context-dependent roles of Brg1 and points to potential therapeutic treatment options based on epigenetic regulation in PDA.