Belle Mont was originally built for Alexander Williams Mitchell, who grew up in Louisa County, Virginia, only 25 miles from Jefferson's home at Monticello. A graduate of the University of Edinburgh, Scotland, he was a physician, planter, and an early elected official of Franklin (present-day Colbert) County, where he settled around 1820.
Mitchell was among the area's largest owners of enslaved African Americans. In the mid-1820s, Mitchell began the construction of Belle Mont as the centerpiece of his large cotton plantation.
In 1832, following the death of his first wife and his subsequent re-marriage, Mitchell put Belle Mont up for sale and eventually moved to Philadelphia, Pennsylvania. An advertisement appearing in a Huntsville newspaper described "a brick dwelling house," 76 feet across the front, along with "all the necessary houses." The advertisement also noted Belle Mont's 1,760 acres which were planted in clover, grass, corn, and cotton and included an orchard.