On a steep knoll overlooking the almost vanished early community of Elk Garden, the Hendricks-Stuart House is one of several large hilltop houses constructed as the centerpiece of an agricultural estate. Although most farms were small-scale operations, a few were owned by families who acquired large tracts of land and became local leaders in agriculture and in the professions and politics. Built by the prominent Hendricks family, Elk Garden was purchased in the late nineteenth century by the Stuart family. The house began as a two-story, single-pile brick building with attic windows and a central-passage plan. Around 1914 it was significantly enlarged by Henry Carter Stuart, governor of Virginia from 1914 to 1918, and who had considerable interests in banking, coal mines, and the Stuart Land and Cattle Company. The Stuarts raised cattle that were shipped to American and European markets.

Under Stuart, extensive landscaping was undertaken, including an impressive driveway approach. He also added two heavily quoined wings and a double-height semicircular Doric colonnade that encompasses the earlier one-story front porch. This colonnade, more monumental than domestic, might seem more appropriate as part of a commemorative memorial. On the interior, columns separate the central passage from the flanking drawing rooms, thereby giving the impression of one large reception area spanning the entire front of the house. The capacious kitchen and service areas were in a large north wing. Stuart's additions and reconfiguration of the dwelling are clear reflections not only of his wealth but also of the social obligations as a statewide politician.