Brookside was built as Colonel Nathaniel Burwell's summer house before the construction (1792–1800) of his main residence, Carter Hall, a stone house nearby that is not visible from the road. The weatherboarded Brookside was originally composed of the two left bays that included an interior-end stone chimney. The three right bays, added in the mid-nineteenth century, include a large exterior-end brick chimney. Brookside was remodeled in the late nineteenth century when two-over-two sash windows, a bay window, and gable-roofed dormers were added.