At the heart of this private boys' school is The Residence, or the William Madison House (1793, Thomas Jefferson; later additions), the home of President James Madison's brother, a local politician and owner of the site in the 1790s. Correspondence between Thomas Jefferson and James Madison discusses a house that apparently was constructed for the plantation at Woodberry Forest. No drawings have been identified. The symmetrical single-story villa with a Doric portico is Jeffersonian in appearance, although executed in wood. In the late nineteenth century Robert Walker purchased the house, greatly enlarged and remodeled it, and in 1884 established a boys' preparatory school bearing the same name. After serving as the headmaster's house, the William Madison House is now used as the guest house for the school. Also on the grounds is the Walker Building (1899, John Minor Botts), which for years served as the school's main building, a tall and awkward Georgian Revival structure built from bricks fired in a kiln on the property. The Georgian Revival mode has been followed for most of the school's later buildings, including the impressive Dick Gymnasium (1939, Turner and Fisher), designed by architects from nearby Culpeper, and for the White Library (1958, White and Wilkerson), by architects from Richmond. More contemporary, with shed roofs and obviously veneered brick walls, but fitting in well, is the Dowd-Finch Dormitory (1978, VMDO), designed by Robert Vickery of Charlottesville.