Set apart at the edge of Manchester village this imposing structure was built for Vermont's first privately endowed academy. Made possible by a legacy of 1828 by resident Joseph Burr, the academy was a residential institution that prepared young men for the ministry. The thirteen-bay-wide, three-story building was conceived as a match to the brick and marble linteled, gable-fronted headmaster's house next door. However, after a disastrous firing of bricks for the school, the builders switched to square-cut local limestone. The building is essentially late Federal in character, with, originally, six-over-six windows (glazing altered in 1873), and a semi elliptical keystone-arched central doorway with fan and sidelights. Perhaps in response to the original theological purposes of the seminary, it has a square, clapboarded, Gothic-styled belfry with pointed-arched openings and a balustrade with tall finials. Old photographs reveal that the balustrade's present form is a simplified reconstruction of the original, which had eight elaborately cusped finials similar to those atop the tower of St. James Episcopal Church in nearby Arlington (BE10).

In 1853, Josiah Burton, a member of the original board of the school, left money to endow a joint institution for girls. The renamed Burr and Burton Seminary was the first coeducational secondary school in Vermont. After the state mandated a free high school education for all students in 1906, Burr and Burton assumed this public role, but as a private institution to which the surrounding towns paid tuition for their students.