Squire William Whittemore, who built this imposing mansion, helped form a company to manufacture card clothing using his brother Amos Whittemore's invention. Patented in 1788 and extended by Congress in both Amos's and William's names in 1809, the invention spurred widespread development of carding mills, one of the first steps in the region's industrial revolution of textile production. The card clothing factory was once located just beyond the Arlington cemetery in view of this house on the town common. The three-story, four square, brick-ender dwelling with a handsome cupola above the low hipped roof originally faced east next to Massachusetts Avenue and was approached by a long curving drive that led to a carriage house beyond. The facade featured an elliptical fanlight above the door flanked by sidelights, entered from a later porch. After Whittemore's death in 1842, it became a female boarding school known as Whittemore Hall. It was later owned by Faneuil Hall (GC4) meat-and-poultry merchant Nathan Robbins, whose heirs' bequest for a new library led to it being moved to provide a site for that building. The house itself was given to the town about 1920, when two Robbins sisters decided to live in Switzerland.

Nearby, a small group of millworkers' houses (NRD) still lines Central Street. Vernacular wooden dwellings for Nicholas Blaisdell (1849, 910 Central Street), J. C. Blanchard (1846, 616 Central Street), and Edward Storer (1846, 618–620 Central Street) recall this workers' neighborhood.