Diagonally across from each other, the town hall and public library anchor the civic core and symbolize the movement of population from Canton Center to South Canton at the end of the nineteenth century. Memorial Hall (1878–1879, Stephen Earle, 801 Washington Street) was the town's first purpose-built municipal building. Elijah Morse, a Civil War veteran who made his fortune locally with Rising Sun stove polish, donated the site and an elaborate drinking fountain. The three-story red brick and brownstone Gothic Revival structure houses town offices and an auditorium designed to seat 944.

The origins of the Canton Public Library included a series of subscription and social libraries beginning in 1766 that became a free public collection in 1875 and moved into Memorial Hall when it opened. When additional space was required, Augustus Hemenway donated funds for the construction of the current library (786 Washington Street), completed in 1902. Winslow and Bigelow designed the red brick and limestone Classical Revival building with hipped copper roof and dome; in 1963 a lower expansion was constructed to the south.