Caleb Fellows bequeathed money for the construction of an athenaeum library in 1852, but the building was not begun until twenty years later on a lot bounded by Dudley and Bartlett streets. Construction had reached the second story when the Metropolitan Railroad acquired the land for a stable, requiring the Fellowes Athenaeum to be dismantled and reconstructed at the corner of Millmont Street and Lambert Avenue. Nathaniel J. Bradlee, who lived nearby on Highland Avenue, had designed the T-shaped building for a level site, the front of the building, with its Italian Renaissance facade of brick and Nova Scotia sandstone with flanking pavilions, clearly intended to mask the two-story rear section containing the stacks for the books.

From the beginning, the building served a combined function of athenaeum and public library. The athenaeum acquired a book collection considered to be suitable for a higher class than those for the public library, described at the time as being of the “more popular order.” The City of Boston annexed Roxbury before the athenaeum was finished, making the Fellowes Athenaeum a branch of the Boston Public Library (BB42). The Refuge Church of Christ now uses the building.