Barthold Schlesinger, a German Jew who married into a prominent Boston family, built Southwood, which survives on fifteen of its original twenty-eight acres. He had served as consul to the United States before deciding to remain in America and becoming successful in business. Schlesinger hired New York architect George Harney to design a Jacobean Revival brick mansion with stepped gables and towers and F. L. Olmsted to lay out the grounds. Then designing the park system for Boston, Olmsted had not yet moved to Brookline from New York, but was a close friend of one of Schlesinger's neighbors, architect H. H. Richardson (25 Cottage Street). Olmsted's plans preserved a character of intimacy that is unusual for such a large house, although the site was a difficult one for a large mansion house and outbuildings, since it had no vista.

In 1917 Francis Oakes acquired the bulk of the property and hired Bigelow and Wadsworth to remodel the interior of the house extensively, using Jacobean combinations of wood and cast stone. The new design also incorporated painted panels by Paul Cornoyer. A member of the Boston Parks Commission, Oakes added specimen trees to the estate. Having not made significant alterations to the property, the holy fathers who now occupy Southwood allow access to the grounds and some of the ground-floor rooms of the mansion.