Adjacent to the Arnold Arboretum (JP4) rose a residential asylum for people with mental disorders who were not insane. J. Gardner Weld built a large house with a mansard roof and tower here in about 1874 for his mansion, its grand pretensions more suitable to an urban boulevard than a semirural setting. He also built a carriage barn with a mansard roof that survives behind the house. Weld died in 1876, shortly after the house was finished, and the estate was acquired for institutional uses. A wealthy sugar refiner named Seth Adams left a fund to establish an asylum for indigent people suffering from depression or nervous breakdown.

The Adams-Nervine Asylum trustees converted the house for administrative uses and held a competition to design a residence for women. J. Pickering Putnam, a European-trained Boston architect, won the 1880 competition and designed a first story of polychromatic stone and second story of wood with novelty siding. With its porches and elaborate architectural detailing, it was meant to resemble a private residence, a design supported by the then current mental health theory recommending for patients a homelike environment. The men's dormitory added in 1895 no longer stands, but the Colonial Revival–style director's house does.