Laburnum, the enormous red brick Beaux-Arts mansion to which the hospital is attached, was built by Joseph Bryan, a prominent Richmond newspaper publisher, after an earlier house on the site burned. Laburnum originally included all the land between Brook and Hermitage roads and between Westwood and Laburnum avenues. The architects were from Philadelphia and freely mixed Virginia and Philadelphia elements. The major exterior feature was the gigantic hexastyle Corinthian portico, which, while recalling the popular vision of the Old South, gave the house an institutional feeling. Though Bryan died within a year of moving into the new house, his widow and one of their sons and his family continued to live in the sumptuously appointed house, entertaining such luminaries as Winston Churchill, Franklin Delano Roosevelt (then governor of New York), and Lady Nancy Astor. In 1949, the Bryans donated the house and the remaining part of the estate to Richmond Memorial Hospital, as a memorial to Richmonders who had died during World War II.

The construction of the seven-story hospital left the house mostly intact, adapting some of it for use as offices. The main hospital is a simple, modern building with a graceful curved canopy indicating the entrance. The lobby is a small, sleek space. The chapel is housed in a concrete extension to the brick body of the hospital above the entrance. It is ethereal, very compact in floor space, but five stories tall. Unfortunately, tall green Italian marble tablets engraved with the names of Richmond's World War II dead have been removed and replaced with an oversized American flag, which seems garish in the tight space. A medical office building was later added, and the hospital has grown with several additions. In 1965 Sheltering Arms Hospital moved from East Clay Street downtown to the Palmyra Avenue side of the site. Most of the Richmond Memorial's functions moved to a new location in Hanover County in 1998; current plans call for reuse of this building as a nursing home.