Charles Washington, who had owned the land earlier, built two wings of Happy Retreat and connected them by a breezeway pending the time when he might erect a central block. Washington died in 1799 before completing the house, and another owner, Isaac A. Douglass, added the central section soon after his purchase in 1837. Douglass named the house Mordington after an ancestral Scottish home. In the 1940s, Raymond J. Funkhouser acquired and restored this and other Washington family houses in Jefferson County (see Charles Town Vicinity).

Happy Retreat is a good example of the transitional 1830s Federal–Greek Revival style so often found in the lower Valley of Virginia. Its broad, three-bay brick facade, painted white, is capped with a pedimented gable with a lunette centering the pediment. The plan is also typical of the type: a transverse entry and stair hall occupies the entire front of the main block, with two large rooms behind. Narrow, barely noticeable hyphens connect the central block to Washington's earlier two-story, one-bay wings, which are fronted with later stepped gable ends. While its original builder would not recognize the house from the exterior, portions of Happy Retreat constitute one of the area's earliest Washington family houses. It can be said with certainty that George Washington slept here, although it might be hard to pinpoint just where.