With two completely different formal elevations constructed of wood framed into a partially brick structure, the Isaac Royall House remains one of the most architecturally unusual Georgian-era houses in New England. John Usher erected the late-seventeenth-century core, a two-story single pile gable house whose profile is visible in the brick end walls. Issac Royall enlarged it in the 1730s by adding a saltbox extension on the west side and a three-story facade on the east. This clap-boarded elevation features aproned windows, a modillioned cornice, and quoining.

Isaac Royall Jr. inherited the house in 1739 and a decade later raised the brick rear to a full three stories, more than doubling the interior volume. Rusticated to simulate ashlar masonry, the new west facade features a more traditional Georgian arrangement with full-height corner pilasters and pediments over the first-and second-floor windows. He probably added much of the very elaborate interior woodwork, including a stair hall modeled on the John Hancock mansion (1737) in Boston. The property also includes a large two-story half-wood and half-brick outbuilding that was used in the past for slave quarters, a unique survival in New England. Since 1905 a nonprofit group has maintained the house and opened it to the public.