A fitting contrast with the Swann-Daingerfield House across the street and an earlier era's symbol of prosperity is this very substantial three-story brick Federal house. Originally built in 1797 by James Patton, it was purchased in 1819 by William Fowle, who owned a substantial portion of the Alexandria Canal Company and a local bank. His enlargements of c. 1820 have been attributed to Charles Bulfinch because of the Palladian window and the fanlighted entrance. Other evidence, though, points to these features as part of a later, Federal–Colonial Revival remodeling.