Like so many of Bristol's finest houses, “The House with the Eagles” is a taut and simple Federal box that has been animated with superb architectural woodwork. Originally built by the lawyer John Howe, a descendent of the influential DeWolf family, it passed to ship captain Benjamin Churchill in 1822. According to local legend, Churchill gave the house its name by having the sailors of his ship, the Yankee, carve four American eagles, which he placed at the corners of the Chippendale balustrade that crowned his roof.

Churchill's tenure was brief, and in 1825 the house passed to Byron Diman, a powerful merchant with interests in whaling, banking, and the local cotton mills who served as governor of Rhode Island in 1846–1847. While Diman repeatedly altered and expanded his house, he maintained its essential Federal character, the hallmark of Bristol's conservative elite.