The first limestone quarry was established at Stone City in 1852; by the late 1870s, the limestone industry was supporting a thriving community.
4 Later, with the increased use of portland cement for reinforced concrete walls and concrete block, the economic base of the community all but disappeared. In 1932 the community experienced a brief reawakening as a summer art colony under the direction of Grant Wood. Many of Stone City's buildings have disappeared, including John Aloysius Green's 1883 Second Empire-style mansion (it burned down in 1963) and the large three-and-a-half-story Columbia Hall (torn down for its stone in 1936), yet a number of structures remain that have architectural merit.

Among these is the two-story rusticated ashlar block Henry Dearborn and Sons Building of 1897, located on the south side of Main Street (route E28), just before the Wapsipinicon River. Across the river, south of Main Street and the former railroad tracks, is Saint Joseph's Catholic Church designed in 1913 by the Dubuque architect Guido Beck. It goes without saying that the church is of native stone (rusticated ashlar); in style it is a version of the Gothic mode featuring a corner entrance tower, a style often built throughout the United States in the late nineteenth and early twentieth centuries. The gabled front of the church is corbeled, the sidewalls are buttressed, and the three-story tower is crenellated. The most impressive and handsome building still to be found within Stone City is the Old Stone Barn of 1888. This three-story structure stands west of the church and south of Main Street. It is of stone and measures 124 feet long, 50 feet wide, and 40 feet high. The two narrow walls are buttressed, and secondary buttresses continue along each of the long sidewalls. A low-pitched hipped roof with very little overhang sits lightly atop the heavy masonry walls. A two-story arched entrance occupies one end of the building, its large scale played off against the three rows of small double-hung windows.