Located at a large bend on the James River, this small town served as the original county seat of Albemarle County, from 1744 to 1762. The community prospered in the late eighteenth and nineteenth centuries as a major trading center. Its ferry served the various turnpikes that passed through, and the James River and Kanawha Canal once passed through the town. When the Richmond and Allegheny Railroad (now CSX) arrived in the 1880s, the town was already in decline, for in 1865 Union cavalry pillaged the place. The town's location, partially on the river's floodplain, has resulted in flood damage, and a series of fires have destroyed much of the original fabric. However, a few buildings of interest remain. The town plan is L-shaped. Main Street parallels the river. Perpendicular to Main Street, Valley Road (Virginia 20) climbs the hill. Parallel to Valley Road is Harrison Street.

On Main Street, all on the north side, are the James Turner Barclay House (c. 1825; Main Street), an elegant three-bay brick town house with fine corbeled cornice and Federal details around the door and windows. Next door is the Scottsville Museum (Disciples of Christ Church) (1846; Main Street; open to the public), a brick building with double entrances, in a Greco-Roman classical revival style that illustrates lingering Jeffersonian influence. The west end of Main Street is terminated by the Scottsville Apartments (Scottsville School) (1939, Department of Education, School Buildings Service; Main Street). The building has a large Colonial Revival portico that masks the standard single-story school plan. On Harrison Street is St. John's Episcopal Church (1875; Harrison Street), a board-and-batten structure with pointed-arched openings that shows the influence of Richard Upjohn and of Upjohn's Rural Architecture (1852). The interior has been altered. Valley Street has a variety of structures, some worthy of note. Two miles north on Virginia 20 is the Scottsville Elementary School (c. 1960; 1979, addition, VMDO), which is a noteworthy rejuvenation of an older school building with a new entrance and a large, diagonally cut cylinder serving as the library, rather Corbusian in form. Clad in glazed red brick, the building is an eye-catching civic presence.