Scuffle Hill was built on the shell of the 1905 brick building that was partially destroyed by fire. The house's name is thought to be from General Joseph Martin's first house in Henry County on Smith River. The house is dominated by a full-height Doric portico, which accords with the early-twentieth-century revival of classicism. The two semioctagonal bays at each end of the facade are part of the older house and look back to forms of the late nineteenth century. Three dormers with segmental heads are barely visible over the massive entablature ornamented with swags and reeding. The elaborate entrance is surmounted by a diminutive balcony with a cast-iron railing. Surviving from the earlier house are the granite water table and belt course. The various owners of the house read like a catalogue of Martinsville's economic history. The house was built by a past president of Liggett and Meyers Tobacco Company, for his daughter Frankie Stevens and her husband, tobacco manufacturer Pannill Rucker. Rives S. Brown, the developer of Forest Park and Druid Hills (HR21), owned it from 1920 until 1933 when he sold it to William L. Pannill, the founder of several knitting companies that expanded to make Martinsville the self-proclaimed “Sweatshirt Capital of the World.”