The rough-faced, brown sandstone construction of this house makes it unusual for the area. Drawing inspiration from Queen Anne and Richardsonian Romanesque, the two-and-a-half-story hipped-roof building has gabled projections and pedimented dormers and rests on a raised limestone basement. It was built for Ayers, who served as attorney general of Virginia from 1886 to 1890, was director of the first coal mining company in Wise County, and was instrumental in bringing rail transportation to Big Stone Gap. The house was one of the first buildings to reflect a sense of permanence and solidity in a new town of mostly simple frame structures. Although a wraparound porch and a bay window were removed in the 1940s, the house retains much of its original character, especially on the interior. An L-shaped hallway has a paneled ceiling with cove cornice, ceiling pendants, a staircase of red oak with a massive square newel, paneled wainscots, and door and window wooden surrounds with corner blocks. The house and its painted brick carriage house is encircled by a low wall of large sandstone blocks.

C. Bascom Slemp, a U.S. congressman and former private secretary to President Calvin Coolidge, acquired the house in 1929, but it did not serve as his primary residence. Instead it housed his large collection of historical and political artifacts that eventually became the nucleus of the Southwest Virginia Museum's collection. Occupying the house since 1948, this state-owned museum is devoted to depicting life in this region of Virginia.