Callaway spent his weekends for many years molding and firing bricks in a nearby kiln to build his house. A former slave from Franklin County, Callaway, who worked at the railroad station in Elliston, reportedly modeled his two-story, single-pile, center-passage residence on his former master's house. The small community of Elliston began in 1890 as Big Spring Depot, a stop on the Virginia and Tennessee Railroad. It was later named after a local landowner, Major William M. Ellis.