A $10,000 lottery was authorized by the General Assembly in 1825 to fund a school building, and, in February 1826, Henry Little was awarded the contract for $5,000. He was to build a two-story edifice “of the best materials and in a plain but substantial manner.” A trustee visited Philadelphia in 1827 to study the latest methods of stuccoing masonry. Called by children the “Yellow Prison” for its exterior paint color, the plain, Federal-style edifice with cupola served as a school until 1929, after which it became dilapidated. In the 1940s, a post office was nearly erected on the lawn in front of it, but a historical society mobilized to prevent this, and court battles ensued. Starting in 1960, it was owned by the town and functioned as a town hall. Rapid growth of Middletown led the mayor to announce in 2004 that he would sell the building and build a larger town hall elsewhere. The old Academy now houses a chamber of commerce and the historical society.