To the west of the present day Municipal Building was a structure as important to the life of Middlebury as was the church at the other end of the street — the Academy, also called the East College.

Middlebury’s children were given a rough and rudimentary education in “common schools” usually in people’s homes; but Painter and others wanted more, a school that would carry on beyond the fundamentals. With Storrs, who had experience in secondary education, Dr. Matthews, and lawyers Daniel Chipman and Samuel Miller, Painter formed the Addison County Grammar School Corporation, chartered by the legislature in November of 1797. Storrs and his neighbors assembled and donated a sizeable school lot and commons. Four thousand dollars were raised by public subscription for the 1798 construction of the Academy building.

The wooden building was forty by eighty feet and three stories high, the largest structure yet built in town. It had an impressive number of windows (glass was very expensive), equally important front and rear entrances, and a crowning cupola. The first floor held classrooms, library, and laboratory; the second, dormitories; and the third, additional dormitories and a central chapel.

Upon the founding of the College in 1800 the building housed both College and Grammar School until 1805, when the latter was moved into the then vacant building of the Female Seminary on Seymour Street. The Grammar School moved back in 1844 and in the 1850s merged with Middlebury School District no. 4.

In 1867 the Academy Building was superseded by a new building located just slightly to the west, a fashionable Italianate structure designed by J.J.R. Randall of Rutland. Of brick with brownstone details, the building had heavy, bracket-supported cornices, a gable centered on each facade, and an elaborate mansarded cupola. A fire on Easter in 1904 gutted the school, but it was rebuilt with only slight changes to the roofline (and the elimination of the cupola), and it long served the town as the College Street Graded School. In 1984 it was acquired and renovated by Middlebury College, at which time it was renamed Twilight Hall in honor of Alexander Lucius Twilight, Middlebury Class of 1823, the first African-American citizen to graduate from an American college.