Son of Jesse and Beulah (Haskin) Caswell, Jesse Caswell (1809-1848) was born in Middletown, Vermont, on April 17, 1809.  He entered Middlebury as a member of the Class of 1832 in 1828.  He taught in Potsdam, New York, from 1832-1834, before studying at the Andover Theological Seminary from 1834-1835, and the Lane Theological Seminary in 1837.  He served as city missionary in Cincinnati, Ohio, from 1837-1838.

Caswell went on to serve as an Agent of American Board of Commissioners for Foreign Missions (ABCFM) from 1838-1839, and was ordained in 1839.  He was sent by the ABCFM with two other Middlebury alumni, Lyman Peet (Class of 1834), and Asa Hemenway (Class of 1835) as missionaries to in Siam. In Siam, Caswell became the tutor to the Crown Prince of Siam, later King Mongkut, immortalized in the book Anna and the King of Siam.  Caswell married his colleague Asa Hemenway’s sister, Anna Turrill Hemenway, on April 14, 1839.  Together they had four children: Francis Hemenway; Martha Melvina; George Shipman; and Stephen Jesse. Jesse Caswell died in Bangkok, Siam, on September 25, 1848.