Caswell went on to serve as an Agent of American Board of Commissioners for Foreign Missions (ABCFM) from 1838-1839, and was ordained in 1839. He was sent by the ABCFM with two other Middlebury alumni, Lyman Peet (Class of 1834), and Asa Hemenway (Class of 1835) as missionaries to in Siam. In Siam, Caswell became the tutor to the Crown Prince of Siam, later King Mongkut, immortalized in the book Anna and the King of Siam. Caswell married his colleague Asa Hemenway’s sister, Anna Turrill Hemenway, on April 14, 1839. Together they had four children: Francis Hemenway; Martha Melvina; George Shipman; and Stephen Jesse. Jesse Caswell died in Bangkok, Siam, on September 25, 1848.