The island was settled by the Dutch West India Company in 1624, and was acquired from its Native American owners in 1637. The British took possession of the island in 1664, and officially acquired it in 1698, changing the name from "Nut Island" to "Governors Island," for the "benefit and accommodation of His Majesty's Governors." A permanent home for the British governors of New York State was built there in 1702.
In April 1776, Continental troops under George Washington occupied and fortified Governors Island against British invasion, but the island and the city fell to the British in September of that same year. The British Royal Navy surrendered the island to Governor George Clinton of New York in 1783, leaving behind three forts and a number of other buildings and facilities.
From 1784 to 1794 the island was used for a racetrack and a summer resort. In 1794, new defenses were constructed on the island, and a garrison was posted there in 1797. In 1800 New York ceded the island to the US. Reconstruction of Fort Jay took place in 1806, and in 1811 Castle Williams was completed. In 1821 Governors Island became the headquarters of the Army of the East, and in 1833 the Ordinance Department of the Army selected the island as a major arsenal. In 1852 Governors Island changed from an artillery post to a recruiting depot, and during the Civil War it was the central Army recruiting station for the Eastern Seaboard. Castle Williams served as a prison camp, sometimes holding over 1,000 Confederate soldiers.
It was at Governors Island in the 1850s and 1860s that the "School of Practice" was held, training hundreds of young field musicians who enlisted in the US Army. From this School of Practice came the idea of a modern-day Field Music School for re-enactor musicians (see Don Hubbard's Legacy page for more background).
Governors Island and Fort, New York Harbour, April 1865