He was born in Homer, New York, briefly attended Geneva (now Hobart) College, and was educated at Yale University. At Yale he was a classmate of Daniel Coit Gilman, who would later serve as first president of Johns Hopkins University; the two were members of the Skull and Bones secret society, and would remain close friends. After his graduation from Yale in 1853, he spent three years studying in France and Germany, before returning to the United States as a professor of history and English literature at the University of Michigan.

He served as attaché in St. Petersburg (1854-55). Later he sat (1864-67) in the New York state senate and was chairman of the education committee, which dealt with the founding of a land-grant college. With the financial aid of a fellow senator, Western Union tycoon Ezra Cornell, in 1865, he founded Cornell University on Cornell's estate in Ithaca, New York, with White as its first president. His farsighted leadership set the university on the path to becoming one of the world's elite educational institutions, with particular excellence in agricultural research and engineering.

As first president (1867-85), White expanded the institution to teach not only agriculture and mechanical arts but also other fields of knowledge. He was one of the first educators to use the system of free elective studies. As Cornell was nonsectarian, the charge of "godlessness" was made against it. White, a practicing Episcopalian, maintained that freedom was beneficial to religion and wrote his History of the Warfare of Science with Theology in Christendom (1896) and Seven Great Statesmen in the Warfare of Humanity with Unreason (1910) to develop his concept of free inquiry.

After 14 years at Cornell, he resigned to serve as the United States' minister to Germany (1879-1881) and Russia (1892-1894), and as ambassador to Germany (1897-1902). He was chairman of the American delegation to the First Hague Conference (1899). He persuaded Andrew Carnegie to build the Palace of Justice to house the Hague Tribunal.

While serving in Russia, White - a noted bibliophile - made the acquaintance of author Leo Tolstoy. Tolstoy's fascination with Mormonism sparked a similar interest in White, who, like most educated Americans at the time, had previously regarded the Latter-Day Saints as a dangerous, deviant cult. Upon his return to the United States, White took advantage of Cornell's proximity to the original Mormon heartland near Rochester to amass a collection of LDS memorabilia (including many original copies of the Book of Mormon) unmatched by any other institution save the church itself and its university, Brigham Young University.

White died in Ithaca, at age 86.