James was the only son of William James, an American railroad magnate who moved to England and married Evelyn Forbes, a Scots socialite, who was reputedly fathered by the Prince of Wales (later Edward VII). He had four older sisters: Audrey, Millicent, Xandra, and Silvia.
James was educated briefly at Eton, and then at Le Rosey in Switzerland, then at Christ Church, Oxford, where he was a contemporary of Evelyn Waugh and Harold Acton. In 1912 he inherited the 8,000-acre (32 km?) West Dean Estate in Sussex, on the death of his father.
James's first sponsorship of note was in publishing John Betjeman's first book of poems when at Oxford. He worked with Brian Howard on the Glass Omnibus. After Oxford, James had a brief career as a trainee diplomat at the embassy in Rome. He was asked to send a coded message to London that the Italians had laid the keels for three destroyers, but got the code wrong; the message said "300 destroyers". Shortly after this he was sent "on indefinite leave".
James was gay, and his marriage in the early 1930s to the ballerina Tilly Losch, an Austrian dancer choreographer, actress and painter, only served to confuse and complicate an already tortured existence.
He had several productions created expressly for her, the most notable of which was Les Ballets 1933, which included Kurt Weill, Lotte Lenya and George Balanchine. James divorced Losch in 1934, accusing her of adultery with Prince Serge Obolensky, an American hotel executive; her countersuit, in which she made it clear that James was homosexual, failed.
After the divorce, James joined a social set in England which included the composer Lord Berners and the Mitford sisters.
In 1940, James showed up in Taos, New Mexico, United States, as a guest of Mabel Dodge Luhan, where he was known for his amusing, clever eccentricity and effeminate manner. In Taos, he met the Hon. Dorothy Brett, an impoverished British aristocrat and painter, who in 1941 sold him nine paintings for $580. He later invited the 70 year-old Brett (as she was known) to return to England and reside at West Dean, but she declined.
Los Pozas ("the Pools") is a sculpture garden built by James, more than 2,000 feet (610 m) above sea level, in a tropical rain forest in the mountains of Mexico. It includes more than 80 acres of natural waterfalls and pools interlaced with towering Surrealist sculptures in concrete.
Las Pozas is near the village of Xilitla, San Luis Potosí, a seven-hour drive north of Mexico City. In the early 1940s, James went to Los Angeles, and then decided that he "wanted a Garden of Eden set up . . . and I saw that Mexico was far more romantic" and had "far more room than there is in crowded Southern California."
Between 1949 and 1984, James built thirty-six concrete follies - palaces, temples and pagodas, including the House on Three Floors Which Will in Fact Have Five or Four or Six, the House with a Roof like a Whale, and the Staircase to Heaven.
In 1964, James gave his English estate at West Dean to a charitable trust. The Edward James Foundation comprises West Dean College, a centre for the preservation of traditional arts and crafts, through short courses and full-time Diplomas and MAs. One of only two professional Tapestry Weaving studios in the UK, an Art Gallery are all housed on a 6,400-acre (26 km2) estate which is open to the public through the West Dean Gardens.
West Dean College is part of the Edward James Foundation set up in 1971 in response to James's vision of establishing "an educational foundation where creative talents can be discovered and developed, and where one can spread culture through the teaching of crafts and the preservation of knowledge that might otherwise be destroyed or forgotten".
Edward James is buried in the Arboretum at West Dean, with the simple inscription Edward James 1907 - 1984 Poet.