Gardner is the son of Deane McKay, an ad executive, and Catherine. He was raised in New York and Paris. He attended Cornell University two years but left when his father died. He worked in advertising for 6 months but found that he could not stand it. At age 20, he became a sculptor and had a piece displayed in the Museum of Modern Art. In 1959, he was spotted by a Hollywood producer who convinced him to join the cast of TV's Adventures in Paradise (1959). When it was over, he decided he really did not like the celebrity spotlight and proceeded to roam the world. He hiked in the Amazon, rode camels in Egypt, and crewed on Caribbean yachts. Finally, in 1980, he met Madeleine Madigan, who became his wife. The life-long bachelor settled down and decided to become a writer.