Robert Pim Butchart, a Portland cement maker from Ontario, came west with his wife Jennie for the limestone, and in 1904 they built their home beside the quarry. When the stone ran out in 1909, Jennie Butchart began turning the pit into a garden; the result, the Sunken Garden, was finished in 1921. The couple called their house "Benvenuto", Italian for "welcome", and opened the gardens to visitors, and in 2004 the site was named a National Historic Site of Canada.

The Sunken Garden is the heart of a visit, with greenery and colourful flowers filling what was once the bottom of the quarry. Around it the estate grew garden by garden. An Italian garden replaced the family tennis courts in 1926, and in 1929 the vegetable plot became a large rose garden designed by Butler Sturtevant of Seattle, which holds 2,500 plants of 280 varieties. Jennie Butchart also commissioned a Japanese garden from Isaburo Kishida, a designer from Yokohama who worked in Victoria between 1907 and 1912. The gardens give their range of bedding plants as 900 varieties.

Robert Butchart's own passion was birds; he kept ducks on the Star Pond and peacocks on the lawn and built elaborate birdhouses for the grounds. Bronze statues are scattered through the gardens, among them "Tacca", a wild boar cast in Florence after a 1620 original by Pietro Tacca, whose snout is polished bright by visitors rubbing it for luck, and a fountain of three sturgeon near the Japanese garden. The Ross Fountain, which changes its patterns, was set in the lower reservoir in 1964, and two totem poles 30 feet tall went up in 2004 for the centenary.

The gardens were lit at night for the first time in 1953, and in the holiday season lights and decorations are strung through them. Since 1977, summer evenings have brought firework displays set to show tunes, and July and August see local music from jazz to classical. The Children's Pavilion, opened in 2009, houses the Rose Carousel, whose thirty hand-carved basswood animals range from bears and horses to ostriches and zebras. In summer a small electric boat, the Jennie B, takes passengers along the nearby shoreline.