The garden was already there by 1492, when an aqueduct with a 15 m span was built to irrigate it, and that aqueduct still carries water to the plants. The Croatian Academy of Sciences and Arts has run the arboretum since 1950.

The grounds combine two gardens of different ages. Around the family's 15th-century summer residence lies a park regarded as a monument of garden architecture, with a Renaissance garden laid out in 1502 and filled with exotic plants; a second park, at Drvarica, dates from the nineteenth century. Since 1962 Trsteno has been protected as a monument of landscape architecture, and around 255,000 square metres of ground fall under that protection. In the third and fourth seasons of Game of Thrones, the arboretum stood in for the palace gardens of the Red Keep in King's Landing.

The arboretum has been badly hurt twice. On 2 and 3 October 1991, during the Croatian War of Independence, gunboat and air attacks by the Yugoslav People's Army set it on fire, destroying much of it and partly damaging the summer residence and the oldest section; it was also looted. In 2000 a forest fire during a drought burned around 120,000 square metres. Two Oriental Planes on Trsteno's central square came through both disasters untouched: they are more than 500 years old, with trunks 5 m across, and are counted the pride of the arboretum.