The compound's 16 historic buildings went up between 1841 and 1925, and it is a rare survivor: most of the city's colonial architecture had been cleared for development before Hong Kong was handed back to China in 1997. Its name is the old colloquial name for the compound.

The first building on the site was the Magistrate's House, put up with its jail blocks in 1841. By 1862 the prison population had risen to 650, and the government developed the surrounding land, so that over the following century the compound grew through repeated extensions and rebuilding. In 1899 the Central Prison took the name Victoria Prison, and it stayed in use until 2006.

In 2008 the Hong Kong Jockey Club became the government's partner in conserving the site and give it a new use; work started in 2011 and ran for eight years, interrupted in 2016 when a wall and roof collapsed. The 16 heritage buildings were restored, and two new buildings were added with designs drawing on the old brickwork. The compound opened to the public on 29 May 2018 with the exhibition "100 Faces of Tai Kwun", and the remaining parts followed in three phases. In 2019 the project won an award of excellence from UNESCO's Asia-Pacific heritage conservation awards.

The restored compound sets heritage and contemporary architecture side by side, so that the long corridors of the old police headquarters stand alongside the two new buildings and their brick-inspired design.