The caves lie at Cape Spartel, 14 kilometres west of Tangier, and have two openings, one onto the sea and the other onto land. Tradition credits the Phoenicians with cutting the sea opening, and with a set of eye-shaped marks on the wall said to form a map of the surrounding area.

Only part of the complex is natural. Berber people quarried the walls for round stones to make millstones, and the circular cuts they left enlarged the cave a great deal. Long before that, around 6000 BC, Neolithic people lived here.

The caves carry their name from Greek and Roman myth. Heracles was said to have rested and slept in the cave before his 11th labour, fetching the golden apples from the garden of the Hesperides, which some ancient Greek writers placed nearby at Lixus. A Roman version has him smashing through the Atlas mountain instead of climbing it, joining the Atlantic to the Mediterranean and creating the Strait of Gibraltar. Local legend adds that the cave had no bottom and was one end of a tunnel running under the strait to St. Michael's Cave in Gibraltar, the route by which the Barbary macaques reached the Rock.

The caves were opened to the public in 1920, declared a National Heritage site in 1952 and given electric lighting in 1982. The ashes of the artist and writer Brion Gysin were scattered here in 1986. A renovation commissioned by King Mohammed VI was completed in October 2015, when the caves reopened.