Athenians call it Kallimarmaro, "beautiful marble". It is the stadium where the first modern Olympic Games opened in 1896, and the marble seats visitors see are the 1890s rebuilding of a Roman stadium that stood on the same spot.

From the 6th century BC a simple racecourse here served the Panathenaic Games, the festival held every four years in honour of Athena, with spectators sitting on the natural slopes. Around 330 BC the statesman Lykourgos built a proper stadium of poros stone, and in the mid-2nd century AD Herodes Atticus, an Athenian-born Roman senator, rebuilt it entirely in Pentelic marble with seats for 50,000. Owls, the bird of Athena, were carved in relief on the seating. After Theodosius I banned the old festivals in the late 4th century the stadium fell into ruin, its marble was carried off for other buildings, and wheat grew over the site.

Ernst Ziller excavated it in 1869–70, and in 1870 and 1875 it staged the Zappas Olympics, an early effort at bringing back the ancient games. For the 1896 Games the businessman George Averoff paid for a second refurbishment; the architect Anastasios Metaxas copied the dimensions and layout of the Roman stadium, arranging the tiers around the U-shaped track, and rebuilt it in Pentelic marble. A statue of Averoff, unveiled on 5 April 1896, still stands outside. The next day King George I opened the Games before an estimated 60,000 spectators, and the stadium staged athletics, gymnastics, weightlifting and wrestling.

The stadium went on to host the 1906 Intercalated Games and, in 2004, the Olympic archery competition and the finish of both marathons. It is the finishing point of the Athens Classic Marathon and the place where Greece hands over the Olympic flame to the next host nation. Concerts and ceremonies use it too, from a staging of Aida in 1916 to the welcome given to the national football team after its 2004 victory. Its design shaped stadiums abroad: Harvard Stadium, built in 1903, was modelled on it.