The older one, on Paulus Potterstraat, is a Dutch Neo-Renaissance building of pale stone and red brick by Adriaan Willem Weissman, opened on 14 September 1895. Joined to it is the 21st-century wing by Benthem Crouwel Architects, nicknamed "The Bathtub", which holds the present entrance and opened with the renovated museum on 23 September 2012.

The museum was set up by the city and private citizens, and much of the first building was paid for by Sophia Adriana de Bruyn, who gave her collection of art and antiques to Amsterdam. Its character as a modern art museum came above all from Willem Sandberg, curator from 1938 and director from 1945. In 1938 he had the gallery walls covered in white jute, an early version of the neutral "white cube", and under him the museum began collecting applied art, prints and drawings and, from 1956, photography. Edy de Wilde, director from 1963 to 1985, started its collection of American contemporary art, and from the early 1970s the museum bought video art, which by 2024 had grown to around 900 works and installations, with Bruce Nauman, Bill Viola and Nam June Paik among the artists.

The collection now numbers almost 90,000 objects gathered since 1874, with particular strength in De Stijl, Bauhaus, CoBrA and Pop Art and an extensive group of paintings and drawings by Kazimir Malevich. Late 19th-century works by Paul Cézanne and Vincent van Gogh lead into the 20th century, with artists such as Henri Matisse, Wassily Kandinsky, Piet Mondrian, Marc Chagall, Ernst Ludwig Kirchner, Jackson Pollock, Karel Appel, Willem de Kooning, Andy Warhol, Lucio Fontana, Marlene Dumas and Gilbert & George. Design and photography have their own place, including Dutch and international avant-garde photographers from between the wars such as Man Ray and László Moholy-Nagy.

Highlights of the collection hang in the old building in presentations that change over time, while the new wing, with 8,000 square metres of exhibition space across the museum, holds temporary exhibitions on its upper level and in the basement. The wing also contains the shop, restaurant, library and an auditorium, and its glass entrance opens directly onto Museum Square. Graphic design is part of the museum's own story too: from 1963 Wim Crouwel built its posters and catalogues on a strict grid set in the Univers typeface, and the logo introduced at the 2012 reopening is a large S made from the letters of the museum's name.