Founded in 1862 as the city's museum, it grew from art that Haarlem had collected since the 16th century, and it moved into the almshouse in 1913. Its holdings run to some 17,000 objects, about half of them modern art, including about 800 Dutch paintings of the 17th century.

The almshouse was founded in 1609 for elderly men: its little houses were grouped around a courtyard, two men lived in each, and residents had to be at least 60 and unmarried. A statue of a man holding a poor box, recalling the weekly collections the residents had to make, stands in the entrance hall. Much of the complex was later altered, but the regents' rooms have been fitted with interiors rescued from demolished Haarlem houses, and the corridors are lined with salvaged Dutch tiles, Haarlem silver from local churches, stained glass and 17th-century clocks, chairs and chests.

Frans Hals was born in Antwerp around 1582 and grew up in Haarlem, where he joined the painters' guild of Sint-Lucas in 1610. His first large group portrait, of the St George's militia, followed in 1616, and he went on to paint four more militia pieces; all five hang in the museum. Late in life, in his eighties, he painted the regents of this very almshouse, and the loose brushwork of these portraits later drew young impressionists to Haarlem; Manet and Van Gogh also came to see his group portraits.

Around Hals the museum shows other Haarlem masters such as Judith Leyster, Cornelis van Haarlem, Jacob van Ruisdael and Hendrick Goltzius, as well as Jan van Scorel's group portrait of a dozen Haarlem pilgrims who had travelled to Jerusalem and Maarten van Heemskerck's St Luke Painting the Virgin. The collection continues from 1880 to contemporary work, with artists including Jan Toorop, Isaac Israëls, Charley Toorop and Dana Lixenberg. Families can try a scavenger hunt, dress up as a civic guard, make their own group portrait or look into the doll's house.