Its three-storey bulk is anchored by large square and rectangular corner towers whose outer walls carry the geometric motifs and niches typical of the Berber oasis regions of southern Morocco. Overhanging balconies with windows look down on the entrance courtyard, and the upper windows are guarded by iron bars.

Oral tradition credits the Imzwarn, a powerful local family, with first building the kasbah in the 17th century, but it is the Glaoui family who took it over and enlarged it in the 19th century. At its height in the late 19th century it commanded the meeting point of several river valleys, among them the Draa and the Dadès, on the Saharan trade routes. Thami El Glaoui, a member of the family, was pasha of Marrakesh throughout the French colonial period.

After Glaoui rule ended, squatters moved in and the building fell into decay. It joined the Moroccan National Heritage List in 1954 and was sold by the Glaoui family to the Ouarzazate municipality in 1972. A small part was restored in the 1990s with UNESCO's help under CERKAS, the centre set up in the late 1980s to conserve the kasbahs of the south, and that section is open to visitors. Families still live in some of the unrestored parts.

Inside the restored section, several rooms keep their decoration and tataoui ceilings of woven reed. Among the unrestored buildings is a former qa'id's residence from the time of Qa'id Hammadi, who held office from 1882 to 1939, richly decorated with carved stucco, wall paintings and ornate arches that show the influence of Morocco's imperial cities. The kasbah has served as a film set, appearing in Gladiator and Prince of Persia.