Inside its walls is a lived-in neighbourhood of lanes painted blue and white, with a monumental Almohad gate at one end and a formal Andalusian Garden at the other. It is part of the UNESCO World Heritage Site of Rabat, listed in 2012.

In 1150 or 1151 Abd al-Mu'min, the Almohad caliph, raised a citadel here, with a palace and a mosque, on the site of an earlier fortified ribat. In the early 17th century Moriscos expelled from Spain, many from the town of Hornachos, took over the kasbah and ran the Republic of Salé, a base for the corsairs known as the Salé Rovers, until the 'Alawi sultan Moulay Rashid brought them under his control in 1666. The name comes from the Udaya, an army tribe settled here by Moulay Ismail; it became attached to the kasbah only in the 19th century, after the rest of the tribe was expelled from Fez and joined them.

The main entrance is Bab Oudaia, also called Bab al-Kbir, which stands uphill overlooking the city and was built between 1195 and 1199 under Ya'qub al-Mansur. Its horseshoe arch is wrapped in a carved band of darj wa ktaf, a lattice of interlacing geometric forms, inside a frame of Kufic Qur'an verses, with arabesques and scallop shells in the corners. At the foot of the arch are serpentine forms that probably represent eels, a very rare motif in Moroccan architecture. The gate was more ceremonial than defensive, and inside, three chambers form a bent passage to the inner facade, decorated in much the same way.

From the gate, Rue Jamaa runs through the older upper part of the kasbah past the Old Mosque, its oldest surviving building, which dates from 1150; its minaret belongs to an 18th-century restoration, the work of an English corsair called Ahmed el-Inglizi. The street ends at a broad platform on the northeastern edge, built by the corsairs in the early 17th century for semaphore signalling, with views over the river mouth. Below it to the north stood a sqala, an artillery platform, and just upriver the 18th-century Tower of the Corsairs, which rose only 3 metres above the water.

The lower, southern part dates mostly from the Alaouite period. Here Moulay Ismail built a palace pavilion around a courtyard at the end of the 17th century, marked outside by a tower; it became a museum in 1915 and, after a restoration in 2006, the National Jewellery Museum, with rooms decorated in zellij and a former bathhouse. Next to it lies the Andalusian Garden, laid out between 1915 and 1918 on the old parade ground of the palace, a formal garden of bougainvillea and citrus trees inspired by those of al-Andalus. Close by is the café and terrace known as Café Maure.