The Irish-born architect James Hoban won the 1792 design competition, and his plan borrowed details from Leinster House in Dublin, such as the upper windows that alternate triangular and segmented pediments. The walls are Aquia Creek sandstone; once the building was finished, the porous stone was coated with a whitewash of lime, rice glue, casein and lead, and that colour eventually gave the house its name. Theodore Roosevelt made "The White House" the formal name in 1901.

The house has been rebuilt more than once. On 24 August 1814, during the War of 1812, British forces set it on fire in the Burning of Washington, leaving only the outer walls, and most of those had to come down; Hoban and Latrobe oversaw the reconstruction from 1815 until 1817. The curved South Portico followed in 1824 and the North Portico in 1830, its stone carved by Italian artisans who had come to work on the Capitol. By 1948 the structure was judged in danger of collapse, and under Harry S. Truman the interior was stripped out entirely, a steel frame was set inside the old walls and the rooms were rebuilt within it while the Trumans lived at Blair House across the street.

Seen from Pennsylvania Avenue, the north front is entered under a porte cochere with Ionic columns, with a fanlight and a carved floral festoon above the door; Ionic capitals on that portico carry a swag of roses between the volutes, matching the older carved roses over the entrance. The south side is more varied: a rusticated ground floor in the Palladian manner, a projecting bow of three bays at the centre, and a double staircase rising to a colonnaded loggia and the Truman Balcony. The grounds cover just over eighteen acres, laid out to a 1935 plan by Frederick Law Olmsted Jr., with the Rose Garden beside the West Colonnade.

The complex today centres on the Executive Residence, six stories with 132 rooms, linked by Jefferson's colonnades to the West Wing, where the Oval Office, the Cabinet Room and the press briefing room sit. The State Floor holds the rooms used for receptions: the East Room, the Green, Blue and Red Rooms, the State Dining Room and the Grand Staircase. Jacqueline Kennedy gave several of them a historical theme during the Kennedy years, such as French Empire for the Blue Room and the Federal style for the Green Room, and in 1961 Congress declared the house a museum so that its furniture and art could no longer be sold off. On the ground floor, the Diplomatic Reception Room keeps a "Vue de l'Amérique Nord" wallpaper designed by Zuber & Cie in 1834.

Thomas Jefferson allowed the public to tour the house, and tours have continued since his time except in wartime; after the September 11 attacks they were suspended for a time. Tour groups enter by a security building on the eastern side of the house. Since 1995 the stretch of Pennsylvania Avenue in front of the house has been closed to vehicle traffic.