Its name joins the Balinese words for edge (ulu) and rock (watu), and local folklore holds that the rock is part of the goddess Dewi Danu's boat turned to stone. Its worship is directed to Sang Hyang Widhi Wasa, manifested as Rudra, and it is unusual in being both a sea temple and one of Bali's nine directional temples.

A small temple may have stood here before the Javanese sage Empu Kuturan enlarged it in the 11th century. Around the start of the 16th century another sage from East Java, Dang Hyang Nirartha, is credited with introducing its padmasana shrines, and he is said to have reached moksha on this spot; the event, called ngeluhur, "to go up", gave the temple its epithet Luhur, "height". Customary law of the Pecatu community protects the forest around the temple as part of its sacred space.

Long-tailed macaques live in the temple grounds and are known for grabbing visitors' belongings, such as sunglasses, cameras and flip-flops, and handing them back in exchange for food; researchers studying them have found that the young learn this bartering from their elders. On the cliff-side, a Kecak dance based on the Ramayana is performed in the open air with the sunset behind it.