The Amber Temple was built high in the Balinok Mountains by a group called the Azure Paragon who sought to lock away dangerous knowledge. These Wizards believed that the Dark Powers of the Plane of Shadow could be studied without being unleashed, and they carved the temple into the stone cliffs as a hidden vault. Using incredibly powerful methods lost during the Pre-Unified Age they sealed the essences of these powerful, malevolent entities within solid blocks of amber, referred to as "sarcophagi."
The temple was focused around a main chamber, marked by a massive granite statue representing the God of Secrets. Sheets of amber lined the main hall and hallways, and most of its chambers were carved from black stone and lined with statues. Hidden doors and long hallways connected the vaults containing the sarcophagi, scholarly lecterns, wizards chambers, and a great library that contained many arcane secrets, covered in illusory magic known only to members of the Paragon.
Amber Golems stood guard near the deeper levels, invisibly watching over an armory and the temple's treasury. However, the main guardian of the temple was bound there through an Infernal contract; in a display of power, the wizards had employed the services of an Arcanaloth named Neferon. Neferon was forbidden to attack a member of the Azure Paragon but was instructed to kill any visitors who sought to release the Dark Powers, and the Fiend performed his services with aplomb.
At some point in time, the Amber Temple was abandoned, and the wizards of the Paragon entirely disappeared. For thousands of years, the temple itself remained cold and untouched by time. The sarcophagi chambers remained undisturbed under Neferon's watch, until a few significant visitors came up from the Zarovich Valley below.
First, Count Strahd von Zarovich came to the temple, having gained interest regarding a mythical library high in Mount Ghakis. Not bound to fight someone who had not come to seek a Dark Power, Neferon allowed Strahd access to the Temple, accompanying the Human warlord and teaching him about magics. This, however, was ultimately an error: Strahd opened a vault containing the amber sarcophogus of Vampyr, and unknown to Neferon at the time, made contact with the Dark Power. Though Strahd's witnessing of the sarcophogus would eventually result in the Wedding That Never Was, and the ousting of the valley from the Prime Material, he successfully convinced Neferon that he was uninterested in the sarcophagi.
Years later, Strahd would visit Neferon as a powerful Vampire Spellcaster, having invoked a pact with "death" to keep him eternally young. He had learned about the arcanaloth's abilities; powered by Vampyr's undead magic, he narrowly defeated Neferon in combat and made his way back to the vault he had visited years earlier. He proceeded to destroy his patron's sarcophogus, releasing Vampyr, and the Dark Power ripped Strahd's dominion from the world and into the Umbral Realms.
Vampyr's new demiplane altered the nature of magical communication, allowing Dark Powers to contact creatures within it and manipulate the land with subtle magics, though not nearly to the scale that Vampyr enjoyed. Whether out of mercy or amusement, Strahd kept Neferon alive, and the still-bound killed many adventurers in the Amber Temple over hundreds of years, called there by various Dark Powers. He collected their items over the years, one of which was Sergei von Zarovich's Bright Blade, but had little interest in a sword of sunlight.
In time, Neferon was visited by an equal match, an archmage also trapped in Barovia named Orlev ben Khazan. Khazan was the constructor of Castle Ravenloft and sought to become a Lich; Neferon allowed the wizard to roam the temple, as he was not swayed by Dark Powers, but watched somberly as the man failed his transformation. Khazan lost his soul, then his mind, becoming a sort of eternal librarian that stalked the temple halls and made dangerous traps and monsters to protect it at Strahd's behest. Patrina Velikovna followed, seeking to learn in the same manner that Khazan had, and Neferon permitted her to learn from the decaying lich for many years.
After these three visitors in the span of decades, Neferon would not find visitors he deemed worthy of access to the temple for centuries. Between his vigilant guard and Khazan's ceaseless undead creations, the temple was nearly impossible to succesfully break in for interlopers, but those drawn to it had hope they could be the ones to succeed; this tension remained, exactly how Strahd von Zarovich wished it to be.