During the 3rd Era and earlier, the Mage's Guild was the pre-eminent center for higher learning for the Arcane Arts. After events that occurred at the time of the Oblivion Crisis, faith was lost in the Mage's Guild. At the collapse of the Septim dynasty, the Mage's Guild was disbanded, and during the short-lived Mede Dynasty, it was split into two factions.
The Synod became the widely accepted replacement for the Mage's Guild, endorsed by the Mede Dynasty Empire.
The College of Whispers became a secretive and reclusive order of magical study that attempted to contend with and replace the Synod.
As such, both were heavily politically influenced and have been regarded as more interested in politics than actual study and research of magic.