A mortal can only attune one crystal at a time. Common crystals wear out eventually, but the more rare colored ones do not. Common magic is practiced with spells, whereas magic powered by colored crystals are at-will. Magic crystals are very rare, and the dwarves keep the market for them on a tight leash.
The more inborn magic a race has access to, the less likely they are to attune to a crystal, and if they do, they are not likely to be powerful with it. The power of a crystal seems to be related to its size and the will of the wielder. Those with exceptionally powerful magic have crystals too large to even hold in one hand, whereas subtle abilities can be granted by smaller fragments.
The ability is granted by the crystal itself. If a mage dies, someone new can attune the same crystal, and they will get the same abilities. Crystals choose people to attune to when they're touched, seemingly at random. Since magical theory and science are both unable to explain who is chosen, the best explanation that mankind has been able to come up with is that it's simply fate. There is also a ritual that can bind a crystal to a person regardless of the crystal's choice, but it's risky, often resulting in destruction of the crystal or death of the prospective mage.
A crystal can't bond with another person until the wielder dies. If the crystal is taken away from the mage, it will somehow find its way back to them hours or days later. It might appear in their pocket or in their desk drawer. The mage has no control over where or when. Destroying a crystal kills the mage too, but it's hard to do, requiring something special like mithral, cold iron, dragonfire, powerful Vathian magic, etc. There is a spell that Jesnite mages can do to temporarily seal a gem away from a mage for longer than it could normally stay away, but doing so means that when the gem inevitably returns to them, it will unleash wild magic.