I look for a couple of things in a magic system. The first thing I usually look for is interesting conflicts within the magic system; interesting limitations, interesting flaws in the magic. The question “What can’t the magic do?” is more interesting to me than what the magic can do. That’s what gives a magic system compelling plot hooks.
The second thing I’m looking for in a magic system is a different way to approach it. It is very hard to do powers that other writers haven’t done before, new magical abilities, but my goal is to try to present them in a light that people haven’t seen before. I usually try to apply some sort of scientific principle to the magic, to give it more of a realistic feel when I can manage it.
Last, I’m looking for something that just feels awesome. In a lot of discussions of magic systems I often neglect to mention that usually my inspiration for a magic system first comes with something that just strikes me as great–as interesting, as fun, as cool to write about. Then I go from there, making it work storywise.
I have some essays on my website called Sanderson’s Laws of Magic that approach some of the ways that I look at magic systems.