Quick answer
I am happy to share some of what I have learned. One caveat, my ‘advice’ is just that and you will need to adapt it to what works best for you.
I am glad that you are interested in magic systems. Like you, I feel that a good magic system is essential to a good fantasy novel (although I also believe that characters are what makes a novel truly powerful.) I wrote some essays about this very topic. Start with Sanderson’s First Law. You’ll find links to the others in the FAQ Question “What are Sanderson’s Laws of Magic.” I would rather have you read them, if you will, than rehash most of my arguments here.
If you haven’t already checked it out, you may enjoy listening to my podcast, Writing Excuses. Some episodes from Season One that you might find helpful are Episode 15: Costs and Ramifications of Magic and Episode 14: Magic Systems and their Rules.
Magic systems in depth
I don’t know if I can answer that question in the short space afforded by a discussion forum. But in general with my magic systems I’m looking for a variety of components. Most of them start with just an “Aha, there’s something there!” moment in my head—either it’s a plot hook or a conflict hook or a visual hook or something like that. I’m usually looking for something that does what I find exciting about magic, which is straddling the line between mysticism and science. And I’m looking for new ways to explore that. So when an interesting scientific concept occurs to me, and I can take it in the direction of “what if,” that’s something that I find fascinating.
For MISTBORN, for instance, telekinesis mixed with vector science was interesting to me. In WARBREAKER it was the concept of sympathetic magic—the idea that you can create something that’s like something else and it will have power over that. I wanted to try and take it in a direction I hadn’t seen before and blend that with the concept of animation, bringing inanimate objects to life. Those were intersting concepts because at one point people believed in both of these things as real forms of magic. They believed they could make it work. The myth of the golem goes way back, and the idea of sympathetic magic was around not too long ago—in fact there are still plenty who believe in it, in various forms of superstition.
So I look for a blend of concepts. I usually look for an interesting visual paradigm—something that will work in a way that helps the reader visualize the magic. I don’t want it to all happen nebulously in the back of someone’s head. But anyway, I’m looking for something that you can see and follow the process of what the character’s doing in a way that makes sense.
I find that if there’s one thing to take away from this, limitations on magic are more interesting than the powers themselves. And so I’m always looking for interesting limitations, because that forces me to be creative and forces my characters to be creative with what they have.
How difficult is it to come up with new magic systems? So much has already been done.
I’ve got a few very nifty ones reserved for the future. Don’t worry; I’m not nearly out of ideas yet. And I’m constantly having new ones I don’t have time to use.
There IS a lot of fantasy out there. And yet, I think there’s a great deal of room left for exploration in magic. The frontiers of imagination are still rough-and-tumble, unexplored places, particularly in this genre. It seems that a lot of fantasy sticks very close to the same kinds of magic systems.
One of the things I’ve come to believe is that limitations are more important than powers in many cases. By not limiting themselves in what their characters can do, authors often don’t have to really explore the extent of the powers they’ve created. If you are always handing your characters new powers, then they’ll use the new and best–kind of like giving your teen a new car every year, rather than forcing them to test the limits of what that old junker will do. Often, those old cars will surprise you. Same thing for the magic. When you’re constrained, as a writer, by the limits of the magic, it forces you to be more creative. And that can lead to better storytelling and a more fleshed out magic.
Now, don’t take this as a condemnation of other books. As writers, we all choose different things to focus on in our stories, and we all try different things. Jordan’s ability to use viewpoint, Martin’s use of character, Pratchett’s use of wit–these are things that far outshine anything I’ve been able to manage in my works so far.
But I do think that there is a great deal of unexplored ground still left to map out in some of these areas (specifically magic and setting). A great magic system for me is one that has good limitations that force the characters to be creative, uses good visuals to make the scenes more engaging while written, and has ties to the culture of the world and the motivations of the viewpoint characters.
Your magic systems are all very detailed and internally consistent. Do you come up with the world and then write a story within it, or build the world to fit the story?
The answer is, “Yes.” Which is one of those unsatisfying authorly answers. It depends on the story. Like for instance with the Wax and Wayne books, I already had the world built, so in that I’m building a story around a setting that already existed. With the Reckoners, what happened is I had the idea for people who gained superpowers all going evil, and that concept spun me into building a story about it. So that’s more of an idea that spins a story rather than a setting. Sometimes I’ve had a character I’ve really wanted to tell a story about, like Raoden or something like this, and then I build magic to match. It happens all different ways, and really it’s a give and a take. Once you start with a character, you start building a story around them, and then you stop and you work on the magic for a while, then you go back to the character and then you go back to the magic and then you go to the setting and then you go to the plot. As you build an outline you weave all these things together. You’re not just spending time on one till it’s done and then the next until it’s done. It’s happened all different ways for me.