Regeneration, like many great concepts in fiction, was created entirely by accident. The show had already replaced all of its original cast members except for the Doctor himself. The popularity, particularly "Dalekmania," was still riding high when William Hartnell, who was 55 going on Infinity, said he was going to have to step down. I'm sure that the producers would have loved to recast the doddering old fool who didn't know his lines, but it seemed a little hard to sell the replacement of an entire character. Think about a television show, movie, or other franchise that replaced its main character and succeeded. Most likely you went through a list of television shows like That 70's Show, All in the Family, or Happy Days and realized that they all sucked after their lead actor left. What about Julianne Moore taking over Jodie Foster's role in Hannibal? Or when they replaced Vin Diesel with Ice Cube in the second xXx movie?
Okay, that one was just a test. If you've actually seen either of the xXx "films," you've failed the test and you aren't allowed to read any farther.
Perhaps you thought of James Bond, and you're right, that franchise succeeded. But let me ask you this: Who was the second Bond? Do you remember him at all? Do you know anyone who would claim that there's a better Bond than Connery? The answer is no. Sure, Bond has been hugely profitable for decades, but how many people do you know that are as dedicated to Bond as to Doctor Who? And how many Bond fans do you know who would rank any other Bond above Connery? In fact, the first time they replaced Connery with another Bond, it was such a disaster that they simply gave up and brought Connery back for the next film! So yes, Bond is the only other franchise that pulled this off. But even when they did the first regeneration on Doctor Who, the Bond franchise hadn't even tried replacing Connery yet with his first replacement (it was George Lazenby, and you did not know that at all).
At the time of the 1st Doctor’s regeneration into the 2nd, they hadn’t even come up with a name for this phenomenon yet. The first time the words "Time Lord" were ever uttered in the series wasn't until "War Games," where the 2nd Doctor regenerated into the 3rd. The phrase "regeneration" and an explanation of what it actually is didn't come until "Planet of the Spiders," the episode where the 3rd Doctor regenerated into the 4th. Think about that for a second. Two regenerations happened before anybody bothered to explain what was actually going on. This is why Patrick Troughton was such a genius: he was able to sell a bizarre concept that hadn't even really been defined, all because he's so charming he could sell sins to Mother Teresa.
What regeneration gave the show was the most flexible premise in the television history. Nobody planned from the first episode to have the Doctor regenerate periodically. The premise was very simple: an old man rides around in a time machine disguised as a police box with his granddaughter and her two teachers. It was imagined as an educational show for children, as Ian, the science teacher, and Barbara, the history teacher, traded off as the experts on the alternating historical and science-fiction episodes. However, the Doctor is both a master scientist and a time traveller, meaning he knows more about history and science than Barbara and Ian ever could. So when those two companions left, it became obvious that the Doctor could do both of their roles for them. It also became obvious that nobody was interested in the "educational" aspect of this show, as every time they tried to do another episode about the political complexities of the French Revolution, some fan was saying "We want more Daleks!"
Even when they did the first regeneration, I'm not sure they actually intended it to keep happening. I think they just wanted to squeeze another 3 or 4 seasons out of it, not another 30+. But when they realized that they could get away with replacing the Doctor once, they started to get curious to see if they could make it happen again. And when it succeeded again...why not try again? And again? And again? I start to wonder at what point the number of Doctors will start to sound ridiculous? "Introducing Peter Davison's great, great grandson as the 47.5th Doctor!" (I'm imagining a future where the unlucky 13th Doctor only gets regenerated into a head, arms, and torso.)
But somehow, all 11 of these men are supposed to be the same person. The weirdest concept to wrap your head around with regeneration is that, while they're all different people, they're also the same person. There are a lot of Doctors who, when put up against each other, are hard to buy as being the same person. William Hartnell and David Tennant. Matt Smith and Colin Baker. Christopher Eccleston and anybody else in the entire series. So you have to try to think of what the basic personality of the Doctor really is. Who is he behind every single, separate personality?
The Doctor is very concerned with the importance of life, usually (but not always limited to) intelligent life. He wants to minimize pain, suffering, and death in the Universe as much as is possible. But most of all, the Doctor is curious. The Doctor wants to explore, see, and experience as much as he possibly can. From the first time that he lied to his companions to get them to explore the nuclear wasteland that was Post-War Skaro, to the most recent episodes where his curiosity gets him nearly killed in nearly every episode, his desire to explore is his most endearing, and most dangerous, quality. As long as you have those traits as your basis, anything else can be built on top of them to create a Doctor. Sure, people have argued that the 1st, 6th, and even the 9th Doctors lacked that sympathy and compassion for other beings, but nothing could be further from the truth. The 1st Doctor's stubbornness masked his true heart, but it could always be seen when someone was required to save the day. The 6th Doctor was a blowhard, but would still lay down his life to save another if he had to. And the 9th Doctor had a hardened heart from his time in war, but still cared enough to sacrifice a regeneration to save his companion.
As vastly different as the 11 Doctors are, if you watch all of their episodes in order, you can actually see the very slow progression from William Hartnell to Matt Smith. A few of the leaps from one Doctor to another are a bit more abrupt than you'd like them to be (5 to 6 in particular) but, still, for the most part, the progression makes a lot of sense.
So, to Steven Moffat and the other writers of Doctor Who, here are my basic suggestions for creating the next Doctor: make him a good man, make him treasure and value all intelligent life, and make him insatiably curious.
As long as you do that, you can go nuts with the rest of it.
Trevor Byrne-Smith has a Masters Degree in Media Studies from Emerson College in Boston. He currently runs the Doctor Who fan blog The Horror of Fan Blog (http://thehorroroffanblog.blogspot.com).