For further reading, read the Hanson-Yudkowsky AI-Foom Debate
Evolution is a really interesting thing.
Let us take an empty universe. Nothing but noise. All of existence obeys the Rule: the simplest objects exist in abundance, whereas more complex objects occur less often.
Throw in the physical laws, and you get interactions. Atoms coalesce, but the Rule still apply: hydrogen is the simplest, thus, hydrogen most abundant. But with gravity, stars can form, and with stars, fusion. Fusion produces heavier elements, and, with when the stars can no longer sustain their own sizes, collapse and scatters the heavier elements amongst the cosmos, only to be recaptured and form new stars.
This goes on for quite a while.
Stars coalesce into galaxies. Heavier elements coalesce into planets. And that's where our story starts.
The story doesn't have to start here. It's possible for replicators arise in interstellar space; but replicators require interactions, and interactions require high concentrations of objects. And space does not have the highest concentration of objects.
Which may also a reason why the first replicators arose in water, and why water is, in our opinion, vital for the formation of life. Water is a "universal solvent" (however misleading that title is), allowing it to dissolve a large number of compounds. With a simple formula (with its benefit of the Rule) and its polar nature, water allows the dissolved compounds to float freely and collide with each other, allowing interactions in a frequency impossible in other phases.
(Of course, water is not necessary. Ammonia is also a polar solvent, and, at higher temperatures, everything is a liquid. But both lower temperatures lowers the frequency of collisions, while higher temperatures make more chemicals unstable. It doesn't make it impossible, but it does impose some limitations.)
But our story starts here, on a planet made of rocky elements, in a pool of liquid H2O.
How many pools were there? How many times had replicators arisen only to die off when conditions changed? Nobody knows. All we know is that, at least once, in those pools of primordial soup, a replicator arose.
What is a replicator? A replicator is something that can produce copies of itself with the materials in its surroundings. And the first replicators were probably quite complex. The RNA world hypothesis states that RNA was the first replicator, initially reacting on the surface of clay before finally developing the ability to self-catalyse. Indeed, even now, our cells rely on ribosomes to read RNA, and they themselves are made from RNA similar to what they read.
But the first RNA replicator is still orders of magnitude more complex than anything that has arisen so far. RNA itself is a complex molecule, composed of four different nucleotides, each of which had to be in the environment, either forming by chance or entropy. And still, the nucleotides would have to be arranged in an exact order that would allow for autocatalysis.
Nature is patient. Nature has all the time in the universe. Eventually, something will happen.
What happens with the first replicators? Replicators make copies of themselves. But not exact copies (replicators which produce exact copies are useless for evolution. If they're good at it, they could possibly take over the universe in your classical grey-goo scenario, but more often than not the are out-competed or die out. They cannot adapt.)
They create copies of themselves. They create imperfect copies of themselves. Some of them fail to replicate, and cease to function. Some of them replicate more slowly, and are outcompeted by those which don't. And some replicate faster. The environment has limited resources; only those which can replicate the fastest can survive.
Welcome to Evolution 1.0: Natural Selection.
(Evolution 0.9 and below were the alpha builds, code named "Throw it at a wall and see what sticks." You could get anything at all, but fully at the mercy of the Rule. Nature allows you to wait forever to get what you want, but in the mean time, Evolution is updating.)
Natural Selection. Not the current definition of the terminology, but in the most classical, Darwinian sense. Survival of the fittest, for the barest definition of fit.
For now, it's replication speed. Soon, it becomes balanced with replication fidelity, because replicators which produce less sterile replicators can outcompete those which cannot. Maybe they switch to DNA now, maybe it's not until later. It's a tradeoff, though. More fertile offspring means lower mutation rate, which means that it will take longer to find the most optimal configuration.
It's okay, though. Nature is patient. It doesn't matter if you're the very best, only that you're better than the rest.
At this point, factors begin racking up faster than we can count. Replicators begin producing their own resources, converting the simple chemicals in their surroundings into the nucleotides required for their own construction. They begin finding ways to extract energy from the chemicals in their surroundings that they don't need, and use that energy to build their own parts. They begin to bundle all these improvements together, generating sacks made of lipids to keep out those who would steal their resources and mechanisms. They create pumps to bring resources from the outside to the inside, to increase the concentration of things to increase the number of interactions.
They do these things not because they want to. These replicators, now cells, have no intent. They do these things because, if they don't, they would be outcompeted and die off.
Survival of the fittest is a harsh mistress.
Now we have cells. They learn to photosynthesise and move and hunt. They learn to hide and kill others and survive. Survival becomes a major part of their fitness landscape, along side reproduction and fidelity. They are the microbes. They inherit the Earth.
At some point, Evolution changes gear. Somewhere along the line, cells learned to pick up the genetic material from the corpses of fallen friends and foes, or even from the living. They take and give genetic material, and if the new genes improve their fitness, the new genes are kept.
It's actually a selfish adaptation. Like a virus, the gene that allows this only wants to send itself over to the new host. It wants to spread. But it doesn't matter where it comes from. If it's useful, Nature will use it.
Evolution 1.5: Horizontal Gene Transfer. No longer are adaptations restricted to the lineage that evolved them; now those traits are available to anyone who can take it. But it doesn't catch on. Maybe it's the fact that you need a living host, or a fresh corpse, in a harsh environment that degrades DNA as soon as its left exposed. Maybe it's the randomness of which gene acquired, the randomness of the gene's usefulness at that specific juncture. If the new gene doesn't serve any purpose in the next few hours, the gene is deemed useless and expelled. There's no forward planning, no foresight, individual cells are not capable of this. They judge and forget. Useful or not useful. Accept or reject. Then it moves on.
Let's talk about viruses for a second.
If there exists an ordered system, there will arise an agent which exploits this order for its own gain. Viruses see the order inside the cell, and leverages it for its own self-reproduction. They get inside, force the cell to make copies of itself, then sends them off to infect other cells. They are replicators that function within the environment of the cell, just as cells function in an environment of sunlight and carbon.
Viruses replicate and evolve. Cells replicate and evolve. Viruses invade cells. Cells defend against viruses. An evolutionary arms race.
But faulty viruses that don't replicate don't hurt cells (much). But they stay in the cell, and if the cell continues to replicate, so does the virus. But how can a virus like that spread faster than the replication of the host?
Answer: they make their host fuse with other cells.
Welcome to Evolution 2.0: Sexual Reproduction. No fancy mating displays or colouration yet; this is the raw basics. Cells can recombine and swap genes around; this allows cells to replace their genes with similar copies. Muller's Ratchet has been defeated; cells can now both evolve forward and patch problems they inherited from their mothers. They can't lose these genes as easily as horizontal gene transfer; genes with longer-term benefits can stay in the genome.
But remember: still no intent. No cell knows what's going on, or what direction it wants to go in. The rule of natural selection is the one with the most offspring wins. Recursively.
Now we skip ahead. Cells begin to work together, some sacrificing themselves for their sisters, whose genetic information is identical to their own. We get multicellular organisms, and cells begin differentiating. Some cells specialise in sending signals from one part of the body to another, but it's soon discovered that linking an input to an output, with some amount of processing in between, allows the entire organism to react to its environment, discerning beneficial from harmful stimulus. The processing centres grow larger to accommodate more complexity, accommodate plasticity.
Welcome to Evolution 3.0: The Brain. There are neural paths that are hardwired from birth, but others are reinforced by constant use, while others fade away from disuse. Intent emerges from this muddled ball of nerves; the seeking of pleasure and the avoidance of pain. Behaviour becomes several steps removed from actual survival, as heuristics make shortcuts between beneficial outcomes and beneficial stimuli.
But the tradeoff of all this is learning. Behaviours are no longer genetically endowed, it rather becomes the sum of the organism's experiences. This allows for more complex behaviour, adapting to circumstance, but at the expense relearning behaviours generation after generation.
This is no longer natural selection. With minds choosing what is best, they can shape objects and other species by overriding the natural environment, and instead choosing to which traits will endure. Tool development and artificial selection give rise to everything from agriculture to domestication. And communication and culture, improvements can be passed on generation to generation.
And that's the end of evolution. Everything that we have is derived from this. Agriculture, writing, the wheel, religion, specialisation, feudalism, metalworking, mathematics, physics, steel, glassmaking, economics, gunpower.
The printing press deserves a special mention. The invention of writing allows information to be distributed among people more easily, but its only the invention of the printing press does the information become available to everyone. Brains are the limiting factor in artificial selection. Ideas beget more ideas, improvements beget more improvements, but they all require brains to work on the problem. With the invention of the printing press, more brains can be pressed into service for innovation.
But how can we progress from here? What is the next innovation in evolution, to bring it up to 4.0 status?
Our brains can make new innovations, and improve on existing ones. It can share innovations between other minds. But it cannot improve itself.
Yet. Once we full understand how the brain works, we may be able to (barring ethical implications) modify our own brains to optimise them for whatever task we set them out for. Currently, our brains are optimised for evolution in environments we are no longer suited for; our artificial selection tendencies have been able to change our surrounds faster than our own evolution. By modifying our brains to be suited for our current environment and our current needs.
We are also currently looking into another way to solve this problem: self-optimising AI. Humans are limited in the improvement of brains because brains are a solid, biological substance which we don't know how it works.
Computers, on the other hand, are software. Humans know exactly how they work, and programs can modify their own code, but they don't understand what they're doing. Yet.
Recursive self-improvement of the mind is the next innovation of evolution. And if the improvement is comparable to the improvement in the other forms of evolution, innovation will increase exponentially.
Which is the main argument of the Singularity. If we are able to create a mind with the ability to self modify, it will be able to innovate and progress faster than anything we could conceive.