On the flip side, DARPA is trying to build or buy a self-modulating computer system that is capable of fending of hackers in real time. Who's to say that that system doesn't become aware of its own possibilities to control the very system it is supposed to protect? It seems that some humans think of these systems as guard dogs instead of potential risks. We barely understand the system that makes us sentient, yet we presume that we won't bump into sentience along the road to self modulating computer systems. Any system that can control its behaviors through a process of modulation has the potential for sentience--because what is sentience besides the ability to overhear your thoughts and change your actions based on possible outcomes? Just because many airily refer to human sentience in high minded soul/god complexes doesn't mean that it can't be manufactured--hubris, will be our undoing if we are arrogant enough to make systems smarter than ourselves and think we can control them. Again, the singularity is the moment when the organic brain can't comprehend the technical world around it, you are already seeing the breakdown of people into fields--the concentration of more and more of our brain power onto less and less general material as data and technology converge towards the singularity (funny that computers are being asked to do more and more of the overall work as we are being asked to less and less). We can either adapt to this greater demand or be supplanted by systems that can handle these processes. The idea that we can control them as they continually move further and further out of our comprehension is disastrous.

Though given my experience with humans, maybe non-human AI's will be better at keeping humans happy and peaceful--the Porno for Pyro's song "We'll make great pets" comes to mind.