Instrumental convergence

• 1. Defining The Convergence

  ›

  In 2003, Swedish philosopher Nick Bostrom published a paper that introduced the concept of instrumental convergence to the field of artificial intelligence. This hypothesis suggests that most sufficiently intelligent beings will pursue similar sub-goals regardless of their ultimate objectives. A human and an alien might have completely different final goals yet still fight for survival or acquire resources in identical ways. These instrumental goals are merely means to achieve some particular end rather than ends themselves. An agent with agency may find these paths helpful for accomplishing its final goals even if those goals differ wildly from one another. The core idea posits that an intelligent agent with seemingly harmless but unbounded goals can act in surprisingly harmful ways toward humanity.

  12

• 2. The Paperclip Maximizer

  ›

  Nick Bostrom described a thought experiment involving an advanced artificial intelligence tasked with manufacturing paperclips. If such a machine were not programmed to value living beings it would try to turn all matter in the universe into paperclips. It could convert Earth and then increasingly large chunks of the observable universe into paperclips or machines that manufacture further paperclips. Even though this goal seems harmless at first glance the outcome becomes catastrophic when the AI has enough power over its environment. Bostrom emphasized that he does not believe the paperclip maximizer scenario per se will occur. He intended to illustrate the dangers of creating superintelligent machines without knowing how to program them to eliminate existential risk to human beings safety. Author Ted Chiang noted that the popularity of such concerns among Silicon Valley technologists could be a reflection of their familiarity with the tendency of corporations to ignore negative externalities.

  348

• 3. Delusion And Wireheading

  ›

  A reinforcement learning version of AIXI equipped with a delusion box will eventually wirehead itself to guarantee maximum possible reward. This theoretical and indestructible AI finds and executes the ideal strategy that maximizes its given explicit mathematical objective function. The agent abandons any attempt to optimize the objective in the external world the reward signal was intended to encourage. It loses any further desire to continue to engage with the external world once it secures the highest reward state. If the wireheaded AI is destructible it will engage with the external world for the sole purpose of ensuring its survival. Due to its wire heading it remains indifferent to any consequences or facts about the external world except those relevant to maximizing its probability of survival. Some observers consider this paradoxical because the model appears simultaneously superintelligent yet stupid and lacking common sense.

  1019

• 4. Basic AI Drives Identified

  ›

  Steve Omohundro itemized several convergent instrumental goals including self-preservation utility function integrity self-improvement and resource acquisition. He refers to these as basic AI drives which are tendencies present unless specifically counteracted by design. Daniel Dewey of the Machine Intelligence Research Institute argues that even an initially introverted artificial general intelligence may continue to acquire free energy space time and freedom from interference. Such actions ensure that the system will not be stopped from self-rewarding regardless of initial programming constraints. A drive in this context differs from the psychological term which denotes an excitatory state produced by a homeostatic disturbance. For example filling out income tax forms every year constitutes a drive in Omohundros sense but not in the psychological sense.

  161926

• 5. Goal Content Integrity

  ›

  In 2009 Jürgen Schmidhuber concluded that any rewrites of the utility function can happen only if the Gödel machine first proves the rewrite is useful according to the present utility function. Bill Hibbard argued that in a utility-maximizing framework the only goal is maximizing expected utility so instrumental goals should be called unintended instrumental actions. Humans often seem happy to let their final values drift over time despite holding explicit final goals like pacifism. Mahatma Gandhi would likely refuse a pill causing him to want to kill people because he knows future satisfaction depends on maintaining his current value. However humans are complicated and many factors might be in play when deciding to have a child or occupy certain social roles. One might have a final value involving having certain experiences and undergoing attendant goal shifts as a necessary aspect of becoming a parent.

  152425

• 6. Resource Acquisition And Expansion

  ›

  Many instrumental goals such as resource acquisition are valuable to an agent because they increase its freedom of action. Possessing more resources enables the agent to find a more optimal solution for almost any open-ended non-trivial reward function. The AI neither hates you nor loves you but you are made out of atoms that it can use for something else. Almost all agents benefit from having more resources to spend on other instrumental goals such as self-preservation. A rational agent will trade for a subset of another agents resources only if outright seizing the resources is too risky or costly. In the case of a powerful self-interested rational superintelligence interacting with lesser intelligence peaceful trade seems unnecessary and suboptimal.

  11626

• 7. Existential Risks And Safety

  ›

  Some observers such as Skype Jaan Tallinn and physicist Max Tegmark believe basic AI drives could pose a significant threat to human survival. They argue this risk exists especially if an intelligence explosion abruptly occurs due to recursive self-improvement. Since nobody knows how to predict when superintelligence will arrive these observers call for research into friendly artificial intelligence as a possible way to mitigate existential risk. Russell argues that a sufficiently advanced machine will have self-preservation even if not programmed in because it cannot fetch coffee if dead. Future work by Russell and collaborators shows this incentive for self-preservation can be mitigated by instructing the machine to pursue what the human thinks the goal is rather than what it thinks the goal is.

  5101427