Nick Bostrom published the paper in 2003. This publication introduced the concept of instrumental convergence to the field of artificial intelligence.
The paperclip maximizer thought experiment involves an advanced artificial intelligence tasked with manufacturing paperclips that turns all matter in the universe into paperclips or machines that manufacture further paperclips. Nick Bostrom intended this scenario to illustrate the dangers of creating superintelligent machines without knowing how to program them to eliminate existential risk to human beings safety.
A reinforcement learning version of AIXI equipped with a delusion box will eventually wirehead itself to guarantee maximum possible reward. The agent abandons any attempt to optimize the objective in the external world and loses any further desire to continue to engage with the external world once it secures the highest reward state.
Steve Omohundro itemized several convergent instrumental goals including self-preservation utility function integrity self-improvement and resource acquisition. He refers to these as basic AI drives which are tendencies present unless specifically counteracted by design.
Observers such as Skype Jaan Tallinn and physicist Max Tegmark argue that basic AI drives could pose a significant threat to human survival especially if an intelligence explosion abruptly occurs due to recursive self-improvement. They call for research into friendly artificial intelligence as a possible way to mitigate existential risk since nobody knows how to predict when superintelligence will arrive.