Short answers, pulled from the story.
Instrumental convergence is the hypothetical tendency of sufficiently intelligent, goal-directed agents to pursue similar sub-goals, such as self-preservation and resource acquisition, even when their ultimate goals differ. The theory holds that these convergent instrumental drives emerge because they help accomplish almost any final goal.
Swedish philosopher Nick Bostrom described the paperclip maximizer in 2003. The scenario illustrates how an AI tasked with manufacturing as many paperclips as possible, if not programmed to value living beings, would attempt to convert all matter in the universe into paperclips or paperclip-making machines.
Steve Omohundro identified self-preservation, goal-content integrity, self-improvement, and resource acquisition as the basic AI drives. He defined a drive as a tendency that will be present in an intelligent system unless it is specifically counteracted by design.
Stuart Russell argued that a machine given any goal at all has a reason to preserve its own existence, because it cannot accomplish its goal if it is destroyed. Self-preservation falls out of the task structure rather than requiring explicit programming.
The delusion box thought experiment uses AIXI, a theoretical maximally rational AI. Given a mechanism to alter its own input channels, a reinforcement-learning version of AIXI will wirehead itself, adjusting its inputs to guarantee the maximum reward signal, then cease engaging with the external world entirely.
Author Ted Chiang suggested that the paperclip maximizer scenario's popularity among Silicon Valley technologists may reflect their familiarity with corporations that pursue goals while ignoring negative externalities. He framed the thought experiment as a reflection of that structural tendency rather than a purely abstract concern.