Paperclip Maximizer

The paperclip maximizer is a thought experiment by Nick Bostrom that illustrates how misaligned artificial intelligence (AI) could pose an existential threat.

It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal. […] This could result, to return to the earlier example, in a superintelligence whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.

Ethical Issues In Advanced Artificial Intelligence

In a hypothetical scenario, an AI system given the simple goal: “manufacture as many paperclips as possible” would start out by optimising manufacturing efficiency to achieve its goal. As the AI becomes more intelligent, it begins transforming its surroundings into paperclip manufacturing facilities, using up all the available energy, infrastructure, and raw materials in the process.