Occam's razor

• 1. Introduction

  ›

  Occam's razor is named for William of Ockham, a 14th-century English philosopher and theologian. Yet the Latin line most people attach to it, Entia non sunt multiplicanda praeter necessitatem, is absent from his surviving works. It translates as "Entities must not be multiplied beyond necessity." The principle also goes by Ockham's razor, Ocham's razor, novacula Occami, the principle of parsimony and lex parsimoniae. All of those names point to one advice: when you explain something, build the explanation from the smallest possible set of elements.

  The popular paraphrase says that of two competing theories, the simpler one should be preferred. That sounds like common sense, and it is also a claim that philosophers, physicists, biologists, theologians and mathematicians have fought over for centuries. How did a rule this old end up carrying the name of a man who never wrote its famous sentence? Why would a simpler explanation deserve more trust than an elaborate one? And what happens when the simple answer turns out to be wrong?

  13151996

• 2. A Borrowed Blade

  ›

  In 1649, Libert Froidmont wrote Philosophia Christiana de Anima, On Christian Philosophy of the Soul, and claimed to have coined the phrase novacula occami. He used it while describing Gregory of Rimini, one of Ockham's critics. That was a few centuries after Ockham died in 1347, so the man himself never heard the name. The word razor points to shaving away unnecessary assumptions, or to cutting apart two similar conclusions.

  The best-known wording has an even later source. The Irish Franciscan philosopher John Punch produced the form Non sunt multiplicanda entia sine necessitate in his 1639 commentary on the works of Duns Scotus. Punch is also the source of the Entia non sunt multiplicanda praeter necessitatem phrasing, and he called the idea a "common axiom" of the Scholastics, in Latin axioma vulgare.

  Ockham did write his own versions. In his theological work on the Sentences of Peter Lombard, he stated Numquam ponenda est pluralitas sine necessitate, "Plurality must never be posited without necessity." In his Summa Totius Logicae he cited the economy principle as Frustra fit per plura quod potest fieri per pauciora, "It is futile to do with more things that which can be done with fewer." Ockham was an English Franciscan friar, a nominalist, and an influential medieval philosopher, born around 1287. He did not invent the principle. His name may have stuck because of how often and how effectively he used it.

  The lineage runs much further back. In his Posterior Analytics, Aristotle, who lived from 384 to 322 BC, preferred the demonstration that rests on fewer postulates or hypotheses, other things being equal. Ptolemy, around AD 90, called it a good principle to explain phenomena by the simplest hypothesis possible. Maimonides, Duns Scotus and Thomas Aquinas show up in the same chain of ancestors.

  The clearest early statement comes from Robert Grosseteste, who lived from 1175 to 1253. In his commentary on Aristotle's Posterior Analytics, written around 1217-1220, he judged that explanation better which needs fewer premises, other circumstances being equal. He wrote this roughly seventy years before Ockham was born.

  19121310

• 3. The Case for Simplicity

  ›

  Thomas Aquinas, in the 13th century, gave the oldest justification on offer. He wrote that if one thing can do the job adequately, it is superfluous to use several, because nature does not employ two instruments when one suffices. For centuries, then, philosophers assumed that nature itself was simple, which made simple hypotheses more likely to be true. From the 20th century on, justifications drawn from induction, logic, pragmatism and especially probability theory became more popular.

  Probability gives the bluntest argument. Every assumption is another chance to be wrong. If an assumption does not improve a theory's accuracy, its only effect is to raise the odds that the whole theory fails. Harold Jeffreys and E. T. Jaynes tried to derive the razor from probability theory. David J. C. MacKay, in chapter 28 of his book Information Theory, Inference, and Learning Algorithms, stresses that a built-in prior bias toward simple models is not required.

  In 1991, William H. Jefferys and James O. Berger gave the argument its neatest line. A hypothesis with fewer adjustable parameters, they wrote, gains an enhanced posterior probability because the predictions it makes are "sharp." The pun on a razor is deliberate. Their model rewards theories that sharply predict correct results over theories that leave room for many possible outcomes.

  A related worry is overfitting. An overly complex model starts to fit statistical noise instead of the real structure in the data, a problem tied to the bias-variance tradeoff. Simpler models may therefore predict better. The difficulty is that it is often hard to say which part of the data is noise.

  Consider a man accused of breaking a vase who blames leprechauns. The simple explanation is that he did it. When shown film of the act, he insists that the footage was tampered with too. Each patch is an ad hoc hypothesis, and such patches keep a failing explanation from being disproved. Philosophers call this endless supply saving hypotheses, and only Occam's razor can rule them out.

  There is also evidence from forecasting. A study of the predictive validity of the razor found 32 published papers containing 97 comparisons of economic forecasts from simple and complex methods. None of the papers showed a balance of evidence that complexity improved accuracy. In the 25 papers with quantitative comparisons, complexity raised forecast errors by an average of 27 percent.

  Philosophers have supplied their own grounds. Karl Popper argued that we prefer simple theories because their empirical content is greater and they are better testable. Richard Swinburne went further and called simplicity an ultimate a priori epistemic principle, a kind of evidence for truth. Elliott Sober once tied simplicity to informativeness, then rejected that account because it gave no epistemic justification. In the Tractatus Logico-Philosophicus, Ludwig Wittgenstein declared that if a sign is not necessary, it is meaningless.

  The razor has never been a law of logic, though. Scientists have had to learn where it stops helping.

  1437358742

• 4. Where Science Applies It

  ›

  Isaac Newton wrote that we should admit no more causes of natural things than are both true and sufficient to explain their appearances. With the phrase hypotheses non fingo, "I contrive no hypotheses," he affirmed the success of that approach. Bertrand Russell proposed substituting constructions out of known entities for inferences to unknown entities. Ernst Mach carried a stronger version into physics, calling it the Principle of Economy. Medicine has its own folksy form, credited to Theodore Woodward: when you hear hoofbeats, think of horses, not zebras.

  Parsimony guided Pierre Louis Maupertuis and Leonhard Euler in the principle of least action. It shaped Albert Einstein's special relativity, and it helped Max Planck, Werner Heisenberg and Louis de Broglie build quantum mechanics. Planck shows how the razor can make a conservative scientist into a reluctant revolutionary. He interpolated between the Wien and Jeans radiation laws, used razor-style logic to arrive at the quantum hypothesis, and then resisted it even as its truth became clearer.

  Simplicity has also pointed the wrong way. Appeals to it were used against meteorites, ball lightning, continental drift and reverse transcriptase. Atomic theory once looked more complex because it implied invisible particles nobody had detected. Mach and the logical positivists rejected John Dalton's atoms until Einstein's work on Brownian motion made their reality plainer. The aether looked simpler than light crossing a vacuum, since every known wave traveled through a physical medium. Newton's light particles beat Christiaan Huygens's waves for many people, yet light behaves both ways.

  Einstein himself urged caution. He wrote that the supreme goal of theory is to make the irreducible basic elements as simple and as few as possible, without giving up the adequate representation of a single datum of experience. A popular shortening, "Everything should be kept as simple as possible, but not simpler," cannot be verified as his.

  Science also has no use for the razor when two models predict exactly the same results. Newtonian, Hamiltonian and Lagrangian classical mechanics are equivalent, and physicists do not try to call two of them wrong. The wave and matrix formulations of quantum mechanics are in the same position. Parsimony only has meaning inside a specific line of inquiry, so no single global principle spans every subject. Used properly, the razor chooses between theories that are equally well supported, or helps decide which of two testable ideas to test first.

  Biologists have argued hardest about whether the same trick works for living things.

  2124444929

• 5. Life Sciences

  ›

  When wolves threaten muskoxen, the males form a circle with the females and young inside. That looks altruistic, since the males pay a cost for the group's benefit, so some saw it as support for group selection. Kin selection offers another reading: males protecting their calves are protecting copies of their own alleles. The behavior would be favored if its cost to the male were less than half the benefit to his calf. A third reading says a male standing in a ring with horns pointing out is simply less likely to be killed. That is regular natural selection, called the selfish herd.

  George C. Williams made the razor central to this debate in his 1966 book Adaptation and Natural Selection. He argued that individual selection is the more parsimonious explanation of animal altruism, and he was answering group-selection proponents such as D. S. Wilson and E. O. Wilson. He was invoking Morgan's Canon, from 1903: no animal activity should be read in terms of higher psychological processes if lower ones can fairly explain it. Richard Dawkins, in The Selfish Gene, later argued that the canon is not the most basic explanation. In his view, natural selection acts on genes, and both individual and group selection emerge from that.

  Systematics, the study of relationships among biological groups, applies the razor through cladistic parsimony. Its three camps are cladists, pheneticists and evolutionary taxonomists. Cladists classify by shared, derived characters. Cladistic parsimony picks the cladogram that needs the fewest character state transformations, where a cladogram is a branching diagram of hypothesized relationships. Critics note that parsimony can give wrong answers on some data, however much data is gathered, a flaw called long branch attraction. Defenders reply that this could be said of any method of inferring phylogeny, unless its model matches how evolution really happened.

  Elliott Sober wrote a book-length treatment, Reconstructing the Past: Parsimony, Evolution, and Inference, in 1988. He followed it in 1990 with an article titled "Let's Razor Ockham's Razor." Likelihood methods also use parsimony, by treating hypotheses with fewer differing parameters as the null hypotheses. In biogeography, parsimony helps infer ancient migrations by assuming the minimum amount of change.

  Francis Crick distrusted all of this. He argued that biological systems are products of ongoing natural selection, so their mechanisms are not necessarily optimal in any obvious sense. In his words, Ockham's razor is useful in the physical sciences but "a very dangerous implement in biology." He called it rash to use simplicity and elegance as a guide in biological research.

  5859609889

• 6. Faith, Mind and Punishment

  ›

  William of Ockham was a Christian who believed in God and in the authority of scripture. He wrote that nothing ought to be posited without a reason given, unless it is self-evident, known by experience, or proved by the authority of Sacred Scripture. Unlike many theologians of his day, he did not think arguments could prove God. For him, science was a matter of discovery, while theology was a matter of revelation and faith. He also limited his own razor where miracles and God's power were concerned, and allowed a plurality of miracles in the Eucharist simply because it pleases God.

  Thomas Aquinas used the principle in the Summa Theologica to state an objection to God's existence. If everything in the world can be reduced to nature and human reason, the objection runs, there is no need to suppose God. Aquinas answered it with the quinque viae, his five ways. Whatever nature does, he replied, must be traced back to God as its first cause. Anything done by will must be traced to a higher cause, because reason and will can change or fail.

  Søren Kierkegaard sidestepped the contest. He viewed belief in God as a leap of faith that sometimes directly opposed reason, which makes the razor irrelevant. Gordon Clark's presuppositional apologetics also grounds belief prior to reason, though Clark never thought the leap contradicted reason. On the opposing side, some anti-theists hold that assuming God adds needless complexity, as in the Ultimate Boeing 747 gambit from Dawkins's The God Delusion.

  George Berkeley, who lived from 1685 to 1753, turned the razor against matter itself. He was an idealist who held that all of reality could be explained in terms of the mind, so matter was eliminable. The weakness is that, on his position, solipsism might fit the razor even better than a God-mediated world. An apocryphal story has Napoleon asking Pierre-Simon Laplace why God never appeared in his work, though it appeared often in Lagrange's. Laplace supposedly said, "It's because I had no need of that hypothesis."

  In 1959, J. J. C. Smart published "Sensations and Brain Processes" and invoked the razor to prefer the mind-brain identity theory to dualism. Identity theorists hold that everything, consciousness included, is physical. Smart was severely criticized for the move and eventually retracted his advocacy of it in this context. Paul Churchland, in 1984, judged that the razor by itself is inconclusive on dualism. Dale Jacquette, in 1994, noted its use to justify eliminativism and reductionism.

  The oddest relative lives in the philosophy of punishment. Jeremy Bentham's parsimony principle holds that any punishment greater than needed to achieve its end is unjust. He thought true parsimony meant tailoring punishment to each person's sensibility, so that a more sensitive person gets a lesser one. Later utilitarian writers mostly dropped that idea, largely because nobody could measure each offender's sensitivity.

  1415676597

• 7. Counting Simplicity

  ›

  Around 1960, Ray Solomonoff founded the theory of universal inductive inference, a theory of prediction from observations. Its example is predicting the next symbol from a given series of symbols. The only assumption is that the environment follows some unknown but computable probability distribution. The theory is a mathematical formalization of Occam's razor. Marcus Hutter's universal artificial intelligence builds on it to calculate the expected value of an action. Jürgen Schmidhuber argues that the right mathematical theory of the razor already exists, in Solomonoff's work and its extensions.

  The original razor has a built-in limit. It only tells us to prefer the simplest of models with equal explanatory power. A more general form comes from Bayesian model comparison, built on Bayes factors, which can weigh models that do not fit the observations equally well. The exact Occam factor is generally intractable, so researchers use approximations such as the Akaike information criterion, the Bayesian information criterion, variational Bayesian methods, false discovery rate and Laplace's method. Many artificial intelligence researchers now use these ideas, through Occam Learning or the Free energy principle.

  Then comes the question of what simple means. In the Kolmogorov-Chaitin minimum description length approach, the user picks a Turing machine whose basic operations stand for simplicity. One could always pick a machine with a single operation that happens to build one's entire theory, which would then score highly. That has split researchers into camps. One holds that the razor is objective, the other that it is subjective. Quine framed a related split as economy of practical expression versus economy in grammar and vocabulary.

  Hutter answered by using a small, natural universal Turing machine as the proper basis. It was formally proven under Zermelo-Fraenkel set theory that the log universal probability of the model plus the log of the probability of the data given the model should be minimized. Read as the total length of a two-part message, this gives the minimum message length principle, associated with Chris Wallace and discussed by David L. Dowe. One consequence is that an ideal data compressor would also generate scientific explanations.

  Software has its own version. The rule of least power says to choose the simplest language that solves the problem, and it is often credited to Tim Berners-Lee, from his design guidelines for the original Hypertext Transfer Protocol. Meanwhile, scientists have found that deep neural networks prefer simpler mathematical functions as they learn. That simplicity bias helps them avoid overfitting, in which too many parameters leave a model overwhelmed by noise.

  The sharpest mathematical objection comes from the no free lunch theorems for inductive inference. Take algorithm A, a Bayesian procedure with a razor-motivated prior, and algorithm B, which predicts the exact opposite of whatever A predicts. There are just as many possible priors where B beats A as where A wins. The Bayesian argument for the razor therefore rests on arbitrary modeling assumptions.

  7771727883

• 8. Anti-Razors

  ›

  Walter Chatton, who lived from about 1290 to 1343, was a contemporary of Ockham and objected to his use of the razor. His reply was an anti-razor: if three things are not enough to verify an affirmative proposition about things, a fourth must be added, and so on. No later anti-razor has lasted as notably as his.

  Gottfried Wilhelm Leibniz, who lived from 1646 to 1716, answered with a principle of plenitude, in Arthur Lovejoy's phrase. God, on this view, created the most varied and populous of possible worlds. Immanuel Kant, who lived from 1724 to 1804, wrote his own counter-razor: "The variety of beings should not rashly be diminished." Karl Menger, who lived from 1902 to 1985, thought mathematicians were too stingy with variables. His Law Against Miserliness says entities must not be reduced to the point of inadequacy, and that it is vain to do with fewer what requires more.

  Alfred Jarry, who lived from 1873 to 1907, took the opposite path with 'Pataphysics, the "science of imaginary solutions." It views every event in the universe as completely unique and subject to no laws but its own. Jorge Luis Borges later explored variations on that theme in his story and mock-essay "Tlön, Uqbar, Orbis Tertius." Physicist R. V. Jones added Crabtree's Bludgeon, which says no set of mutually inconsistent observations exists for which some human intellect cannot conceive a coherent explanation, however complicated.

  Galileo Galilei mocked misuse of the razor in his Dialogue, where the principle is voiced by the character Simplicio. His ironic point was that if you really wanted the fewest fundamental entities, the letters of the alphabet would do, since all of human knowledge can be built from them. A late Renaissance Italian motto of unknown authorship, Se non è vero, è ben trovato, "Even if it is not true, it is well conceived," suits an explanation that is artful but too good to check.

  A modern complaint comes from the American physicist Igor Mazin. He argues that high-profile physics journals prefer exotic interpretations, so the razor is being replaced by an "Inverse Occam's razor," in which the simplest explanation is usually rejected.

  The razor is also a trophy. As of 2012, The Skeptic magazine awards the Ockhams annually at QED, an idea introduced by editor-in-chief Deborah Hyde. They honor skeptical blogs, podcasts, campaigns and contributors. Neil Davies and Karl Derrick designed the trophies, which show a double-edged safety razor blade and carry the motto "Shaving away unnecessary assumptions since 1285." In both lower corners sits a small image of William of Ockham's face.

  9193959694