Natural language processing

• 1. Introduction

  ›

  Natural language processing, known as NLP, sits at the intersection of computer science, linguistics, and artificial intelligence. In 1950, Alan Turing published a paper titled "Computing Machinery and Intelligence," posing a question that would anchor decades of research: can a machine interpret and generate human language well enough to pass as human? That question planted a seed. It grew into one of the most ambitious scientific projects of the modern era. How do you teach a machine to read, to understand ambiguity, to recognize sarcasm, to translate between languages it has never seen paired before? This documentary follows the long road from a room full of hand-coded rules to the neural networks reshaping medicine, commerce, and communication today.

• 2. The Symbolic Era

  ›

  The Georgetown experiment in 1954 set an optimistic tone. Researchers demonstrated fully automatic translation of more than sixty Russian sentences into English and predicted the problem of machine translation would be solved within three to five years. That prediction proved spectacularly wrong. The ALPAC report in 1966 concluded that ten years of research had failed to meet expectations, and funding for machine translation dried up almost overnight. The symbolic approach that dominated this era rested on a simple premise: give the computer a rulebook. John Searle's famous Chinese room thought experiment later captured the logic precisely. A person inside a room follows a phrasebook of questions and matching answers, producing responses that look like understanding without any genuine comprehension. The programs of the 1950s, 1960s, and 1970s worked on the same principle. ELIZA, written by Joseph Weizenbaum between 1964 and 1966, simulated Rogerian psychotherapy. Despite having almost no model of human thought or emotion, ELIZA generated exchanges that struck users as startlingly human-like. When a patient's input exceeded the program's small knowledge base, ELIZA fell back on a mirror question: responding to "My head hurts" with "Why do you say your head hurts?" Ross Quillian's work on natural language operated with a vocabulary of only twenty words, constrained by how little text could fit in the computer memory of the day. Through the 1970s, programmers moved toward something they called conceptual ontologies, structures that mapped real-world knowledge into machine-readable form. Systems such as MARGIE, SAM, PAM, TaleSpin, QUALM, Politics, and Plot Units emerged from that decade. The 1980s saw symbolic methods at their peak, with research pushing into rule-based parsing, morphology, and the Rhetorical Structure Theory of discourse. One quiet development during the 1980s would eventually overturn all of it: the rising importance of quantitative evaluation, which made it possible to measure, and therefore to challenge, how well these rule-heavy systems actually performed.

• 3. The Statistical Turn

  ›

  Starting in the late 1980s, a different philosophy took hold. Researchers stopped writing rules by hand and started training algorithms on large bodies of text. The shift drew energy from two forces: raw computing power was growing steadily, and Chomskyan theories of transformational grammar, which had discouraged the kind of corpus-based work machine learning required, were losing their grip on the field. IBM Research became an early center of this new approach. Its IBM alignment models for machine translation were able to exploit multilingual corpora that existed for administrative reasons: the Parliament of Canada and the European Union had both produced large volumes of text in multiple official languages as a legal requirement. That gave researchers a ready-made training set. Even so, most statistical systems remained tied to corpora built specifically for their tasks, a limitation that constrained how broadly any single system could perform. The introduction of hidden Markov models for part-of-speech tagging marked the moment the old rule-based paradigm genuinely began to crack. In 2003, the best statistical algorithm of the time, a word n-gram model, was outperformed by a multi-layer perceptron trained on up to fourteen million words, built by Bengio and colleagues. That result signaled that the statistical era itself was already giving way to something newer.

• 4. Neural Networks Arrive

  ›

  Tomáš Mikolov, then a PhD student at Brno University of Technology, applied a simple recurrent neural network with a single hidden layer to language modeling around 2010. That work led directly to Word2vec, one of the most influential tools in modern NLP. Through the 2010s, deep neural networks featuring many hidden layers spread across the field, displacing statistical methods in task after task. The key advantage of neural approaches is that they capture semantic properties of words through learned representations, word embeddings, rather than through hand-engineered features. Neural machine translation made earlier intermediate steps obsolete. Statistical translation had required explicit word alignment as a preparatory stage; neural sequence-to-sequence models absorbed that work internally. Since 2015, neural methods have increasingly dominated. One practical consequence: grammatical error correction, which requires reasoning across phonology, morphology, syntax, and semantics simultaneously, was by 2019 considered largely a solved problem for English, driven by powerful language models like GPT-2. In healthcare, NLP now analyzes clinical notes in electronic health records that would otherwise remain inaccessible for research, helping both to improve care and to protect patient privacy.

• 5. The Landscape of Tasks

  ›

  Speech recognition sits among the tasks NLP researchers classify as "AI-complete," meaning that solving it fully requires essentially all the knowledge a human brings to communication. In natural speech, words run together without clear pauses; sounds blend across word boundaries in a process called coarticulation; and the same word sounds different from speaker to speaker and accent to accent. Tokenization looks far simpler but reveals its own complications at language boundaries. For English, dividing text into words mostly means splitting on spaces. For Chinese, Japanese, and Thai, no such markers exist, and segmentation requires vocabulary and morphological knowledge. Part-of-speech tagging shows how much ambiguity hides inside everyday language: the word "book" can be a noun or a verb, "set" can be a noun, verb, or adjective, and "out" qualifies as at least five distinct parts of speech. Named entity recognition faces the opposite problem from simple capitalization rules. German capitalizes all nouns, not just names. French and Spanish do not capitalize names used as adjectives. Arabic and Chinese use no capitalization at all. Coreference resolution, the task of determining which words in a text refer to the same real-world object, handles cases like the bridging relationship in "He entered John's house through the front door": a system must infer that "the front door" refers specifically to the front door of John's house. At the highest level of complexity, machine translation is also classed as AI-complete, requiring grammar, factual knowledge, and real-world reasoning all at once. Book generation offers a vivid illustration of how far automated language has come: the first machine-generated book, Racter's "The policeman's beard is half-constructed," appeared in 1984 via a rule-based system, followed in 2018 by "1 the Road," a neural-network novel running to sixty million words, and in 2019 by "Lithium-Ion Batteries," the first machine-generated science book, published by Springer, grounded in factual knowledge rather than unconstrained generation.

• 6. Cognition and Future Directions

  ›

  George Lakoff proposed a method for building NLP algorithms through the lens of cognitive science, centering on the theory of conceptual metaphor. The principle is that one idea is understood in terms of another. The English word "big" illustrates the point: in the phrase "That is a big tree," the word signals physical scale; in "Tomorrow is a big day," it signals importance; in "She is a big person," the intent remains genuinely ambiguous. Capturing that kind of context-dependent meaning mathematically requires tracking not just what a word is, but what surrounds it. Strong ties between NLP and cognitive linguistics characterized the symbolic era, but those connections weakened during the statistical turn of the 1990s. More recently, cognitive approaches have returned as a path toward explainability in AI systems. The CoNLL Shared Tasks, a long-running series of NLP competitions, reveal the field's direction through the problems it keeps choosing to study: the sequence runs from shallow parsing in 1999-2001, to named entity recognition in 2002-03, to dependency syntax, semantic role labelling, coreference, discourse parsing, and finally semantic parsing in 2019. Each step moves further from surface form toward abstract meaning. A parallel trend in those competitions is the expansion of language coverage, from English alone in 1999, to Spanish and Dutch in 2002, to more than sixty languages by 2018. British neuroscientist Karl J. Friston's work on the free energy principle at University College London has recently entered discussions about new directions in artificial general intelligence, connecting NLP research to some of the deepest questions about how minds, biological or artificial, model the world.