Lobster

• 1. Introduction

  ›

  LOBSTER was a European network monitoring system that could watch traffic across 2.3 million IP addresses at any given moment. Buried in server rooms across nine countries, its sensors were recording every packet crossing every monitored link, quietly building one of the continent's most ambitious internet surveillance infrastructures. By the time the project wound down in 2007, its operators claimed to have detected more than 400,000 internet attacks. But LOBSTER was more than a security alarm. It was an attempt to answer a question that was growing urgent in the early years of the broadband era: how do you actually see what is happening inside a network at continental scale? How do you measure something as vast and fast as European internet traffic without slowing it down, distorting it, or missing what matters? And how do you protect the privacy of millions of users whose data flows through your sensors?

• 2. Origins in SCAMPI

  ›

  LOBSTER did not appear from nowhere. Its direct ancestor was SCAMPI, a European project active in 2004-5 that set out to build a scalable monitoring platform for the internet. SCAMPI confronted the foundational problem: existing tools for measuring network traffic were either too coarse or too intrusive. The researchers working on SCAMPI began developing both the conceptual framework and the core programming tools that LOBSTER would later depend on. When LOBSTER received funding from the European Commission, it inherited SCAMPI's ambitions and carried them further, deploying actual sensors across a real multi-country infrastructure rather than operating as a theoretical exercise. The programming interface that would power all of LOBSTER's measurement applications, known as MAPI, was developed across both projects.

• 3. Passive Monitoring Explained

  ›

  Most network measurement systems work by sampling: they collect summaries of traffic flows, or they send their own probe packets to test the network's behaviour. LOBSTER took a different path. Passive network monitoring records every IP packet passing through a monitored link, capturing both the header, which carries addressing and routing information, and the payload, which carries the actual content. Nothing is synthesised or approximated. The result is a complete picture of real traffic as it actually occurred. This approach gives researchers the ability to tackle monitoring problems that flow-level statistics simply cannot reach. The tradeoff is overhead: recording everything costs more than sampling. MAPI, the Monitoring Application Programming Interface, was built specifically to manage this tension. It allowed developers to express exactly how much information they needed, so they could calibrate the balance between completeness and computational cost.

• 4. MAPI and Distributed Sensing

  ›

  MAPI was described as an expressive programming interface for building network monitoring applications. What made it distinctive was not just the ability to write complex monitoring logic, but the ability to run that logic across geographically scattered sensors simultaneously. MAPI gave programmers the capacity to build remote and distributed applications that could pull monitoring data from multiple sensors at once. This mattered because LOBSTER's infrastructure was inherently distributed: 36 sensors deployed across nine countries, operated by several different organisations. Each organisation had its own measurement priorities. MAPI's distributed variant, called DiMAPI, or Distributed Monitoring Application Interface, let those priorities coexist within a single coherent system. One of the applications built on DiMAPI was ABW, which also relied on a companion library called tracklib.

• 5. Applications Built on the Network

  ›

  Three measurement applications developed within the LOBSTER project illustrate what the infrastructure was actually used for. Appmon focused on accurate per-application network traffic classification, the problem of determining which software was generating which flows. Stager handled the aggregation and presentation of network statistics, turning raw packet data into something operators could act on. ABW, the application built on DiMAPI and tracklib, addressed a different measurement need, one specific enough that each organisation's deployment could be tuned to its own requirements. The fact that all three were developed inside the project using MAPI reflects a design philosophy: the sensor network and the applications for analysing it would be built together, not bolted together after the fact.

• 6. Privacy and Anonymisation

  ›

  Recording full packet payloads across millions of IP addresses raises an obvious concern: the same completeness that makes passive monitoring powerful also makes it a potential privacy risk. LOBSTER's stated objectives acknowledged this directly. One of its formal goals was to develop data anonymisation tools capable of prohibiting unauthorised access or tampering with the original traffic data. The infrastructure was designed with the assumption that the raw captures should not be freely readable. Anonymisation was treated as a first-class engineering problem alongside performance and security monitoring, not as an afterthought. LOBSTER's outputs fed into a broader European initiative labelled IST 2.3.5 Research Networking testbeds, which aimed to contribute to improving internet infrastructure across the continent.