What is this?

UK sewers carry foul water from our properties to sewage treatment works, where it's treated and returned to watercourses. When it rains, some of the rainwater gets mixed in with the foul water. At times of heavy or continued rainfall, the sewer system can't always cope with the extra volume.

Storm overflows are used to prevent sewers flooding our homes, gardens and streets. They act as a safety valve, diverting some of the rainwater and foul water into watercourses. Thames Water call this mixture 'storm discharge'. Although it does contain some untreated sewage, storm discharge is heavily diluted because it's mostly rainwater.

Storm overflows are regulated by the Environment Agency. Storm discharges are legally allowed, under the conditions of the Environment Agency permit. Storm overflows should only be used when necessary. Thames Water don't actively switch them on, they operate automatically when the flow levels increase.

This is an interactive dashboard that shows the discharge activity at permitted locations by Thames Water.

If you want to see the more updated version of the dashboard click on this link. The passcode to access the dashboard is m8TUHZh92p%L

Why I did it?

I've always found it rather remarkable that we can simply turn on a tap and have fresh, clean drinking water at our disposal. It really hits home when you travel beyond Europe and are warned against drinking tap water - it makes you wonder: how is it that something as fundamental and vital as water isn't a guaranteed basic service worldwide? It raises all sorts of questions, doesn't it? Where does our water actually come from? Why do we sometimes face shortages or, conversely, have too much? How is it stored and distributed throughout our communities?

I have always been interested in understanding how this works.

Some history...

If you dig deeper in the history of Europe, the Roman Empire is credited with building extensive water distribution systems in Europe through the use of aqueducts and related infrastructure. These systems, which included pipes, fountains, and public wells, were instrumental in supplying water to cities and private residences.

What’s particularly fascinating is how advanced Roman engineering was for its time. Their aqueducts, often stretching for miles across valleys and mountains, were built with such precision that water could flow steadily using only gravity. This feat alone is impressive, but even more so is how these systems laid the groundwork for the water infrastructure we benefit from today. The Romans didn’t just focus on transportation—they also developed methods for storage, like cisterns and reservoirs, and even integrated public health features such as public baths and latrines, emphasising hygiene in urban life centuries ahead of its time.

However, the Roman water system wasn’t without its flaws. In distributing water within cities, the Romans often used lead pipes, known as fistulae. At the time, lead was valued for its durability and malleability. Yet, we now understand the serious health risks associated with lead exposure. While some historians argue that mineral buildup inside the pipes may have reduced lead contamination in the water, others suggest that prolonged exposure—especially through other uses of lead, such as in cookware and wine preservation—could have contributed to chronic health problems among the Roman elite. It’s a stark reminder that even the most impressive innovations can carry unintended consequences.

How it works today?

Today, London’s water supply is managed primarily by Thames Water, which sources about 70% of its water from the River Thames and the rest from groundwater. After abstraction, the water undergoes extensive treatment to remove contaminants before being pumped into the distribution network. However, London’s aging infrastructure still struggles under modern pressures—especially when it comes to wastewater. The city uses a combined sewer system, meaning rainwater and sewage flow through the same pipes. During heavy rainfall, the system can’t cope, leading to overflows of untreated sewage directly into the Thames. To tackle this, the city is currently constructing the Thames Tideway Tunnel, a massive underground sewer designed to intercept and store overflow water until it can be properly treated—an ambitious project aimed at protecting both the river and public health.

In contrast, Madrid benefits from a more modern and separate water and wastewater system, overseen by Canal de Isabel II. The city's drinking water comes primarily from reservoirs in the Sierra de Guadarrama mountains and is treated in advanced purification plants before distribution. Madrid’s sewer network is separated, meaning rainwater and wastewater generally travel through different systems, significantly reducing the risk of overflow contamination. Additionally, the city has invested heavily in water recycling and stormwater control, making its system more resilient to climate change and sudden downpours. While challenges remain, Madrid’s infrastructure offers a model of how thoughtful, updated systems can manage water sustainably in a major urban area.

By using Thames Water's API, we can monitor sewage discharge locations, their duration, and other related data across the network.

How I did it?

Well, the first step is to create an account in Thames Water Open Data API website .

In particular we will use the Discharge To Environment (DTE) Data.

Discharge To Environment (DTE) Data

The DTE resource provides data on storm discharge activity at permitted locations by Thames Water, as measured by event duration monitoring (EDM). There's more information on our river health pages. There are two endpoints provided for access to DTE data.

We are using the endpoint 2, DTE - Current Status

The DTE Current Status resource provides details about all of our permitted discharge locations, along with a snapshot of the current storm discharge activity at each of them. As an example, you might use this resource to find out which sites are discharging now, or have done within the last 48 hours. This resource is also used to populate our storm discharge map. Take a look at our frequently asked questions for more information on storm discharges and EDM.

Source code

This is the source code used to gather the data from the API. After the data is retrieved, we use the library osgridref.js to convert coordinates received from the API in WGS84 format (World Geodetic System 1984 (WGS84) to latitude and longitude to be compatible with MongoDB.

After inserting the data, we run a MongoDB aggregation to process the data and then we create a MongoDB Atlas Chart to draw all the entries.

import OsGridRef, { LatLon } from 'geodesy/osgridref.js';
exports = async function (arg) {


  // Find the name of the MongoDB service you want to use
  var serviceName = "mongodb-atlas";

  // Update these to reflect your db/collection
  var dbName = "ThamesWater";
  var collName = "thameswaterTemp";
  var collName2 = "ThamesWater";

  // Get a collection from the context
  var collection = context.services.get(serviceName).db(dbName).collection(collName);
  var collection2 = context.services.get(serviceName).db(dbName).collection(collName2);

  //Get the variables to call the API
  var client_id = context.values.get("thames_water_client_id");
  var client_secret = context.values.get("thames_water_client_secret_value");
  var url = "https://prod-tw-opendata-app.uk-e1.cloudhub.io/data/STE/v1/DischargeCurrentStatus";
  let headers = {
    "Content-Type": ["application/json"],
    "client_id": ['id1'],
    "client_secret": ['secret1']
  }
  let getArguments = {
    "url": url,
    "headers": headers
  };
  const response = await context.http.get(getArguments);
  let parsedResponse = JSON.parse(response.body.text());
  let result = parsedResponse.items;
  result.forEach(element => {
    let X = element.X;
    let Y = element.Y;
    let gridref = new OsGridRef(X, Y);
    let pWgs84 = gridref.toLatLon();
    let latitude = pWgs84.lat;
    let longitude = pWgs84.lon;
    let coordinatesObj = {
      "type": "Point",
      "coordinates": [longitude, latitude]
    }
    element.coordinates = coordinatesObj;

  });

  let aggregation = [
  {
    '$unset': [
      'X', 'Y'
    ]
  }, {
    '$set': {
      'MostRecentDischargeAlertStop': {
        '$ifNull': [
          '$MostRecentDischargeAlertStop', new Date()
        ]
      }
    }
  }, {
    '$set': {
      'StatusChange': {
        '$toDate': '$StatusChange'
      }, 
      'MostRecentDischargeAlertStart': {
        '$toDate': '$MostRecentDischargeAlertStart'
      }, 
      'MostRecentDischargeAlertStop': {
        '$toDate': '$MostRecentDischargeAlertStop'
      }, 
      'AlertPast48Hours': {
        '$toBool': '$AlertPast48Hours'
      }
    }
  }, {
    '$set': {
      'MinutesPolluting': {
        '$dateDiff': {
          'startDate': '$MostRecentDischargeAlertStart', 
          'endDate': '$MostRecentDischargeAlertStop', 
          'unit': 'minute'
        }
      }
    }
  }, {
    '$merge': 'ThamesWater'
  }
];

  var insertResult;
  var deleteResult;
  var aggregateResult;
  try {
    await collection2.deleteMany({});
    deleteResult = await collection.deleteMany({});
    insertResult = await collection.insertMany(result);
    aggregateResult = await collection.aggregate(aggregation).toArray();
    deleteResult = await collection.deleteMany({});


  } catch (err) {
    console.log("Error occurred while executing findOne:", err.message);

    return { error: err.message };
  }

  // To call other named functions:
  // var result = context.functions.execute("function_name", arg1, arg2);

  return { result: aggregateResult };
};