{"id":7062,"date":"2022-07-24T23:47:41","date_gmt":"2022-07-25T06:47:41","guid":{"rendered":"https:\/\/tdengine.com\/?p=7062"},"modified":"2025-11-03T08:09:51","modified_gmt":"2025-11-03T16:09:51","slug":"environmental-monitoring-and-data-governance-at-scale","status":"publish","type":"post","link":"https:\/\/tdengine.com\/environmental-monitoring-and-data-governance-at-scale\/","title":{"rendered":"Environmental Monitoring and Data Governance at Scale"},"content":{"rendered":"\n

GESRI created a project that would help enterprises use big data technology to manage their environmental management and governance goals and comply with environmental regulations. Instead of leaving enterprises to create their own platform to solve the issues of data governance, quality management and associated tasks like data cleaning, ETL, aggregation and so on, GESRI’s platform would provide the basis for accurate governance and intelligent decision making.<\/p>\n\n\n\n

The real-time data warehouse structure of this project is shown in the figure below. The status of environmental quality and real-time pollution discharge, in air, surface water, coastal waters, fixed and mobile pollution sources, can be captured in real-time. The real-time monitoring data can be further analyzed for early warning of possible harmful environmental events.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Establishing a real-time data warehouse of environmental data and exposing centralized collection, aggregation and analysis services, is more efficient than having various organizations build the same software platforms. This promotes software reuse and also promotes best practices and standards.<\/p>\n\n\n\n

From a relational database to TDengine<\/h2>\n\n\n\n

Since this is a real-time data warehouse, the database solution required for it has much higher performance requirements than those of a general database application. The storage layers must provide extremely high write and read performance with very low latency. Due to the vast amounts of data, it must also provide very efficient and compact storage.<\/p>\n\n\n\n

In our case, there are nearly 5000 monitoring stations and each station monitors 3-5 factors of key pollutant discharge units every 30 seconds. For many years, we used a relational databases for data storage and were only able to keep data for 3-5 days and had to delete old data on a daily basis. We considered using PostgreSQL’s TimescaleDB<\/a> extension, but it could not meet certain regulations having to do with monitoring and control of individual devices. We also tried to use another time-series database<\/a> (TSDB) product, but its performance and general interface were still far from our requirements.<\/p>\n\n\n\n

When we first came into contact with TDengine, we were a little hesitant because the product was so new. Fortunately, TDengine is easy to install and has complete documentation. Even though it is an open source product, product support is very comprehensive. There is also a dedicated technical group on social media. When problems arose, we were always able to get help from TDengine support staff and the TDengine community. Our entire evaluation process was very smooth. <\/strong>Finally, after a long period of research and testing, we chose the open source time series database, TDengine.<\/p>\n\n\n\n

Deployment and performance of TDengine<\/h2>\n\n\n\n

For the deployment of TDengine, we chose three, 4-core servers with 8G RAM each, one 8-core server with 16G RAM and one 8-core server with 32G RAM for a total of 5 servers. These servers were configured as a 5-node single-copy cluster. There are three business lines that have been put into operation. These include the following:<\/p>\n\n\n\n