![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Logstash-1.jpeg?strip=all&sharp=1\" "\\"Logstash-1\\"")
<\/figure>\n\n\n\nData flow is divided into three main parts: Input, Filter, and Output, covering the entire lifecycle of the data. Both Input and Output support codecs, allowing data to be encoded or decoded when it enters or exits the pipeline without needing a separate filter. Raw data is converted into Events in the Input phase, which are then transformed into target data in the Output phase. The configuration file allows for adding, deleting, modifying, and querying attributes within an Event.<\/p>\n\n\n\n The Filter component is the main reason for Logstash’s powerful functionality. It can perform extensive processing on Logstash Events, such as parsing data, deleting fields, and type conversion. Common filters include:<\/p>\n\n\n\n Example Grok filter configuration:<\/p>\n\n\n\n Logstash is suitable for real-time data processing but is not ideal for all scenarios. For large-scale time-series data processing, TDengine offers its own data integration functionality.<\/p>\n\n\n\n If you use Elasticsearch and Kibana for log management, data analysis, and visualization, Logstash might be a good choice due to its tight integration with Elasticsearch. However, Logstash is not suited for handling large-scale time-series data as it consumes significant system resources, including memory and CPU, and can suffer performance issues with large datasets. Logstash’s scalability can be limited when dealing with massive amounts of data. Moreover, Logstash requires a steep learning curve as it needs correct configuration for each stage of data processing (input, filter, output), and incorrect configurations can lead to data loss or format errors.<\/p>\n\n\n\n TDengine’s data integration functionality addresses these limitations, providing a more convenient and cost-effective solution for data conversion tasks if you are using TDengine.<\/p>\n\n\n\n Using TDengine’s data integration features, you can easily fetch data from MQTT and InfluxDB servers and efficiently write it into the TDengine database, ensuring smooth data integration and analysis. This feature automates the entire data integration process, minimizing manual operations and offering the following benefits:<\/p>\n\n\n\n After integrating data into TDengine, users can perform data cleaning and transformation based on business needs, achieving a complete ETL process. With these innovative features, real-time data can seamlessly integrate with the high-performance TDengine database, enabling real-time analysis, predictive maintenance, and data-driven decision-making.<\/p>\n\n\n\n The configuration process is straightforward: log in to the TDengine Enterprise or TDengine Cloud web management interface, select “Data in,” and add MQTT as a data source. Simply configure the parsing rules for InfluxDB\/MQTT data to correspond to TDengine’s database, super tables, and sub-tables.<\/p>\n\n\n\n TDengine 3.0 Enterprise Edition and TDengine Cloud provide efficient, reliable data integration methods with easy-to-use command-line operations. Whether you want to migrate data from InfluxDB\/MQTT or consolidate data from multiple sources into TDengine, TDengine 3.0 Enterprise Edition and TDengine Cloud can meet your needs.<\/p>\n","protected":false},"excerpt":{"rendered":" Easily convert and integrate data from MQTT and InfluxDB into TDengine, streamlining data conversion and boosting efficiency for real-time analysis and decision-making.<\/p>\n","protected":false},"author":83,"featured_media":20848,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[21],"tags":[],"ppma_author":[169],"class_list":["post-20847","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering"],"authors":[{"term_id":169,"user_id":83,"is_guest":0,"slug":"liaohaojun","display_name":"Haojun Liao","avatar_url":{"url":"https:\/\/tdengine.com\/wp-content\/uploads\/29.04-21-hjliao.jpg","url2x":"https:\/\/tdengine.com\/wp-content\/uploads\/29.04-21-hjliao.jpg"},"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/20847","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/users\/83"}],"replies":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/comments?post=20847"}],"version-history":[{"count":25,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/20847\/revisions"}],"predecessor-version":[{"id":24724,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/20847\/revisions\/24724"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media\/20848"}],"wp:attachment":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media?parent=20847"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/categories?post=20847"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/tags?post=20847"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/ppma_author?post=20847"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Logstash-2.png?strip=all&sharp=1\" "\\"Logstash-2\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Logstash-3.jpeg?strip=all&sharp=1\" "\\"Logstash-3\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Logstash-4.png?strip=all&sharp=1\" "\\"Logstash-4\\"")
<\/figure>\n\n\n\n\n
\n
\n
\n
filter {\n grok => {\n match => {\n "message" => "%{SERVICE:service}"\n }\n pattern_definitions => {\n "SERVICE" => "[a-z0-9]{10,11}"\n }\n }\n}<\/code><\/pre>\n\n\n\nTDengine: Data Collection for Multiple Sources<\/h2>\n\n\n\n
\n