TDengine has invested significant effort in optimizing its schemaless ingestion functionality to enhance both flexibility and efficiency. This article provides an overview of some of these optimizations, giving developers insights into how they can leverage this functionality to boost application performance. Our goal is to help developers reach new heights in performance optimization.<\/p>\n\n\n\n
Performance Optimization Process<\/h2>\n\n\n\nIdentifying Performance Bottlenecks<\/h3>\n\n\n\n
By analyzing flame graphs of data ingestion, we can pinpoint the areas where the system is spending the most time. Specifically, functions like For each of these functions, we can adopt one of two strategies to improve performance: either eliminate the bottleneck entirely or reduce its duration.<\/p>\n\n\n\n How can we eliminate these bottlenecks?<\/strong><\/p>\n\n\n\n To answer this question, we first need to understand the current data parsing framework. Below is a simplified flowchart of how the existing framework processes data:<\/p>\n\n\n\n In the current process, each record is iterated over, extracting the measurement (measure), tags, and columns (col) key-value pairs. These are stored in custom structures, and the system sorts the tag keys and generates corresponding sub-table names based on predefined rules. However, the repeated parsing, sorting of tags, and subtable name generation in this workflow add unnecessary time and computational complexity.<\/p>\n\n\n\n After acquiring the schema metadata for a measurement, the system must check whether the schema needs to be updated. This involves traversing the tags and columns of each data record and determining if operations like For datasets with fewer than 10 records, the system constructs SQL statements individually and inserts them via the The following code snippets illustrate these key functions:<\/p>\n\n\n\n You can find the detailed code in the following GitHub links:<\/p>\n\n\n\n Now, let’s dive into the optimization of the architecture.<\/p>\n\n\n\n Analysis of the data shows that tags with the same prefix often remain consistent. Therefore, we can pre-group the tag strings, placing tags with the same prefix into the same group. Within each group, we only need to parse the tags once, reducing redundant parsing operations.<\/p>\n\n\n\n Additionally, by checking if the data is already ordered, we can bind the data directly in sequence, avoiding the overhead of hash lookups. This reduces computational costs and accelerates the process.<\/p>\n\n\n\n During parsing, we should check whether the schema requires modification, such as adding new columns or adjusting column lengths. If no changes are needed, we bypass the schema modification logic, further improving efficiency.<\/p>\n\n\n\n Data construction is carried out directly during the parsing of each line, binding the data straight to the final Moreover, we can bypass SQL or Below are code snippets from key functions involved in these optimizations:<\/p>\n\n\n\n You can view the full code in the following GitHub links:<\/p>\n\n\n\n We also identified areas where memory usage could be further optimized during data parsing.<\/p>\n\n\n\n In the process of converting data to a schemaless format, numerous memory allocations and copies are made for measurements, tags, and columns. While these steps are common, they represent potential optimization opportunities.<\/p>\n\n\n\n Instead of directly copying and allocating memory during parsing, we can record the pointer locations and lengths of each data item in the original dataset. When the data is ready to be written to the database, we can copy the data from the original dataset based on the recorded pointers.<\/p>\n\n\n\n For example, instead of allocating memory immediately for Alternatively, we can use pointers to avoid unnecessary memory allocation:<\/p>\n\n\n\n By directly accessing memory locations rather than performing conditional checks, we can significantly reduce processing time. For example, when converting time precision, we can avoid checking multiple conditions and instead directly access the appropriate unit.<\/p>\n\n\n\n Since the JSON format for ingestion is usually fixed, we can precompute the offsets for elements like metrics, timestamps, values, and tags. This allows for quicker data handling during parsing.<\/p>\n\n\n\n To minimize entropy in each check, place the most likely In some cases, using the Additionally, excessive logging, especially at high frequencies, can cause performance degradation. We observed that frequent logging increases processing time by about 10%.<\/p>\n\n\n\n In the comparison between TDengine 3.0 and 2.6(the speed is measured in records\/second), we observed a notable improvement in performance:<\/p>\n\n\n\n To analyze performance, we used tools like flame graphs<\/strong> and perf<\/strong>. The Before undertaking performance optimization, it is essential to thoroughly understand the system’s architecture and flow. Performance improvements should focus on three key areas:<\/p>\n\n\n\n By considering factors like CPU usage, memory, network, compiler optimizations, and hardware constraints, you can achieve substantial performance gains. We hope this article provides valuable insights for developers working with TDengine’s schemaless ingestion and helps you unlock higher performance in your applications.<\/p>\n","protected":false},"excerpt":{"rendered":" TDengine has optimized the Schemaless ingestion feature to improve flexibility and efficiency.<\/p>\n","protected":false},"author":81,"featured_media":23475,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[21],"tags":[],"ppma_author":[167],"class_list":["post-23457","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering"],"authors":[{"term_id":167,"user_id":81,"is_guest":0,"slug":"chait","display_name":"Chait Diwadkar","avatar_url":{"url":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg","url2x":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg"},"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/23457","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/comments?post=23457"}],"version-history":[{"count":1,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/23457\/revisions"}],"predecessor-version":[{"id":23472,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/23457\/revisions\/23472"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media\/23475"}],"wp:attachment":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media?parent=23457"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/categories?post=23457"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/tags?post=23457"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/ppma_author?post=23457"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}parseSmlKey<\/code>, parseSmlValue<\/code>, addChildTableDataPointsIntoSql<\/code>, taos_query_a<\/code>, and buildDataPointSchema<\/code> are responsible for a high proportion of the processing time.<\/p>\n\n\n\n![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless1-1024x494.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\nAddressing Performance Bottlenecks<\/h3>\n\n\n\n
![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/whiteboard_exported_image-6-1024x444.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\nAnalysis of the Current Framework<\/h3>\n\n\n\n
Schema Processing<\/h5>\n\n\n\n
create stable<\/code>, add col<\/code>, add tag<\/code>, modify col<\/code>, or modify tag<\/code> are required.<\/p>\n\n\n\nData Insertion<\/h5>\n\n\n\n
taos_query<\/code> function. For larger datasets, a batch insert approach is adopted using stmt<\/code> structures, which allows for more efficient bulk insertion.<\/p>\n\n\n\n\n
tscParseLine<\/code>, parseSmlKvPairs<\/code>, tscSmlInsert<\/code>, buildDataPointSchemas<\/code>, modifyDBSchemas<\/code>, applyDataPointsWithSqlInsert<\/code><\/li>\n<\/ul>\n\n\n\n![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless3-1024x642.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\n\n
![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/whiteboard_exported_image-7-1024x510.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\nMajor Optimizations<\/h3>\n\n\n\n
\n
\n
\n
STableDataCxt<\/code> structure. This approach eliminates the need for data binding and copying in subsequent stages.<\/p>\n\n\n\nstmt<\/code> interfaces, directly constructing BuildRow<\/code> data to avoid the extra step of second-pass parsing.<\/p>\n\n\n\n\n
smlParseInfluxString<\/code>, smlParseTagKv<\/code>, smlParseColKv<\/code>, smlParseLineBottom<\/code>, smlModifyDBSchemas<\/code>, smlInsertData<\/code><\/li>\n<\/ul>\n\n\n\n![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless4-1024x845.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\n![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless5-1024x710.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\n\n
Memory Optimization<\/h3>\n\n\n\n
t1\/t2\/t3\/c1\/c3\/c2<\/code>, we can directly record their pointer locations and perform the memory copy only when needed.<\/p>\n\n\n\nCode Optimization Example:<\/h5>\n\n\n\n
st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4 1626006833639000000<\/code><\/pre>\n\n\n\ntypedef struct {\n char* key;\n uint8_t type;\n uint16_t length;\n char* value;\n uint32_t fieldSchemaIdx;\n} TAOS_SML_KV;<\/code><\/pre>\n\n\n\n![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless6.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\ntypedef struct {\n char *measure;\n char *tags;\n char *cols;\n char *timestamp;\n char *measureTag;\n int32_t measureLen;\n int32_t measureTagsLen;\n int32_t tagsLen;\n int32_t colsLen;\n int32_t timestampLen;\n SArray colArray;\n} SSmlLineInfo;<\/code><\/pre>\n\n\n\n![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless7-1024x470.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\nTime Precision Conversion Optimization<\/h4>\n\n\n\n
if(fromPrecision == ){}\nelse if( fromPrecision == ){}\nelse {}\n\nint64_t smlToMilli[3] = {3600000LL, 60000LL, 1000LL};\nint64_t unit = smlToMilli[fromPrecision - TSDB_TIME_PRECISION_HOURS];\nif (unit > <strong>INT64_MAX <\/strong>\/ tsInt64) {\n return -1;\n}\ntsInt64 *= unit;<\/code><\/pre>\n\n\n\nJSON Parsing Optimization<\/h4>\n\n\n\n
[\n{ \n "metric": "meter_current",\n "timestamp" : 1346846400,\n "value": 18,\n "tags": {\n "location": "LosAngeles",\n "id": "d1001"\n } \n}\n]<\/code><\/pre>\n\n\n\nConditional Logic Optimization<\/h4>\n\n\n\n
if<\/code> conditions first. For example: i64 \/ u64 \/ i8 \/ u8 \/ true \/ L””<\/p>\n\n\n\nif( str equals "i64"){}\nelse if( str equals "i32"){}\nelse if( str equals "u8"){}\n\u00b7\u00b7\u00b7\n\nif(str[0] equals "i"){}\nelse if(str[0] equals "u"){}\n...<\/code><\/pre>\n\n\n\nOther Optimizations<\/h4>\n\n\n\n
likely<\/code> and unlikely<\/code> keywords can guide the compiler to optimize the instruction pipeline for better performance.<\/p>\n\n\n\nif(unlikely(<em>len <\/em>== 0 || (<em>len <\/em>== 1 && <em>data<\/em>[0] == '0'))){\n return taosGetTimestampNs()\/smlFactorNS[toPrecision];\n}<\/code><\/pre>\n\n\n\nPerformance Comparison<\/h2>\n\n\n\n
\n\n
\n\t \nVersion<\/th> SQL<\/th> Line Protocol<\/th> Telnet Protocol<\/th> JSON Protocol<\/th>\n<\/tr>\n<\/thead>\n \n\t 2.6(4ec22e8)<\/td> 4543622<\/td> 1458304<\/td> 2161855<\/td> 1272000<\/td>\n<\/tr>\n \n\t ver-3.0.0.0<\/td> 1638498<\/td> 1650033<\/td> 1945982<\/td> 800000<\/td>\n<\/tr>\n \n\t 3.0(f6793d5)<\/td> 3740774<\/td> 3602947<\/td> 4328447<\/td> 5520000<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n \n
Performance Analysis Tools<\/h2>\n\n\n\n
![](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless8-1024x479.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\nperf top -p<\/code> command helps us identify high CPU usage areas and optimize them further.<\/p>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless9-1.jpg?strip=all&sharp=1\" "\\"schemaless9\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/schemaless10-1.jpg?strip=all&sharp=1\" "\\"schemaless10\\"")
<\/figure>\n\n\n\nConclusion<\/h2>\n\n\n\n
\n
perf<\/code> to accurately identify areas for improvement.<\/li>\n<\/ol>\n\n\n\n