{"id":21187,"date":"2024-05-31T00:56:21","date_gmt":"2024-05-31T07:56:21","guid":{"rendered":"https:\/\/tdengine.com\/?p=21187"},"modified":"2025-03-30T23:31:35","modified_gmt":"2025-03-31T06:31:35","slug":"mastering-memory-leak-detection-in-tdengine","status":"publish","type":"post","link":"https:\/\/tdengine.com\/mastering-memory-leak-detection-in-tdengine\/","title":{"rendered":"Mastering Memory Leak Detection in TDengine"},"content":{"rendered":"\n<p>Memory leaks are a common issue that can cause a program&#8217;s memory usage to gradually increase, eventually leading to the exhaustion of system resources or program crashes. Tools like AddressSanitizer (ASan) and Valgrind are excellent for memory detection, and TDengine&#8217;s CI process uses ASan. However, this memory leak issue occurred on Windows, which our CI currently doesn&#8217;t cover. Thus, the TDengine development team chose Windbg to tackle the problem. The results show that Windbg is also a good choice for dealing with memory leaks on Windows.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Common Methods for Detecting Memory Leaks<\/h2>\n\n\n\n<p><strong>Memory leaks typically occur in the following scenarios:<\/strong><\/p>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li>The program does not properly release allocated memory.<\/li>\n\n\n\n<li>Circular references exist, preventing the garbage collector from reclaiming memory.<\/li>\n\n\n\n<li>Third-party libraries or components with memory leaks.<\/li>\n<\/ol>\n\n\n\n<p><strong>The main methods for detecting memory leaks include:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Static Code Analysis Tools<\/strong>: These can detect unfreed pointers or allocation errors but can&#8217;t detect dynamic memory allocation issues at runtime.<\/li>\n\n\n\n<li><strong>Dynamic Analysis Tools<\/strong>: Tools like Valgrind track memory allocation and deallocation during runtime but might affect program performance.<\/li>\n\n\n\n<li><strong>Debuggers<\/strong>: Tools such as WinDbg and GDB.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Advantages and Disadvantages<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Static Code Analysis<\/strong>: Effective for early detection but not for runtime dynamic allocation issues.<\/li>\n\n\n\n<li><strong>Dynamic Analysis Tools<\/strong>: Effective during runtime but might impact performance and require significant resources for large applications. However, in resource-rich test environments, these issues are mitigated; ASan has helped us identify numerous issues.<\/li>\n\n\n\n<li><strong>Debuggers<\/strong>: Detect issues during runtime and offer powerful analysis tools.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Practical Analysis<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Basic Principle<\/h3>\n\n\n\n<p>Using Windbg to locate memory leaks relies on the gflags component to record all memory allocations and deallocations during program execution, along with the call stack information for these operations. By taking two snapshots with the umdh component during program execution and comparing them, we can identify memory allocations that were not freed. If there is a memory leak, the call stack information for the leak point will usually be at the top of the diff result. The key is to trigger the memory leak as much as possible between the two snapshots for accurate location. The diff results will also include some normal allocations that weren&#8217;t released yet, but the frequency of calls makes it easy to identify leaks.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Problem Introduction<\/h3>\n\n\n\n<p>Taosdump encountered an error on Windows while importing data:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">build and install latest TDengine 3.0 branch on Windows\nuse &quot;taosBenchmark -I stmt -y&quot; to create a lot of tables and data (10000 * 10000).\nuse &quot;taosdump -D test -o outputFile&quot; to dump out\nuse &quot;taos -s &#039;drop database test&#039;&quot; to drop database\nuse &quot;taosdump -i inputFile&quot; to dump in.<\/code><\/pre>\n\n\n\n<p>Error log: <code class=\"\" data-line=\"\">taosd \u201ctsem_init failed, errno: 28\u201d<\/code><\/p>\n\n\n\n<p><code class=\"\" data-line=\"\">Taosdump: dumpInAvroDataImpl() LN7039 taos_stmt_execute() failed! reason: Out of Memory, timestamp: 1500000009256<\/code><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Troubleshooting Process<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Configuring gflags<\/h4>\n\n\n\n<p>The gflags tool should be located at: <code class=\"\" data-line=\"\">C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\gflags<\/code>. If it&#8217;s not available, download it from Microsoft&#8217;s official website: <a href=\"https:\/\/learn.microsoft.com\/en-us\/windows-hardware\/drivers\/debugger\/debugger-download-tools\" rel=\"noopener\">Debugger Download Tools<\/a>.<\/p>\n\n\n\n<p>After installation, run <code class=\"\" data-line=\"\">gflags.exe \/i your_application.exe<\/code> from the command line to set the tracking target and related parameters. Alternatively, double-click to run it, set the Image File to your application, press the Tab key, and select other configurations.<\/p>\n\n\n\n<figure class=\"gb-block-image gb-block-image-1403a81a\"><img decoding=\"async\" width=\"541\" height=\"589\" class=\"gb-image gb-image-1403a81a\" src=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-1.jpeg?strip=all&sharp=1\" srcset=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-1.jpeg?strip=all&amp;sharp=1 541w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-1-276x300.jpeg?strip=all&amp;sharp=1 276w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-1.jpeg?strip=all&amp;sharp=1&amp;w=108 108w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-1.jpeg?strip=all&amp;sharp=1&amp;w=216 216w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-1.jpeg?strip=all&amp;sharp=1&amp;w=432 432w\" sizes=\"(max-width: 541px) 100vw, 541px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Steps to Locate the Leak<\/h3>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li>Start your_application.exe (I need to debug taosdump.exe, so the following steps are for taosdump.exe). <br><code class=\"\" data-line=\"\">&quot;C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\gflags&quot; -i taosdump.exe +ust<\/code><\/li>\n<\/ol>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li>Copy the pdb file to the mysymbols directory. The pdb file contains the debug information of the compiled program and is generated along with the executable file.<\/li>\n\n\n\n<li>Set pdb directory:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">set _NT_SYMBOL_PATH=c:\\mysymbols;srv*c:\\mycache*https:\/\/msdl.microsoft.com\/download\/symbols<\/code><\/pre>\n\n\n\n<p>    4. Generate the first memory snapshot:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><\/li>\n<\/ol>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li><\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">&quot;C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\umdh&quot; -pn:taosdump.exe -f:C:\\xstest\\umdhlog\\taosdump11.log<\/code><\/pre>\n\n\n\n<p>    5. Generate the second memory snapshot:<\/p>\n\n\n\n<ol start=\"4\" class=\"wp-block-list\">\n<li><\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">&quot;C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\umdh&quot; -pn:taosdump.exe -f:C:\\xstest\\umdhlog\\taosdump12.log<\/code><\/pre>\n\n\n\n<p>    6. Generate the snapshot comparison result (umdh):<\/p>\n\n\n\n<ol start=\"5\" class=\"wp-block-list\">\n<li><\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">&quot;C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\umdh&quot;  C:\\xstest\\umdhlog\\taosdump11.log C:\\xstest\\umdhlog\\taosdump12.log -f:C:\\xstest\\umdhlog\\taosdumpdiff11_12.log<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Analysis and Solution<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Result File<\/h3>\n\n\n\n<p>Since the taosdump program does a lot of work from start to finish, memory leaks easily occur between the two snapshots. <code class=\"\" data-line=\"\">988040 \u2013 6ecf0<\/code> indicates &#8220;allocation count \u2013 release count,&#8221; showing a clear memory leak at the <code class=\"\" data-line=\"\">buildRequest<\/code> function&#8217;s <code class=\"\" data-line=\"\">sem_init<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">+  919350 ( 988040 - 6ecf0)  201b0 allocs        BackTrace9CB6973F\n+   1ea5c ( 201b0 -  1754)        BackTrace9CB6973F        allocations\n\n        ntdll!RtlpAllocateHeapInternal+948D5\n        taos!heap_alloc_dbg_internal+1F6 (minkernel\\crts\\ucrt\\src\\appcrt\\heap\\debug_heap.cpp, 359)\n        taos!heap_alloc_dbg+4D (minkernel\\crts\\ucrt\\src\\appcrt\\heap\\debug_heap.cpp, 450)\n        taos!_calloc_dbg+6C (minkernel\\crts\\ucrt\\src\\appcrt\\heap\\debug_heap.cpp, 518)\n        taos!calloc+2E (minkernel\\crts\\ucrt\\src\\appcrt\\heap\\calloc.cpp, 30)\n        taos!sem_init+5D (C:\\workroom\\TDengine\\contrib\\pthread\\sem_init.c, 109)\n        taos!buildRequest+209 (C:\\workroom\\TDengine\\source\\client\\src\\clientImpl.c, 192)\n        taos!stmtCreateRequest+73 (C:\\workroom\\TDengine\\source\\client\\src\\clientStmt.c, 15)\n        taos!stmtSetTbName+115 (C:\\workroom\\TDengine\\source\\client\\src\\clientStmt.c, 588)\n        taos!taos_stmt_set_tbname+7F (C:\\workroom\\TDengine\\source\\client\\src\\clientMain.c, 1350)\n        taosdump!dumpInAvroDataImpl+E25 (C:\\workroom\\TDengine\\tools\\taos-tools\\src\\taosdump.c, 6260)\n        taosdump!dumpInOneAvroFile+3D2 (C:\\workroom\\TDengine\\tools\\taos-tools\\src\\taosdump.c, 7229)\n        taosdump!dumpInAvroWorkThreadFp+20B (C:\\workroom\\TDengine\\tools\\taos-tools\\src\\taosdump.c, 7306)\n        taosdump!ptw32_threadStart+CD (C:\\workroom\\TDengine\\contrib\\pthread\\ptw32_threadStart.c, 233)\n        taosdump!thread_start&lt;unsigned int (__cdecl*)(void *),1&gt;+9C (minkernel\\crts\\ucrt\\src\\appcrt\\startup\\thread.cpp, 97)\n        KERNEL32!BaseThreadInitThunk+10\n        ntdll!RtlUserThreadStart+2B<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Fixing the Leak<\/h3>\n\n\n\n<p>Next, examine and modify the code. In C language, memory management is flexible but can be cumbersome. It\u2019s evident that some paths missed calling <code class=\"\" data-line=\"\">tsem_destroy<\/code>.<\/p>\n\n\n\n<figure class=\"gb-block-image gb-block-image-ee24b4c5\"><img decoding=\"async\" width=\"705\" height=\"1024\" class=\"gb-image gb-image-ee24b4c5\" src=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-2.png?strip=all&sharp=1\" srcset=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-2.png?strip=all&amp;sharp=1 705w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-2-207x300.png?strip=all&amp;sharp=1 207w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-2.png?strip=all&amp;sharp=1&amp;w=141 141w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-2.png?strip=all&amp;sharp=1&amp;w=282 282w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-2.png?strip=all&amp;sharp=1&amp;w=423 423w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/Windbg-2.png?strip=all&amp;sharp=1&amp;w=564 564w\" sizes=\"(max-width: 705px) 100vw, 705px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>To do a good job, one must first sharpen their tools. Mastering more tools and methods enables you to handle issues more confidently. Using Windbg to locate memory leaks is very simple yet effective. However, it relies on pdb files, so remember to keep pdb files when releasing applications. These files contain the program&#8217;s symbol information, helping pinpoint issues accurately during debugging.<\/p>\n\n\n\n<p>Additionally, the problematic code shows that the memory management method is prone to errors. The RAII (Resource Acquisition Is Initialization) mechanism can effectively prevent resource leaks. Though C language doesn\u2019t offer RAII as smoothly as C++, it can be simulated for similar effects, and considering optimization in the future could be beneficial.<\/p>\n\n\n\n<p>RAII is a critical resource management technique that associates resource acquisition with object lifecycle. By acquiring resources in the constructor and releasing them in the destructor, it ensures proper resource management, preventing leaks. This mechanism is widely used in C++ and other languages, proving to be an effective resource management strategy.<\/p>\n\n\n\n<p>For more detailed code solutions, see <a href=\"https:\/\/github.com\/taosdata\/TDengine\/pull\/19580\" rel=\"noopener\">TDengine PR #19580<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>To do a good job, one must first sharpen their tools. Mastering more tools and methods enables you to handle issues more confidently.<\/p>\n","protected":false},"author":4,"featured_media":21191,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[21],"tags":[],"ppma_author":[107],"class_list":["post-21187","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering"],"authors":[{"term_id":107,"user_id":4,"is_guest":0,"slug":"slguan","display_name":"Shengliang Guan","avatar_url":{"url":"https:\/\/tdengine.com\/wp-content\/uploads\/29.04-30-slguan.jpg","url2x":"https:\/\/tdengine.com\/wp-content\/uploads\/29.04-30-slguan.jpg"},"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/21187","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/comments?post=21187"}],"version-history":[{"count":3,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/21187\/revisions"}],"predecessor-version":[{"id":24725,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/21187\/revisions\/24725"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media\/21191"}],"wp:attachment":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media?parent=21187"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/categories?post=21187"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/tags?post=21187"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/ppma_author?post=21187"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}