{"id":17828,"date":"2023-09-18T04:16:52","date_gmt":"2023-09-18T11:16:52","guid":{"rendered":"https:\/\/tdengine.com\/?p=17828"},"modified":"2025-03-30T23:26:29","modified_gmt":"2025-03-31T06:26:29","slug":"tdengine-high-availability-cluster","status":"publish","type":"post","link":"https:\/\/tdengine.com\/tdengine-high-availability-cluster\/","title":{"rendered":"TDengine High Availability Cluster"},"content":{"rendered":"\n<h2 class=\"gb-headline gb-headline-df34efe7 gb-headline-text\">Introduction<\/h2>\n\n\n\n<p>In the last decade and especially in the post-COVID world, along with the ramifications of climate change, digital transformation has taken center stage as companies strive to become resilient to shocks to the supply chain, power outages and resource shortages. Optimizing manufacturing is a large part of it, as evident by the rise in projects related to Industry 4.0 and IIoT and the increasing use of ML and robotics. This has also led to an exponential increase in time-series data that needs to be ingested at very high frequencies, stored, analyzed and streamed to ML pipelines and to geographically dispersed stakeholders.<\/p>\n\n\n\n<p>This time-series data needs to be stored in purpose-built platforms that are equally resilient and can be deployed on highly available and scalable infrastructure.<\/p>\n\n\n\n<h2 class=\"gb-headline gb-headline-1eb19212 gb-headline-text\">TDengine: Natively Distributed and Scalable<\/h2>\n\n\n\n<p>TDengine, as a <a href=\"https:\/\/tdengine.com\/what-is-a-time-series-database\/\">time-series database<\/a> purpose-built for Industry 4.0 and Industrial IoT, is easily deployed on any platform to ensure high availability and scalability. In this blog we specifically show how TDengine can be deployed in cluster of VMs and remain operational even with the loss of a node.<\/p>\n\n\n\n<p>Firstly, it is important to note that the <a href=\"https:\/\/docs.tdengine.com\/inside-tdengine\/architecture\/\">logical architecture<\/a> of TDengine is itself designed to be distributed, highly available and scalable. It will be useful if you read about the distributed architecture of TDengine first, in order to follow this blog. Specifically, it will help to know the concepts of &#8220;dnodes&#8221;, &#8220;vnodes&#8221; and &#8220;mnodes&#8221;.<\/p>\n\n\n\n<h2 class=\"gb-headline gb-headline-f9a8e5a7 gb-headline-text\">Setting up a VM Cluster<\/h2>\n\n\n\n<p>For the purposes of this blog, I set up 3 Ubuntu VMs in VirtualBox locally, and demonstrate how to set up a TDengine cluster for high availability.<\/p>\n\n\n\n<p>If you already have other virtualization software, you can, of course, use that software and skip this section.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-14acd21e gb-headline-text\">Install VirtualBox<\/h3>\n\n\n\n<p>Firstly, you must install <a href=\"https:\/\/www.virtualbox.org\" rel=\"noopener\">VirtualBox<\/a>. Please follow the instructions for your particular host operating system.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-0432f37a gb-headline-text\">Create a NAT Network<\/h3>\n\n\n\n<figure class=\"gb-block-image gb-block-image-5e2b30e3\"><img decoding=\"async\" width=\"1235\" height=\"643\" class=\"gb-image gb-image-5e2b30e3\" src=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-1.png?strip=all&sharp=1\" alt=\"Click the NAT Networks tab and then specify a name in the General Options tab.\" srcset=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-1.png?strip=all&amp;sharp=1 1235w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-1-300x156.png?strip=all&amp;sharp=1 300w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-1-1024x533.png?strip=all&amp;sharp=1 1024w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-1-768x400.png?strip=all&amp;sharp=1 768w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-1.png?strip=all&amp;sharp=1&amp;w=247 247w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-1.png?strip=all&amp;sharp=1&amp;w=494 494w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><\/figure>\n\n\n\n<h3 class=\"gb-headline gb-headline-c79d5089 gb-headline-text\">Create VMs<\/h3>\n\n\n\n<p>For this blog we chose to create Ubuntu 22 VMs.<\/p>\n\n\n\n<p>For each VM we allocated 2G RAM and 1 CPU.<\/p>\n\n\n\n<p>We also created network adapters as follows in each VM.<\/p>\n\n\n\n<p>Note that you choose the NAT network that you created in in the previous step.<\/p>\n\n\n\n<figure class=\"gb-block-image gb-block-image-c3d6d94e\"><img decoding=\"async\" width=\"741\" height=\"514\" class=\"gb-image gb-image-c3d6d94e\" src=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-2.png?strip=all&sharp=1\" alt=\"From the menu, select Network. Click Enable Network Adapter. In the Attached to menu, select NAT Network. In the Name menu, select the name of the network that you created in the previous step.\" srcset=\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-2.png?strip=all&amp;sharp=1 741w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-2-300x208.png?strip=all&amp;sharp=1 300w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-2.png?strip=all&amp;sharp=1&amp;w=148 148w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-2.png?strip=all&amp;sharp=1&amp;w=444 444w, https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/image-2.png?strip=all&amp;sharp=1&amp;w=592 592w\" sizes=\"(max-width: 741px) 100vw, 741px\" \/><\/figure>\n\n\n\n<p><a href=\"https:\/\/ubuntu.com\/tutorials\/how-to-run-ubuntu-desktop-on-a-virtual-machine-using-virtualbox#1-overview\" rel=\"noopener\">See the Ubuntu website for information on how to create an Ubuntu VM in VirtualBox.<\/a><\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-e19ff855 gb-headline-text\">Update \/etc\/hosts files on each VM<\/h3>\n\n\n\n<p>You can run the following to get the IP address of each node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">tdengine@tdc01:~$ ifconfig\nenp0s3: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500\n        inet 10.0.2.4  netmask 255.255.255.0  broadcast 10.0.2.255\n        inet6 fe80::77b9:e21f:2466:22b2  prefixlen 64  scopeid 0x20&lt;link&gt;\n        ether 08:00:27:96:fb:81  txqueuelen 1000  (Ethernet)\n        RX packets 20760  bytes 4686683 (4.6 MB)\n        RX errors 0  dropped 0  overruns 0  frame 0\n        TX packets 10612  bytes 3444110 (3.4 MB)\n        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0<\/code><\/pre>\n\n\n\n<p>Set the hostname on each node if you have not already done so.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">sudo hostnamectl set-hostname &lt;hostname&gt;<\/code><\/pre>\n\n\n\n<p>Now you can add the hostname and the IP addresses of each node to the \/etc\/hosts file on each node.<\/p>\n\n\n\n<p>It should look something like the following. In my case, I&#8217;ve named the nodes tdc01, tdc02 and tdc03 and their IP addresses are dynamically assigned by the VirtualBox NAT Network that you created earlier. You can also choose static IP addresses if you want.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">127.0.0.1        localhost\n10.0.2.4        tdc01\n10.0.2.5        tdc03\n10.0.2.6        tdc02\n\n# The following lines are desirable for IPv6 capable hosts\n::1     ip6-localhost ip6-loopback\nfe00::0 ip6-localnet\nff00::0 ip6-mcastprefix\nff02::1 ip6-allnodes\nff02::2 ip6-allrouters<\/code><\/pre>\n\n\n\n<p>You should also ensure that each of the nodes can ping the other nodes using the hostnames that you have given them, after you edit the \/etc\/hosts files on each node.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-40d54225 gb-headline-text\">Deploy and start the cluster<\/h3>\n\n\n\n<p>Now we can start the process of creating and deploying a TDengine HA cluster.<\/p>\n\n\n\n<h4 class=\"gb-headline gb-headline-b295119a gb-headline-text\">Install the latest TDengine version on your first node<\/h4>\n\n\n\n<p>On this node you will leave the FQDN blank since you are setting up the cluster. This node will be the first EP (first End Point) of the cluster.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">tdengine@tdc01:~\/downloads$ sudo dpkg --install TDengine-server-3.1.0.0-Linux-x64.deb \nSelecting previously unselected package tdengine.\n(Reading database ... 186303 files and directories currently installed.)\nPreparing to unpack TDengine-server-3.1.0.0-Linux-x64.deb ...\nUnpacking tdengine (3.1.0.0) ...\nSetting up tdengine (3.1.0.0) ...\n.\/post.sh: line 107: \/var\/log\/taos\/tdengine_install.log: No such file or directory\nStart to install TDengine...\n.\/post.sh: line 107: \/var\/log\/taos\/tdengine_install.log: No such file or directory\n\nSystem hostname is: tdc01\n\nEnter FQDN:port (like h1.taosdata.com:6030) of an existing TDengine cluster node to join\nOR leave it blank to build one:\n\nEnter your email address for priority support or enter empty to skip: \nCreated symlink \/etc\/systemd\/system\/multi-user.target.wants\/taosd.service \u2192 \/etc\/systemd\/system\/taosd.service.\n\nTo configure TDengine : edit \/etc\/taos\/taos.cfg\nTo start TDengine     : sudo systemctl start taosd\nTo access TDengine    : taos -h tdc01 to login into TDengine server\n\n\nTDengine is installed successfully!\nProcessing triggers for man-db (2.10.2-1) ...<\/code><\/pre>\n\n\n\n<h4 class=\"gb-headline gb-headline-da397391 gb-headline-text\">Install TDengine on the remaining nodes<\/h4>\n\n\n\n<p>When you install TDengine on the remaining nodes, you must enter the hostname:port of the first node. In my case, it is: tdc01:6030 as shown below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">System hostname is: tdc02\n\nEnter FQDN:port (like h1.taosdata.com:6030) of an existing TDengine cluster node to join\nOR leave it blank to build one:tdc01:6030\n\nEnter your email address for priority support or enter empty to skip: \nCreated symlink \/etc\/systemd\/system\/multi-user.target.wants\/taosd.service \u2192 \/etc\/systemd\/system\/taosd.service.\n\nTo configure TDengine : edit \/etc\/taos\/taos.cfg\nTo start TDengine     : sudo systemctl start taosd\nTo access TDengine    : taos -h tdc01 -P 6030 to login into cluster, then\nexecute : create dnode &#039;newDnodeFQDN:port&#039;; to add this new node\n\n\nTDengine is installed successfully!\nProcessing triggers for man-db (2.10.2-1) ...\n<\/code><\/pre>\n\n\n\n<h4 class=\"gb-headline gb-headline-8e5d8688 gb-headline-text\">Add the nodes to the cluster<\/h4>\n\n\n\n<p>Please ensure that the <code class=\"\" data-line=\"\">taosd<\/code> service is running on each node.<\/p>\n\n\n\n<p>To start <code class=\"\" data-line=\"\">taosd<\/code> you can use the following:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">sudo systemctl start taosd<\/code><\/pre>\n\n\n\n<p>Then start the taos CLI on any node and add the nodes as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; create dnode &#039;tdc02:6030&#039;;\nCreate OK, 0 row(s) affected (0.009944s)\n\ntaos&gt; show dnodes;\n     id      |            endpoint            | vnodes | support_vnodes |   status   |       create_time       |       reboot_time       |              note              |\n====================================================================================================================================================================\n           1 | tdc01:6030                     |      0 |              4 | ready      | 2023-09-04 18:58:10.391 | 2023-09-04 18:58:10.367 |                                |\n           2 | tdc02:6030                     |      0 |              4 | ready      | 2023-09-04 19:01:21.746 | 2023-09-04 19:00:18.628 |                                |\nQuery OK, 2 row(s) in set (0.003692s)\n\ntaos&gt; create dnode &#039;tdc03:6030&#039;;\nCreate OK, 0 row(s) affected (0.010349s)\n\ntaos&gt; show dnodes;\n     id      |            endpoint            | vnodes | support_vnodes |   status   |       create_time       |       reboot_time       |              note              |\n====================================================================================================================================================================\n           1 | tdc01:6030                     |      0 |              4 | ready      | 2023-09-04 18:58:10.391 | 2023-09-04 18:58:10.367 |                                |\n           2 | tdc02:6030                     |      0 |              4 | ready      | 2023-09-04 19:01:21.746 | 2023-09-04 19:00:18.628 |                                |\n           3 | tdc03:6030                     |      0 |              4 | ready      | 2023-09-04 19:02:48.980 | 2023-09-04 19:02:40.144 |                                |\nQuery OK, 3 row(s) in set (0.001831s)\n<\/code><\/pre>\n\n\n\n<h4 class=\"gb-headline gb-headline-e1c2d785 gb-headline-text\">Check the number of mnodes<\/h4>\n\n\n\n<p>If you read the documentation on the distributed logical architecture of TDengine, you may recall that an &#8220;mnode&#8221; is a logical unit, part of taosd, that manages and monitors the running status of all data nodes (dnodes) and also manages and stores metadata. By default only 1 mnode is created.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; show mnodes;\n     id      |            endpoint            |     role     |  status   |       create_time       |        role_time        |\n==============================================================================================================================\n           1 | tdc01:6030                     | leader       | ready     | 2023-09-04 18:58:10.393 | 2023-09-04 18:58:10.385 |\nQuery OK, 1 row(s) in set (0.002263s)<\/code><\/pre>\n\n\n\n<h4 class=\"gb-headline gb-headline-9f137883 gb-headline-text\">Add mnodes<\/h4>\n\n\n\n<p>For HA, we must add 2 more mnodes on dnodes 2 and 3.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; create mnode on dnode 2;Create OK, 0 row(s) affected (4.062746s)taos&gt; show mnodes;     id      |            endpoint            |     role     |  status   |       create_time       |        role_time        |==============================================================================================================================           1 | tdc01:6030                     | leader       | ready     | 2023-09-04 18:58:10.393 | 2023-09-04 18:58:10.385 |           2 | tdc02:6030                     | follower     | ready     | 2023-09-04 20:23:33.318 | 2023-09-04 20:23:37.365 |Query OK, 2 row(s) in set (0.002233s)<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; create mnode on dnode 3;\nCreate OK, 0 row(s) affected (4.077106s)\n\ntaos&gt; show mnodes;\n     id      |            endpoint            |     role     |  status   |       create_time       |        role_time        |\n==============================================================================================================================\n           1 | tdc01:6030                     | leader       | ready     | 2023-09-04 18:58:10.393 | 2023-09-04 18:58:10.385 |\n           2 | tdc02:6030                     | follower     | ready     | 2023-09-04 20:23:33.318 | 2023-09-04 20:23:37.365 |\n           3 | tdc03:6030                     | follower     | ready     | 2023-09-04 20:24:38.435 | 2023-09-04 20:24:42.501 |\nQuery OK, 3 row(s) in set (0.004268s)<\/code><\/pre>\n\n\n\n<h2 class=\"gb-headline gb-headline-e9a49f36 gb-headline-text\">Demonstrate High Availability<\/h2>\n\n\n\n<p>Now we use the tool taosBenchmark to create a database. By default taosBenchmark will create a very large database with 100 million rows, but for the purposes of this blog we create a small database to show how HA works.<\/p>\n\n\n\n<p>Note that you must install taosTools in order to have taosBenchmark installed.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-bd43e62e gb-headline-text\">Create a small database of simulated Smart Meter data<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">tdengine@tdc01:~$ taosBenchmark -I stmt -d test -n 100 -t 100 -a 3\n&#091;09\/04 20:28:19.612119] INFO: client version: 3.1.0.0\n\n         Press enter key to continue or Ctrl-C to stop\n\nfailed to create dir:\/var\/log\/taos\/ since Operation not permitted WARING: Create taoslog failed:Operation not permitted. configDir=\/etc\/taos\/\n&#091;09\/04 20:28:27.559290] INFO: command to create database: &lt;CREATE DATABASE IF NOT EXISTS test VGROUPS 1 replica 3 PRECISION &#039;ms&#039;;&gt;\n&#091;09\/04 20:28:27.569557] SUCC: created database (test)\n&#091;09\/04 20:28:27.575220] WARN: failed to run command DESCRIBE `test`.`meters`, code: 0x80002603, reason: Table does not exist\n&#091;09\/04 20:28:27.575403] INFO: stable meters does not exist, will create one\n&#091;09\/04 20:28:27.575465] INFO: create stable: &lt;CREATE TABLE test.meters (ts TIMESTAMP,current float,voltage int,phase float) TAGS (groupid int,location binary(24))&gt;\n&#091;09\/04 20:28:27.582501] INFO: generate stable&lt;meters&gt; columns data with lenOfCols&lt;80&gt; * prepared_rand&lt;10000&gt;\n&#091;09\/04 20:28:27.593078] INFO: generate stable&lt;meters&gt; tags data with lenOfTags&lt;62&gt; * childTblCount&lt;100&gt;\n&#091;09\/04 20:28:27.598267] INFO: start creating 100 table(s) with 8 thread(s)\n.\n.\n.\n.\n.\n&#091;09\/04 20:28:29.828752] INFO: Total 100 tables on bb test&#039;s vgroup 0 (id: 2)\n&#091;09\/04 20:28:29.833427] INFO: Estimate memory usage: 3.93MB\n\n         Press enter key to continue or Ctrl-C to stop\n\n&#091;09\/04 20:28:37.095207] INFO:  pthread_join 0 ...\n&#091;09\/04 20:28:37.201477] SUCC: thread&#091;0] progressive mode, completed total inserted rows: 10000, 97145.85 records\/second\n&#091;09\/04 20:28:37.201584] SUCC: Spent 0.106339 seconds to insert rows: 10000 with 1 thread(s) into test 94038.88 records\/second\n&#091;09\/04 20:28:37.201593] SUCC: insert delay, min: 0.4890ms, avg: 1.0294ms, p90: 1.3420ms, p95: 3.5180ms, p99: 11.9660ms, max: 11.9660ms<\/code><\/pre>\n\n\n\n<p>Note that you will have to press the &#8220;Enter&#8221; key when prompted and there will be several lines of output. We are only displaying the first few lines and the last few lines.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-e0e65fb8 gb-headline-text\">Check the number of rows in the database<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; select count(*) from test.meters;       count(*)        |========================                 10000 |Query OK, 1 row(s) in set (0.020480s)<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-de5f1597 gb-headline-text\">See the distribution of vgroups<\/h3>\n\n\n\n<p>Vnodes on different data nodes can form a virtual node group to ensure the high availability of the system.<\/p>\n\n\n\n<p>As you can see below, there is one vgroup.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; show test.vgroups\\G;\n*************************** 1.row ***************************\n    vgroup_id: 2\n      db_name: test\n       tables: 100\n     v1_dnode: 1\n    v1_status: leader\n     v2_dnode: 2\n    v2_status: follower\n     v3_dnode: 3\n    v3_status: follower\n     v4_dnode: NULL\n    v4_status: NULL\n    cacheload: 0\ncacheelements: 0\n         tsma: 0\nQuery OK, 1 row(s) in set (0.005120s)<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-c3208b46 gb-headline-text\">Create a new database with 3 replicas to test HA<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; create database if not exists testha replica 3;\nCreate OK, 0 row(s) affected (6.371307s)\n\ntaos&gt; show databases;\n              name              |\n=================================\n information_schema             |\n performance_schema             |\n test                           |\n testha                         |\nQuery OK, 4 row(s) in set (0.002791s)<\/code><\/pre>\n\n\n\n<p>Create a table named t1 and insert some data into it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; use testha;\nDatabase changed.\n\ntaos&gt; create table if not exists t1(ts timestamp, n int);\nCreate OK, 0 row(s) affected (0.001430s)\n\ntaos&gt; insert into t1 values(now, 1)(now+1s, 2)(now +2s,3)(now+3s,4);\nInsert OK, 4 row(s) affected (0.005576s)\n\ntaos&gt; select * from t1;\n           ts            |      n      |\n========================================\n 2023-09-04 20:42:00.449 |           1 |\n 2023-09-04 20:42:01.449 |           2 |\n 2023-09-04 20:42:02.449 |           3 |\n 2023-09-04 20:42:03.449 |           4 |\nQuery OK, 4 row(s) in set (0.001884s)<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-63081671 gb-headline-text\">Check online dnodes<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; show dnodes;     id      |            endpoint            | vnodes | support_vnodes |   status   |       create_time       |       reboot_time       |              note              |===================================================================================================================================================================          1 | tdc01:6030                     |      3 |              4 | ready      | 2023-09-04 18:58:10.391 | 2023-09-04 18:58:10.367 |                                |           2 | tdc02:6030                     |      3 |              4 | ready      | 2023-09-04 19:01:21.746 | 2023-09-04 19:00:18.628 |                                |           3 | tdc03:6030                     |      3 |              4 | ready      | 2023-09-04 19:02:48.980 | 2023-09-04 19:02:40.144 |                                |Query OK, 3 row(s) in set (0.004388s)<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-8af08a04 gb-headline-text\">RAFT Protocol<\/h3>\n\n\n\n<p>TDengine uses the RAFT protocol for data replication. For the RAFT protocol to work, it needs a quorum of (N\/2)+1 nodes and can tolerate a failure of (N-1)\/2 nodes in the cluster.<\/p>\n\n\n\n<h3 class=\"gb-headline gb-headline-ef9fa16c gb-headline-text\">Disconnect the tdc01 node<\/h3>\n\n\n\n<p>You can either power off the VM or just disconnect the network.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; show dnodes;     id      |            endpoint            | vnodes | support_vnodes |   status   |       create_time       |       reboot_time       |              note              |===================================================================================================================================================================&lt;strong&gt;1&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt; tdc01:&lt;\/strong&gt;&lt;strong&gt;6030&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt;3&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt;0&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt; offline    &lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt;2023-09-04&lt;\/strong&gt;&lt;strong&gt;18&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;58&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;10.391&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt;1969-12-31&lt;\/strong&gt;&lt;strong&gt;16&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;00&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;00.000&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt; status &lt;\/strong&gt;&lt;strong&gt;not&lt;\/strong&gt;&lt;strong&gt; received            &lt;\/strong&gt;|           2 | tdc02:6030                     |      3 |              4 | ready      | 2023-09-04 19:01:21.746 | 2023-09-04 19:00:18.628 |                                |           3 | tdc03:6030                     |      3 |              4 | ready      | 2023-09-04 19:02:48.980 | 2023-09-04 19:02:40.144 |                                |Query OK, 3 row(s) in set (0.004617s)<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-d2464e69 gb-headline-text\">Insertion and querying still works<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; insert into testha.t1 values(now, 1)(now+1s, 2);\nInsert OK, 2 row(s) affected (0.002733s)\n\ntaos&gt; select count(*) from testha.t1;\n       count(*)        |\n========================\n                     6 |\nQuery OK, 1 row(s) in set (0.013952s)\n\ntaos&gt; insert into testha.t1 values(now, 1)(now+1s, 2);\nInsert OK, 2 row(s) affected (0.002751s)\n\ntaos&gt; select count(*) from testha.t1;\n       count(*)        |\n========================\n                     8 |\nQuery OK, 1 row(s) in set (0.002328s)\n\ntaos&gt; select * from testha.t1;\n           ts            |      n      |\n========================================\n 2023-09-04 20:42:00.449 |           1 |\n 2023-09-04 20:42:01.449 |           2 |\n 2023-09-04 20:42:02.449 |           3 |\n 2023-09-04 20:42:03.449 |           4 |\n 2023-09-04 20:47:58.913 |           1 |\n 2023-09-04 20:47:59.913 |           2 |\n 2023-09-04 20:49:20.395 |           1 |\n 2023-09-04 20:49:21.395 |           2 |\nQuery OK, 8 row(s) in set (0.003532s)<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-9f302126 gb-headline-text\">Restore node tdc01<\/h3>\n\n\n\n<p>As soon as node tdc01 is restored, it rejoins the cluster.<\/p>\n\n\n\n<p>Note that the mnodes leader is tdc03.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; show dnodes;\n     id      |            endpoint            | vnodes | support_vnodes |   status   |       create_time       |       reboot_time       |              note              |\n===========================================================================================================================================================================\n           1 | tdc01:6030                     |      3 |              4 | ready      | 2023-09-04 18:58:10.391 | 2023-09-04 20:50:34.226 |                                |\n           2 | tdc02:6030                     |      3 |              4 | ready      | 2023-09-04 19:01:21.746 | 2023-09-04 19:00:18.628 |                                |\n           3 | tdc03:6030                     |      3 |              4 | ready      | 2023-09-04 19:02:48.980 | 2023-09-04 19:02:40.144 |                                |\nQuery OK, 3 row(s) in set (0.002064s)<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; show mnodes;\n     id      |            endpoint            |     role     |  status   |       create_time       |        role_time        |\n==============================================================================================================================\n           1 | tdc01:6030                     | follower     | ready     | 2023-09-04 18:58:10.393 | 2023-09-04 20:50:36.959 |\n           2 | tdc02:6030                     | follower     | ready     | 2023-09-04 20:23:33.318 | 2023-09-04 20:45:43.682 |\n&lt;strong&gt;3&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt; tdc03:&lt;\/strong&gt;&lt;strong&gt;6030&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt; leader       &lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt; ready     &lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt;2023-09-04&lt;\/strong&gt;&lt;strong&gt;20&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;24&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;38.435&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;&lt;strong&gt;2023-09-04&lt;\/strong&gt;&lt;strong&gt;20&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;45&lt;\/strong&gt;&lt;strong&gt;:&lt;\/strong&gt;&lt;strong&gt;43.693&lt;\/strong&gt;&lt;strong&gt;|&lt;\/strong&gt;\nQuery OK, 3 row(s) in set (0.004637s)<\/code><\/pre>\n\n\n\n<p>Take a look at the vgroups as well.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; show testha.vgroups\\G;\n*************************** 1.row ***************************\n    vgroup_id: 3\n      db_name: testha\n       tables: 0\n     v1_dnode: 1\n    v1_status: follower\n     v2_dnode: 2\n    v2_status: follower\n     v3_dnode: 3\n    v3_status: leader\n     v4_dnode: NULL\n    v4_status: NULL\n    cacheload: 0\ncacheelements: 0\n         tsma: 0\n*************************** 2.row ***************************\n    vgroup_id: 4\n      db_name: testha\n       tables: 1\n     v1_dnode: 1\n    v1_status: follower\n     v2_dnode: 2\n    v2_status: leader\n     v3_dnode: 3\n    v3_status: follower\n     v4_dnode: NULL\n    v4_status: NULL\n    cacheload: 0\ncacheelements: 0\n         tsma: 0\nQuery OK, 2 row(s) in set (0.016652s)<\/code><\/pre>\n\n\n\n<h3 class=\"gb-headline gb-headline-8656828a gb-headline-text\">Only 1 node is up<\/h3>\n\n\n\n<p>Now we can address the case when only a single node is running.<\/p>\n\n\n\n<p>We power down or disconnect nodes tdc01 and tdc02.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">&lt;strong&gt;tdengine@tdc03:~$ &lt;\/strong&gt;ping tdc01\nPING tdc01 (10.0.2.4) 56(84) bytes of data.\n^C\n--- tdc01 ping statistics ---\n3 packets transmitted, 0 received, 100% packet loss, time 2055ms\n\n&lt;strong&gt;tdengine@tdc03:~$&lt;\/strong&gt; ping tdc02\nPING tdc02 (10.0.2.6) 56(84) bytes of data.\nFrom tdc03 (10.0.2.5) icmp_seq=1 Destination Host Unreachable\nFrom tdc03 (10.0.2.5) icmp_seq=2 Destination Host Unreachable\nFrom tdc03 (10.0.2.5) icmp_seq=3 Destination Host Unreachable\nFrom tdc03 (10.0.2.5) icmp_seq=4 Destination Host Unreachable\n^C\n--- tdc02 ping statistics ---\n6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5066ms\npipe 3<\/code><\/pre>\n\n\n\n<p>Now if we try to connect to the cluster, we cannot. Therefore, it is impossible to query or insert data.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">&lt;strong&gt;tdengine@tdc03:~$&lt;\/strong&gt; taos -h tdc03\nfailed to create dir:\/var\/log\/taos\/ since Operation not permitted WARING: Create taoslog failed:Operation not permitted. configDir=\/etc\/taos\/\nWelcome to the TDengine Command Line Interface, Client Version:3.1.0.0\nCopyright (c) 2022 by TDengine, all rights reserved.\n\nfailed to connect to server, reason: Sync leader is unreachable<\/code><\/pre>\n\n\n\n<p>Now we spin up or connect the node tdc01.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">&lt;strong&gt;tdengine@tdc03:~$ &lt;\/strong&gt;ping tdc01\nPING tdc01 (10.0.2.4) 56(84) bytes of data.\n64 bytes from tdc01 (10.0.2.4): icmp_seq=1 ttl=64 time=0.285 ms\n64 bytes from tdc01 (10.0.2.4): icmp_seq=2 ttl=64 time=1.10 ms\n^C\n--- tdc01 ping statistics ---\n2 packets transmitted, 2 received, 0% packet loss, time 1003ms\nrtt min\/avg\/max\/mdev = 0.285\/0.691\/1.098\/0.406 ms<\/code><\/pre>\n\n\n\n<p>Try to connect to the cluster again.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\" data-line=\"\">taos&gt; select count(*) from test.meters;\n       count(*)        |\n========================\n                 10000 |\ntaos&gt; show databases;\n              name              |\n=================================\n information_schema             |\n performance_schema             |\n test                           |\n testha                         |\nQuery OK, 4 row(s) in set (0.004463s)\n\ntaos&gt; select count(*) from testha.t1;\n       count(*)        |\n========================\n                     8 |\nQuery OK, 1 row(s) in set (0.012661s)<\/code><\/pre>\n\n\n\n<h2 class=\"gb-headline gb-headline-7f5a0e95 gb-headline-text\">TDengine Cloud &#8211; Enterprise-ready<\/h2>\n\n\n\n<p>With <a href=\"https:\/\/cloud.tdengine.com\">TDengine Cloud<\/a> you get a highly secure, fully managed service with backups, scalability, <strong>high-availability <\/strong>and Enterprise-ready features including SSO and replication across public clouds. You can sign up for a Starter TDengine Cloud account at in less than a minute and it does not require a credit card.<\/p>\n\n\n\n<p>TDengine Cloud is the easiest way for anyone to try the technology behind TDengine and experience a high performance next generation data historian.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>TDengine is built for high availability in terms of the logical design which extends itself very optimally to a physical architecture.<\/p>\n","protected":false},"author":81,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[21],"tags":[],"ppma_author":[167],"class_list":["post-17828","post","type-post","status-publish","format-standard","hentry","category-engineering"],"authors":[{"term_id":167,"user_id":81,"is_guest":0,"slug":"chait","display_name":"Chait Diwadkar","avatar_url":{"url":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg","url2x":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg"},"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/17828","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/comments?post=17828"}],"version-history":[{"count":13,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/17828\/revisions"}],"predecessor-version":[{"id":24697,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/17828\/revisions\/24697"}],"wp:attachment":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media?parent=17828"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/categories?post=17828"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/tags?post=17828"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/ppma_author?post=17828"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}