Based on the design principle of the supertable<\/a>, we use the business related information of livestock as tags, which is also convenient to associate with data in MySQL. Each device exists as a subtable, and the device’s measurements are used as columns in the subtable.<\/p>\n\n\n\n The supertable schema is shown in Figure 1 below. Figures 2 and 3 show the supertable schema for environmental sensors installed in fixed positions on the farm. The temperature and humidity table, in addition to the table that stores the original message data, are not listed here.<\/p>\n\n\n\n At present, all the IoT data tables of our company are designed based on the TDengine supertable. The core livestock supertable, has more tags than other tables. It should be noted that, in order to ensure that the tags in TDengine are consistent with MySQL, whenever the basic attributes of livestock are modified for business reasons, tags need to be modified synchronously in TDengine. The supertable design also allows us to trace possible processing delays and other problems. <\/p>\n\n\n\n The collect_time in the table is the collection time, and the insert_time is the time when data is inserted into TDengine. If the time difference between the two is large, there may be network problems, collection device failures or insufficient server throughput etc. Having this data, allows us to investigate. The whole process of livestock data collection to data placement is shown in the figure below. The data collected by the collection device is transmitted through the 4G network. The device gateway performs preliminary processing on the data and then pushes it to MQ (Message Queue). The data is then transmitted to subscribers\/consumers on MQ and finally to TDengine.<\/p>\n\n\n\n Since we were replacing MongoDB, we had to perform a data migration as follows:<\/p>\n\n\n\n During the migration process, we also encountered two small issues:<\/p>\n\n\n\n The ramifications after the migration are very clear. When we used MongoDB, the self-built cluster used six, 4-core, 32G servers. After migrating to TDengine, the self-built cluster only used only two, 8-core, 32G machines. This had led to a significant operational cost decline.<\/strong> In terms of specific performance, I simulated the data of more than 6,000 collectors, and the table data totaled about 300 million records. Most of our queries are based on subtables, and only some businesses need to view aggregated operational data. When querying group by+last_row(*) on the supertable, the data can be returned within 1.5s, and querying on the subtable in about 0.1 seconds (select * from son_table limit 10). This meets our business requirements.<\/p>\n\n\n\n With the development of emerging technologies such as the Internet of Things and artificial intelligence, AIoT has become a major trend that cannot be ignored. To store the massive amounts of data, requires both warehouses and transactional databases. However, with a purpose-built product like TDengine, designed with IoT, time-series data in mind, this complexity can be avoided. In a sense, TDengine with it’s stream computing and high-performance<\/a> query engine, helped save a lot of intermediate steps and resulted in significant cost reduction and increased efficiency. So far, TDengine has been outstanding in cost reduction and performance improvement. We are looking forward to the day when it can support calls to AI models and output the results directly. That will be perfect!<\/p>\n","protected":false},"excerpt":{"rendered":" Learn how TDengine replaced MongoDB and delivered cost reduction and increased efficiency in a livestock management scenario.<\/p>\n","protected":false},"author":81,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[23],"tags":[],"ppma_author":[167],"class_list":["post-5430","post","type-post","status-publish","format-standard","hentry","category-case-studies"],"authors":[{"term_id":167,"user_id":81,"is_guest":0,"slug":"chait","display_name":"Chait Diwadkar","avatar_url":{"url":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg","url2x":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg"},"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/5430","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/comments?post=5430"}],"version-history":[{"count":12,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/5430\/revisions"}],"predecessor-version":[{"id":29455,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/5430\/revisions\/29455"}],"wp:attachment":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media?parent=5430"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/categories?post=5430"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/tags?post=5430"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/ppma_author?post=5430"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.022-01-schema-1.jpg?strip=all&sharp=1\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.022-02-schema-2.png?strip=all&sharp=1\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.022-03-schema-3.png?strip=all&sharp=1\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.022-04-architecture.png?strip=all&sharp=1\")
<\/figure>\n\n\n\nData Migration and Ramifications<\/h2>\n\n\n\n
\n
\n
Closing Thoughts<\/h2>\n\n\n\n