time-series databases<\/a> (TSDB) and selected TDengine as a solution.<\/p>\n\n\n\nAccording to our research, TDengine is an open-sourced IoT big data platform capable of handling large data volumes with industry-leading high performance and high storage efficiency. It has been well received by the developer community since it was made available under an open-source license two years ago, garnering nearly 16,000 Github stars so far and hundreds of clones on a daily basis.<\/p>\n\n\n\n
Referring to the documentation of TDengine, the problems we\u2019ve encountered can be easily solved with its simple SQL-like syntax.<\/p>\n\n\n\n
select last(val) a from super_table_xx where ts >= '2021-06-07 18:10:00' and ts <= '2021-06-07 18:20:00' interval(60s) fill(value, 0);<\/code><\/pre>\n\n\n\nRemarkable STable and Table Design<\/h2>\n\n\n\n Why does TDengine present such a powerful efficiency? It lies in the unique design of STable and table. On the one hand, TDengine uses “one table for one data collection point (device)” to fit in the typical IoT scenario where the data is generated by the single device and timestamped at a regular time interval. In the actual usage scenarios, no matter real-time queries or offline data analysis, reading data from a single device is always filtered by a consistent time interval.<\/p>\n\n\n\n
On the other hand, to achieve higher management efficiency, TDengine development team invented STable to deal with the same type of data points (devices).<\/p>\n\n\n\n
For instance, if we want to store the temperature data and humidity data of the equipment, STable can be created in this way:<\/p>\n\n\n\n
create stable if not exists s_device (ts TIMESTAMP,\n temperature double,\n humidity double\n) TAGS (device_sn BINARY(1000));<\/code><\/pre>\n\n\n\nIn actual use, for example, the SQL statement to perform \u2018INSERT data\u2019 for devices ‘d1’ and ‘d2’ is as follows.<\/p>\n\n\n\n
insert into s_device_d1 (ts, temperature, humidity) USING s_device (device_sn) TAGS ('d1') values (1623157875000, 35.34, 80.24);\ninsert into s_device_d2 (ts, temperature, humidity) USING s_device (device_sn) TAGS ('d2') values (1623157891000, 29.63, 79.48);<\/code><\/pre>\n\n\n\nThe query for data from device \u2018d1\u2019 in a certain time interval is like this:<\/p>\n\n\n\n
select * from s_device where device_sn = 'd1' and ts > 1623157871000 and ts < 1623157890000 ;<\/code><\/pre>\n\n\n\nIf selecting the average temperature data for the last 7 days by 1-hour interval, the statement can be:<\/p>\n\n\n\n
select avg(temperature) temperature from s_device where device_sn = #{deviceSn} and ts >= #{startTime} and ts < #{endTime} interval(1h)<\/code><\/pre>\n\n\n\nTDengine also provides many aggregation functions. Except \u2018last\u2019 and \u2018fill\u2019 used in the cases above, other commonly used functions such as \u2018sum\u2019 and \u2018max\u2019 are also included.<\/p>\n\n\n\n
In the process of integrating with applications, we use MyBatis, a flexible and easy-to-use ORM framework. For example, for the table ‘s_device’ above, we defined the entity at first:<\/p>\n\n\n\n
import com.baomidou.mybatisplus.annotation.TableField;\nimport com.baomidou.mybatisplus.annotation.TableName;\nimport lombok.AllArgsConstructor;\nimport lombok.Builder;\nimport lombok.Data;\nimport lombok.NoArgsConstructor;\n\nimport java.sql.Timestamp;\n\n\/**\n * @author: DaLuo\n * @date: 2021\/06\/25\n * @description:\n *\/\n@Data\n@AllArgsConstructor\n@NoArgsConstructor\n@Builder\n@TableName(value = "s_device")\npublic class TestSuperDeviceEntity {\n\n private Timestamp ts;\n private Float temperature;\n private Float humidity;\n @TableField(value = "device_sn")\n private String device_sn ;\n}<\/code><\/pre>\n\n\n\nAnd then we defined the mapper:<\/p>\n\n\n\n
import com.baomidou.mybatisplus.core.mapper.BaseMapper;\nimport com.hg.device.kafka.tdengine.entity.TestSuperDeviceEntity;\nimport lombok.AllArgsConstructor;\nimport lombok.Builder;\nimport lombok.Data;\nimport lombok.NoArgsConstructor;\nimport org.apache.ibatis.annotations.Insert;\nimport org.apache.ibatis.annotations.Mapper;\nimport org.apache.ibatis.annotations.Param;\nimport org.apache.ibatis.annotations.Select;\n\nimport java.sql.Timestamp;\nimport java.util.List;\n\n\/**\n * @author: DaLuo\n * @date: 2021\/06\/25\n * @description:\n *\/\n@Mapper\npublic interface TestSuperDeviceMapper extends BaseMapper<TestSuperDeviceEntity> {\n\n \/**\n * insert\n * @param entity\n * @return\n *\/\n @Insert({\n "INSERT INTO 's_device_${entity.deviceSn}' (ts ,temperature, humidity ) ",\n "USING s_device (device_sn) TAGS (#{entity.deviceSn}) ",\n "VALUES (#{entity.ts}, #{entity.temperature}, #{entity.humidity})"\n })\n int insertOne(@Param(value = "entity") TestSuperDeviceEntity entity);\n\n\n \/**\n * batch writing\n * @param entities\n * @return\n *\/\n @Insert({\n "<script>",\n "INSERT INTO ",\n "<foreach collection='list' item='item' separator=' '>",\n "'s_device_${item.deviceSn}' (ts ,temperature, humidity) USING s_device (device_sn) TAGS (#{item.deviceSn}) ",\n "VALUES (#{item.ts}, #{item.temperature}, #{item.humidity})",\n "<\/foreach>",\n "<\/script>"\n })\n int batchInsert(@Param("list") List<TestSuperDeviceEntity> entities);\n\n \/**\n * select the average temperature data for the last 7 days by 1-hour interval\n * @param deviceSn\n * @param startTime inclusive\n * @param endTime exclusive\n * @return\n *\/\n @Select("select avg(temperature) temperature from s_device where device_sn = #{deviceSn} and ts >= #{startTime} and ts < #{endTime} interval(1h)")\n List<TempSevenDaysTemperature> selectSevenDaysTemperature(\n @Param(value = "deviceSn") String deviceSn,\n @Param(value = "startTime") long startTime,\n @Param(value = "endTime") long endTime);\n\n @AllArgsConstructor\n @NoArgsConstructor\n @Data\n @Builder\n class TempSevenDaysTemperature {\n private Timestamp ts;\n private float temperature;\n }\n}<\/code><\/pre>\n\n\n\nAnother amazing feature of TDengine is no need for creating tables in advance. By setting the ‘tag’ as a part of \u2018table_name\u2019, it will automatically create tables when inserting data.<\/p>\n\n\n\n
Connectors<\/h2>\n\n\n\n We can connect to TDengine via TDengine JDBC-driver in two ways — JDBC-JNI and JDBC-RESTful. JDBC-JNI demonstrates higher ingestion performance while it requires to install TDengine client on the server where the application is running.<\/p>\n\n\n\n
We have deployed our application in the Kubernetes cluster, while the program is running in Docker. Therefore, we created an image of the application program. Here is the Dockerfile of the base image:<\/p>\n\n\n\n
FROM openjdk:8-jdk-oraclelinux7\n \nCOPY TDengine-client-2.0.16.0-Linux-x64.tar.gz \/\n \nRUN tar -xzvf \/TDengine-client-2.0.16.0-Linux-x64.tar.gz && cd \/TDengine-client-2.0.16.0 && pwd && ls && .\/install_client.sh<\/code><\/pre>\n\n\n\nbuild\uff1a<\/p>\n\n\n\n
docker build -t tdengine-openjdk-8-runtime:2.0.16.0 -f Dockerfile .<\/code><\/pre>\n\n\n\nThe Dockerfile shows:<\/p>\n\n\n\n
FROM tdengine-openjdk-8-runtime:2.0.16.0\n \nENV JAVA_OPTS="-Duser.timezone=Asia\/Shanghai -Djava.security.egd=file:\/dev\/.\/urandom"\n \nCOPY app.jar \/app.jar\n \nENTRYPOINT ["java","-jar","\/app.jar"]<\/code><\/pre>\n\n\n\nThus, the application can be scheduled on any node in Kubernetes.<\/p>\n\n\n\n
In addition, our program involves automated scheduling, which requires frequent message transports between upper machines and lower machines via MQTT. For example, the cloud sends command 1000-“start task A”, and then the lower machine responds a command 2000-“received task A”. If regarding the command as a device and the sequence number of the command as its attributes, this kind of data will also be compatible to be stored in TDengine.<\/p>\n\n\n\n
*************************** 1.row ***************************\n ts: 2021-06-23 16:10:30.000\n msg: {\"task_id\":\"7b40ed4edc1149f1837179c77d8c3c1f\",\"action\":\"start\"}\ndevice_sn: deviceA\n kind: 1000\n*************************** 2.row ***************************\n ts: 2021-06-23 16:10:31.000\n msg: {\"task_id\":\"7b40ed4edc1149f1837179c77d8c3c1f\",\"action\":\"received\"}\ndevice_sn: deviceA\n kind: 2000<\/pre>\n\n\n\nConclusion<\/h2>\n\n\n\n In the end, TDengine also has a STable \u2018log.dn\u2019 which stores metric data including memory usage, CPU usage, etc., hence we can utilize Grafana to visualize those metrics as a reliable reference for monitoring.<\/p>\n\n\n\nHeyGears selected TDengine to handle time-series data storage and processing for its 3D printing devices.<\/p>\n","protected":false},"author":81,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[23,268,231],"tags":[],"ppma_author":[167],"class_list":["post-2992","post","type-post","status-publish","format-standard","hentry","category-case-studies","category-industrial-data","category-manufacturing"],"authors":[{"term_id":167,"user_id":81,"is_guest":0,"slug":"chait","display_name":"Chait Diwadkar","avatar_url":{"url":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg","url2x":"https:\/\/tdengine.com\/wp-content\/uploads\/29.03-05-cdiwadkar.jpg"},"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/2992","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/comments?post=2992"}],"version-history":[{"count":10,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/2992\/revisions"}],"predecessor-version":[{"id":24611,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/2992\/revisions\/24611"}],"wp:attachment":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media?parent=2992"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/categories?post=2992"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/tags?post=2992"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/ppma_author?post=2992"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.013-01-example-1.png?strip=all&sharp=1\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.013-02-example-2.png?strip=all&sharp=1\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.013-03-example-3.png?strip=all&sharp=1\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/eujqw4hwudm.exactdn.com\/wp-content\/uploads\/22.013-04-dashboard.png?strip=all&sharp=1\")
<\/figure>\n","protected":false},"excerpt":{"rendered":"