{"id":7696,"date":"2022-08-10T13:46:05","date_gmt":"2022-08-10T20:46:05","guid":{"rendered":"https:\/\/tdengine.com\/?p=7696"},"modified":"2025-03-30T20:11:48","modified_gmt":"2025-03-31T03:11:48","slug":"synchronizing-data-from-kafka-to-tdengine","status":"publish","type":"post","link":"https:\/\/tdengine.com\/synchronizing-data-from-kafka-to-tdengine\/","title":{"rendered":"Synchronizing Data from Kafka to TDengine"},"content":{"rendered":"\n

This article discusses how to use the TDengine Kafka Connector to synchronize data from Kafka to TDengine and provides a sample script to test its in a data synchronization scenario.<\/p>\n\n\n\n

Background<\/h2>\n\n\n\n

Kafka is a general-purpose message broker featuring a distributed architecture. You can use the Kafka Connect component to read from and write to Kafka, and its plug-ins can be used to read from and write to a variety of data sources. Kafka Connect supports fault tolerance, restarting, logging, elastic scaling, and serialization and deserialization.<\/p>\n\n\n\n

To make integrating Kafka and TDengine a simpler process, the TDengine Team developed the TDengine Kafka Connector as a Kafka Connect plug-in. The TDengine Kafka Connector consists of the TDengine Source Connector and TDengine Sink Connector. In this article, the TDengine Sink Connector is used to integrate Kafka with TDengine.<\/p>\n\n\n\n

TDengine Sink Connector implementation<\/h2>\n\n\n\n

The TDengine Sink Connector synchronizes the data from specified Kafka topics to a TDengine time-series database<\/a> (TSDB) in batches or in real time.<\/p>\n\n\n\n

Before running the TDengine Sink Connector, you must create a properties<\/code> file. For information about the properties<\/code> file, see the official documentation<\/a>.<\/p>\n\n\n\n

The implementation of the TDengine Sink Connector is as follows:<\/p>\n\n\n\n

    \n
  1. The Kafka Connect framework starts a specified number of consumer threads.<\/li>\n\n\n\n
  2. These consumers simultaneously subscribe to data and deserialize the data according to the values of key.converter<\/strong> and value.converter<\/strong> set in the configuration file.<\/li>\n\n\n\n
  3. The Kafka Connect framework sends the deserialized data to a specified number of SinkTask instances.<\/li>\n\n\n\n
  4. SinkTask uses the schemaless write interface provided by TDengine to write data into the database.<\/li>\n<\/ol>\n\n\n\n

    In this process, the first three steps are implemented automatically by the Kafka Connect framework, and the TDengine Sink Connector performs the final step on its own.<\/p>\n\n\n\n

    Supported data formats<\/h3>\n\n\n\n

    The TDengine Sink Connector writes data in schemaless mode. It supports the InfluxDB line format, OpenTSDB telnet format, and OpenTSDB JSON format. You can modify the value of the db.schemaless<\/strong> parameter to choose the format that you want to use. As an example, the following configuration enables InfluxDB line format:<\/p>\n\n\n\n

    db.schemaless=line<\/code><\/pre>\n\n\n\n
    If the data in your Kafka deployment is already in one of the three formats mentioned, set value.converter<\/strong> to the built-in Kafka Connect string converter:<\/p>\n\n\n\n
    value.converter=org.apache.kafka.connect.storage.StringConverter<\/code><\/pre>\n\n\n\n
    Otherwise, you must implement your own converter to process your data into one of the supported formats. For more information, see Converters<\/a>.<\/p>\n\n\n\n
    In this implementation, Kafka Connect is acting as the consumer. Consumer behavior can therefore be controlled by modifying the Kafka Connect configuration.<\/p>\n\n\n\n
    Topic configuration<\/h3>\n\n\n\n
    The topics to which the consumer subscribes are controlled by the topics<\/strong> parameter. For all other parameters, you can override the default configuration by adding the parameter to the properties file with each parameter having the consumer.override<\/strong> prefix. For example, the following line changes the maximum records per poll to 3000:<\/p>\n\n\n\n
    consumer.override.max.poll.records=3000<\/code><\/pre>\n\n\n\n
    Thread configuration<\/h3>\n\n\n\n
    In a Kafka Connect sink connector, each task is a consumer thread that receives data from the partitions of a topic. You can use the tasks.max<\/strong> parameter to control the maximum number of tasks and thereby the maximum number of threads. However, the number of tasks that are actually initiated is related to the number of topic partitions. For example, if you have ten partitions and the value of the tasks.max<\/strong> parameter is 5, each task will receive data from two partitions and keep track of the offsets of two partitions.<\/p>\n\n\n\n
    Note that if the value of the tasks.max<\/strong> parameter is larger than the number of partitions, the number of tasks that Kafka Connect initiates is equal to the number of partitions. The number of tasks is not related to the number of topics.<\/p>\n\n\n\n
    Procedure<\/h2>\n\n\n\n
      \n
    1. Install Kafka.\n
        \n
      1. Download and decompress the Kafka installation package. 
        wget https:\/\/dlcdn.apache.org\/kafka\/3.2.0\/kafka_2.13-3.2.0.tgz<\/code>
        tar -xzf kafka_2.13-3.2.0.tgz<\/code><\/li>\n\n\n\n
      2. Edit the .bash_profile<\/code> file and add the following text:
        export KAFKA_HOME=\/home\/bding\/kafka_2.13-3.2.0<\/code>
        export PATH=$PATH:$KAFKA_HOME\/bin<\/code><\/li>\n\n\n\n
      3. Reload the bash profile.
        source .bash_profile<\/code><\/li>\n<\/ol>\n<\/li>\n\n\n\n
      4. Configure Kafka.\n
          \n
        1. Edit the Kafka Connect properties file:
          cd kafka_2.13-3.2.0\/config\/<\/code>
          vi connect-standalone.properties<\/code><\/li>\n\n\n\n
        2. Add the plug-in directory:
          plugin.path=\/home\/bding\/connectors<\/code><\/li>\n\n\n\n
        3. Edit the log configuration of Kafka Connect:
          vi connect-log4j.properties<\/code><\/li>\n\n\n\n
        4. Set the logging level for the TDengine Sink Connector to DEBUG<\/code>: 
          log4j.logger.com.taosdata.kafka.connect.sink=DEBUG<\/code>
          This modification is necessary because these logs are used to calculate the time spent in synchronizing data.<\/li>\n<\/ol>\n<\/li>\n\n\n\n
        5. Compile and install the TDengine Sink Connector.<\/li>\n<\/ol>\n\n\n\n
             git clone git@github.com:taosdata\/kafka-connect-tdengine.git\n   cd kafka-connect-tdengine\n   mvn clean package\n   unzip -d ~\/connectors target\/components\/packages\/taosdata-kafka-connect-tdengine-*.zip<\/code><\/pre>\n\n\n\n
            \n
          1. Start the ZooKeeper and Kafka servers.<\/li>\n<\/ol>\n\n\n\n
               zookeeper-server-start.sh -daemon $KAFKA_HOME\/config\/zookeeper.properties\n   kafka-server-start.sh -daemon $KAFKA_HOME\/config\/server.properties<\/code><\/pre>\n\n\n\n
              \n
            1. Create a topic.<\/li>\n<\/ol>\n\n\n\n
                 kafka-topics.sh --create --topic meters --partitions 1 --bootstrap-server localhost:9092<\/code><\/pre>\n\n\n\n
                \n
              1. Generate test data.\n
                  \n
                1. Save the following script as gen-data.py<\/code>:
                  #!\/usr\/bin\/python3<\/code>
                  import random<\/code>
                  import sys<\/code>
                  topic = sys.argv[1]<\/code>
                  count = int(sys.argv[2])<\/code>
                  start_ts = 1648432611249000000<\/code>
                  location = ["SanFrancisco", "LosAngeles", "SanDiego"]<\/code>
                  for i in range(count):<\/code>
                      ts = start_ts + i<\/code>
                      row = f"{topic},location={location[i % 3]},groupid=2 current={random.random() * 10},voltage={random.randint(100, 300)},phase={random.random()} {ts}"<\/code>
                      print(row)<\/code><\/li>\n\n\n\n
                2. Run gen-data.py<\/code>:
                  python3 gen-data.py meters 10000 | kafka-console-producer.sh --broker-list localhost:9092 --topic meters<\/code>
                  The script generates 10,000 data points in InfluxDB line format and adds them to the meters topic. Each data point has two label fields and three data fields.<\/p><\/li>\n<\/ol>\n<\/li>\n\n\n\n
                3. Start Kafka Connect.\n
                    \n
                  1. Save the following configuration as sink-test.properties<\/code>:
                    name=TDengineSinkConnector connector.class=com.taosdata.kafka.connect.sink.TDengineSinkConnector<\/code>
                    tasks.max=1<\/code>
                    topics=meters<\/code>
                    connection.url=jdbc:TAOS:\/\/127.0.0.1:6030<\/code>
                    connection.user=root<\/code>
                    connection.password=taosdata<\/code>
                    connection.database=power<\/code>
                    db.schemaless=line<\/code>
                    key.converter=org.apache.kafka.connect.storage.StringConverter <\/code>
                    value.converter=org.apache.kafka.connect.storage.StringConverter<\/code><\/li>\n\n\n\n
                  2. Start Kafka Connect:
                    connect-standalone.sh -daemon $KAFKA_HOME\/config\/connect-standalone.properties sink-test.properties<\/code><\/li>\n<\/ol>\n<\/li>\n\n\n\n
                  3. Use the TDengine CLI to query the meters table in the power database to verify that there are 10,000 data points.<\/li>\n<\/ol>\n\n\n\n
                       [bding@vm95 test]$ taos\n\n   Welcome to the TDengine shell from Linux, Client Version:2.6.0.4\n   Copyright (c) 2022 by TAOS Data, Inc. All rights reserved.\n   taos> select count(*) from power.meters;\n          count(*)        |\n   ========================\n                    10000 |<\/pre>\n\n\n\n
                    TDengine Sink Connector performance testing<\/h2>\n\n\n\n
                    This performance test is similar to steps 4 through 7 in the previous section. Note the following:<\/p>\n\n\n\n
                      \n
                    • The performance test script takes two arguments. The first is the number of partitions, and the second is the number of data points to generate.<\/li>\n\n\n\n
                    • The tasks.max parameter is set to the number of partitions. This controls the number of threads in each test.<\/li>\n\n\n\n
                    • The test database must be empty before the test begins.<\/li>\n\n\n\n
                    • After the test is complete, stop Kafka, ZooKeeper, and Kafka Connect manually.<\/li>\n<\/ul>\n\n\n\n
                      In each test, data is first written to Kafka and then synchronized by Kafka Connect to TDengine. This ensures that the sink connector handles the entire pressure caused by the synchronization task. The total time required for synchronization is calculated from when the sink connector receives the first batch of data to when it receives the last batch of data.<\/p>\n\n\n\n
                      To run the performance test, save the following script as run-test.sh<\/code>:<\/p>\n\n\n\n
                      #!\/bin\/bash\nif [ $# -lt 2 ];then\n        echo  "Usage: .\/run-test.sh <num_of_partitions>  <total_records>"\n        exit 0\nfi\necho "---------------------------TEST STARTED---------------------------------------"\necho clean data and logs\ntaos -s "DROP DATABASE IF EXISTS power"\nrm -rf \/tmp\/kafka-logs \/tmp\/zookeeper\nrm -f $KAFKA_HOME\/logs\/connect.log\nnp=$1     # number of partitions\ntotal=$2  # number of records\necho number of partitions is $np, number of recordes is $total.\necho start zookeeper\nzookeeper-server-start.sh -daemon $KAFKA_HOME\/config\/zookeeper.properties\necho start kafka\nsleep 3\nkafka-server-start.sh -daemon $KAFKA_HOME\/config\/server.properties\nsleep 5\necho create topic\nkafka-topics.sh --create --topic meters --partitions $np --bootstrap-server localhost:9092\nkafka-topics.sh --describe --topic meters --bootstrap-server localhost:9092\necho generate test data\npython3 gen-data.py meters $total  | kafka-console-producer.sh --broker-list localhost:9092 --topic meters\n\necho alter connector configuration setting tasks.max=$np\nsed -i  "s\/tasks.max=.*\/tasks.max=${np}\/"  sink-test.properties\necho start kafka connect\nconnect-standalone.sh -daemon $KAFKA_HOME\/config\/connect-standalone.properties sink-test.properties\n     \necho -e "\\e[1;31m open another console to monitor connect.log. press enter when no more data received.\\e[0m"\nread\n     \necho stop connect\njps | grep ConnectStandalone | awk '{print $1}' | xargs kill\necho stop kafka server\nkafka-server-stop.sh\necho stop zookeeper\nzookeeper-server-stop.sh\n# extract timestamps of receiving the first batch of data and the last batch of data\ngrep "records" $KAFKA_HOME\/logs\/connect.log  | grep meters- > tmp.log\nstart_time=`cat tmp.log | grep -Eo "[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}" | head -1`\nstop_time=`cat tmp.log | grep -Eo "[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}" | tail -1`\n    \necho "--------------------------TEST FINISHED------------------------------------"\necho "| records | partitions | start time | stop time |"\necho "|---------|------------|------------|-----------|"\necho "| $total | $np | $start_time | $stop_time |"\n<\/code><\/pre>\n\n\n\n
                      You can then run the script and specify a number of partitions and data points. As an example, run the following command for a performance test with one partition and 1 million data points:<\/p>\n\n\n\n
                      .\/run-test.sh 1 1000000<\/code><\/pre>\n\n\n\n
                      The test process is shown in the following figure.<\/p>\n\n\n\n
                      \"\"<\/figure>\n\n\n\n
                      Note that you must monitor the connect.log<\/code> file in a separate console and press Enter<\/strong> once all data has been consumed. You can use the tail -f connect.log<\/code> command to monitor progress:<\/p>\n\n\n\n
                      [bding@vm95 ~]$ cd kafka_2.13-3.2.0\/logs\/\n[bding@vm95 logs]$ tail -f connect.log\n[2022-06-21 17:39:00,176] DEBUG [TDengineSinkConnector|task-0] Received 500 records. First record kafka coordinates:(meters-0-314496). Writing them to the database... (com.taosdata.kafka.connect.sink.TDengineSinkTask:101)\n[2022-06-21 17:39:00,180] DEBUG [TDengineSinkConnector|task-0] Received 500 records. First record kafka coordinates:(meters-0-314996). Writing them to the database... (com.taosdata.kafka.connect.sink.TDengineSinkTask:101)<\/pre>\n\n\n\n
                      When new entries are no longer being written to the log file, this indicates that the data consumption has been completed.<\/p>\n\n\n\n
                      The following table shows the data write performance determined by this test.<\/p>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t
                      <\/td>1 thread<\/th>3 thread<\/th>5 threads<\/th>10 threads<\/th>\n<\/tr>\n<\/thead>\n
                      1 million<\/td>105,219<\/td>232,937<\/td>333,000<\/td>489,956<\/td>\n<\/tr>\n
                      3 million<\/td>107,650<\/td>239,330<\/td>363,240<\/td>573,175<\/td>\n<\/tr>\n
                      5 million<\/td>108,321<\/td>246,573<\/td>364,087<\/td>580,720<\/td>\n<\/tr>\n
                      10 million<\/td>107,803<\/td>248,855<\/td>372,912<\/td>562,936<\/td>\n<\/tr>\n
                      15 million<\/td>106,651<\/td>249,671<\/td>377,283<\/td>541,927<\/td>\n<\/tr>\n
                      20 million<\/td>103,626<\/td>244,921<\/td>371,402<\/td>557,460<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
                      \"\"<\/figure>\n\n\n\n
                      The data points indicate the average number of entries written per second for configurations with one, three, five, and ten threads.<\/p>\n\n\n\n
                      Conclusion<\/h2>\n\n\n\n
                      From the preceding figure, we can see that given the same size data set, write speed increases with the number of threads.<\/p>\n\n\n\n
                        \n
                      • With a single thread, 100,000 data points are written per second.<\/li>\n\n\n\n
                      • With five threads, the speed increases to 350,000 data points per second<\/li>\n\n\n\n
                      • With ten threads, the speed increases to 550,000 data points per second.<\/li>\n<\/ul>\n\n\n\n
                        The write speed is relatively stable and is not clearly associated with the total size of the data set.<\/p>\n\n\n\n
                        However, it can be seen that the performance improvement per thread declines as the number of threads increases. Going from one to ten threads only increases speed by a factor of five. This may be caused by uneven distribution of data among partitions. Some tasks take longer to complete than others, and this shift increases with the size of the data set.<\/p>\n\n\n\n
                        As an example, if the data set is created with 10 million data points spread across 10 partitions, the distribution of data points per partition is as follows:<\/p>\n\n\n\n
                        [bding@vm95 kafka-logs]$ du -h .\/ -d 1\n125M    .\/meters-8\n149M    .\/meters-7\n119M    .\/meters-9\n138M    .\/meters-4\n110M    .\/meters-3\n158M    .\/meters-6\n131M    .\/meters-5\n105M    .\/meters-0\n113M    .\/meters-2\n99M     .\/meters-1\n<\/pre>\n\n\n\n
                        Another factor influencing multithreaded write speed is out-of-order data. Because this test allocates data randomly across partitions, only a single-partition configuration can result in strictly ordered data, which provides the highest performance. As the number of threads is raised, the degree to which the data is out of order increases.<\/p>\n\n\n\n
                        For this reason, it is recommended that all data contained within a subtable is stored in the same Kafka partition.<\/p>\n\n\n\n
                        Appendix<\/h2>\n\n\n\n
                        The test environment used in this article is described as follows:<\/p>\n\n\n\n
                          \n
                        • OpenJDK 1.8.0<\/li>\n\n\n\n
                        • CentOS 7.9<\/li>\n\n\n\n
                        • 64 GB of memory<\/li>\n\n\n\n
                        • 16-core i7-10700 (x86-64, 2.90 GHz)<\/li>\n\n\n\n
                        • HDD storage<\/li>\n\n\n\n
                        • TDengine 2.6.0.4<\/li>\n\n\n\n
                        • Kafka 3.2<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
                          The TDengine Team developed the TDengine Kafka Connector to simplify Kafka integration. This article shows how the connector syncs data from Kafka to TDengine.<\/p>\n","protected":false},"author":48,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[21],"tags":[],"ppma_author":[100],"class_list":["post-7696","post","type-post","status-publish","format-standard","hentry","category-engineering"],"authors":[{"term_id":100,"user_id":48,"is_guest":0,"slug":"sangshuduo","display_name":"Shuduo Sang","avatar_url":{"url":"https:\/\/tdengine.com\/wp-content\/uploads\/29.04-28-sdsang.jpg","url2x":"https:\/\/tdengine.com\/wp-content\/uploads\/29.04-28-sdsang.jpg"},"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/7696","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/users\/48"}],"replies":[{"embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/comments?post=7696"}],"version-history":[{"count":8,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/7696\/revisions"}],"predecessor-version":[{"id":24641,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/posts\/7696\/revisions\/24641"}],"wp:attachment":[{"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/media?parent=7696"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/categories?post=7696"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/tags?post=7696"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/tdengine.com\/wp-json\/wp\/v2\/ppma_author?post=7696"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}