{"id":23497,"date":"2025-01-02T00:22:01","date_gmt":"2025-01-02T08:22:01","guid":{"rendered":"https:\/\/tdengine.com\/?p=23497"},"modified":"2025-03-30T23:29:10","modified_gmt":"2025-03-31T06:29:10","slug":"unlocking-database-customization-a-comprehensive-guide-to-python-udfs-in-tdengine","status":"publish","type":"post","link":"https:\/\/tdengine.com\/unlocking-database-customization-a-comprehensive-guide-to-python-udfs-in-tdengine\/","title":{"rendered":"Mastering Python UDFs in TDengine: From Beginner to Pro"},"content":{"rendered":"\n

With the release of TDengine 3.0.4.0, we unveiled an exciting new feature: the ability to create User-Defined Functions (UDFs) using Python. This innovative capability delivers unparalleled flexibility for database operations while making customization more accessible\u2014even for those new to programming. With Python UDFs, users can easily tailor and manage their databases, seamlessly integrating these custom functions into SQL queries as if they were built-in. In this article, we’ll explore how to leverage this feature effectively and help you take the first steps toward creating your own database customizations.<\/p>\n\n\n\n

Getting Started with UDFs<\/strong><\/h2>\n\n\n\n

Creating a <\/strong>UDF<\/strong><\/h3>\n\n\n\n

Here’s how to create a Python UDF function in TDengine:<\/p>\n\n\n\n

       CREATE  [OR REPLACE] [AGGREGATE] FUNCTION function_name\n                               as library_path  OUTPUTTYPE output_type  [BUFSIZE buffer_size] [LANGUAGE 'C\u2018 | 'Python']<\/code><\/pre>\n\n\n\n
Options Explained:<\/strong><\/p>\n\n\n\n
    \n
  1. CREATE [OR REPLACE]<\/code><\/strong>: Use CREATE<\/code> for new functions and add OR REPLACE<\/code> to update existing ones.<\/li>\n\n\n\n
  2. AGGREGATE<\/code><\/strong>: Optional. Indicates an aggregate function. Without it, the function defaults to a scalar function.<\/li>\n\n\n\n
  3. function_name<\/code><\/strong>: The name of the UDF, which can be up to 64 bytes long. Names exceeding this limit will be truncated.<\/li>\n\n\n\n
  4. OUTPUTTYPE<\/code><\/strong>: Specifies the output data type of the UDF. Supported types include:<\/li>\n<\/ol>\n\n\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
    Serial Number<\/th>Supported Data Type<\/th>\n<\/tr>\n<\/thead>\n
    1<\/td>TIMESTAMP<\/td>\n<\/tr>\n
    2<\/td>INT<\/td>\n<\/tr>\n
    3<\/td>INT UNSIGNED<\/td>\n<\/tr>\n
    4<\/td>BIGINT<\/td>\n<\/tr>\n
    5<\/td>BIGINT UNSIGNED<\/td>\n<\/tr>\n
    6<\/td>FLOAT<\/td>\n<\/tr>\n
    7<\/td>DOUBLE<\/td>\n<\/tr>\n
    8<\/td>BINARY<\/td>\n<\/tr>\n
    9<\/td>SMALLINT<\/td>\n<\/tr>\n
    10<\/td>SMALLINT UNSIGNED<\/td>\n<\/tr>\n
    11<\/td>TINYINT<\/td>\n<\/tr>\n
    12<\/td>TINYINT UNSIGNED<\/td>\n<\/tr>\n
    13<\/td>BOOL<\/td>\n<\/tr>\n
    14<\/td>NCHAR<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
      \n
    1. BUFSIZE<\/code><\/strong>: Sets the memory buffer size for aggregate functions (up to 256 KB). This buffer is allocated for the lifetime of the aggregation process, making it useful as a global variable.<\/li>\n\n\n\n
    2. LANGUAGE<\/code><\/strong>: Specifies the programming language. Currently, TDengine supports Python and C.<\/li>\n<\/ol>\n\n\n\n
      Deleting a <\/strong>UDF<\/strong><\/h3>\n\n\n\n
      To remove a UDF, use the following command:<\/p>\n\n\n\n
      DROP FUNCTION function_name;<\/code><\/pre>\n\n\n\n
      Viewing UDFs<\/strong><\/h3>\n\n\n\n
      To view all created UDFs:<\/p>\n\n\n\n
      SHOW FUNCTIONS;<\/code><\/pre>\n\n\n\n
      For detailed information:<\/p>\n\n\n\n
      SELECT * FROM information_schema.ins_functions\\G;<\/code><\/pre>\n\n\n\n
      Setting Up the Environment<\/strong><\/h2>\n\n\n\n
      Before diving into Python UDF development, ensure your environment is ready:<\/p>\n\n\n\n
        \n
      1. CMake<\/strong>: Minimum version 3.0.2.<\/li>\n\n\n\n
      2. GCC<\/strong>: Required for compiling the shared library (.so<\/code> file). Minimum version: 7.5.<\/li>\n\n\n\n
      3. Python<\/strong>: Version 3.7 or higher.<\/li>\n<\/ol>\n\n\n\n
        Install the UDF plugin with the following command:<\/p>\n\n\n\n
        python3 -m pip install taospyudf\nldconfig<\/code><\/pre>\n\n\n\n
        Your development environment is now ready!<\/p>\n\n\n\n
        Writing UDFs<\/strong><\/h2>\n\n\n\n
        UDFs in TDengine fall into two categories: scalar <\/strong>and aggregate<\/strong> functions. Before diving into the details of the functions, let’s first take a look at the process of how custom functions are invoked:<\/p>\n\n\n\n
        <\/figure>\n\n\n\n
        How UDFs Work<\/strong><\/h3>\n\n\n\n
        TDengine processes data in blocks for efficiency. When a UDF is invoked, the input data is a block, and you can access any cell in the block using the data(row, col)<\/code> method. This approach minimizes the overhead of calls between the C framework and Python, boosting performance.<\/p>\n\n\n\n