Merge pull request #5215 from sysown/feature/add-tap-test-documentation

Add TAP test writing guide and GitHub automation improvements

Author: renecannao

.gitignore

@@ -151,3 +151,4 @@ proxysql-save.cfg
 test/tap/tests/test_cluster_sync_config/cluster_sync_node_stderr.txt
 test/tap/tests/test_cluster_sync_config/proxysql*.pem
 test/tap/tests/test_cluster_sync_config/test_cluster_sync.cnf
+GEMINI.md

doc/TAP_TESTS_GUIDE.md

@@ -0,0 +1,186 @@
+# How to Write TAP Tests for ProxySQL
+
+This guide provides instructions and best practices for writing Test Anything Protocol (TAP) tests for ProxySQL. The tests are written in C++ and leverage a common framework to interact with ProxySQL instances, check for expected behavior, and report results.
+
+## Anatomy of a TAP Test File
+
+A typical TAP test file (`*-t.cpp`) has the following structure:
+
+1.  **Includes:** Essential headers are included.
+    *   `"tap.h"`: The core TAP library for test reporting (`plan`, `ok`, `diag`, `exit_status`).
+    *   `"command_line.h"`: A helper for reading connection parameters (host, port, user, password) from environment variables.
+    *   `"utils.h"`: Provides various utility functions and macros, like `MYSQL_QUERY`.
+    *   Protocol-specific headers: `"mysql.h"` for MySQL or `"libpq-fe.h"` for PostgreSQL tests.
+    *   Standard C++ libraries (`<string>`, `<vector>`, `<chrono>`, etc.).
+
+2.  **`main()` Function:** The entry point for the test.
+    *   **`plan(N)`:** The first thing you should do is declare how many tests you plan to run. `N` is the total number of `ok()` calls in your test. This can be calculated dynamically if the number of tests depends on other factors.
+    *   **`CommandLine cl;`:** An object to manage command-line and environment variables. `cl.getEnv()` reads the necessary configuration.
+    *   **Connections:** Establish one or more connections to ProxySQL. Typically, you need at least two:
+        *   An **admin connection** to configure ProxySQL, load settings, and check statistics tables.
+        *   A **client connection** to the standard proxy port to simulate application behavior.
+    *   **Setup (Arrange):** Prepare the test environment. This is a critical step to ensure your test is isolated and predictable. Common setup tasks include:
+        *   Deleting existing rules or data from previous test runs (e.g., `DELETE FROM mysql_query_rules`).
+        *   Setting global variables (`UPDATE global_variables SET ...`).
+        *   Inserting new configuration (e.g., `INSERT INTO mysql_users ...`).
+        *   Loading the new configuration into the running ProxySQL instance (e.g., `LOAD MYSQL USERS TO RUNTIME`).
+    *   **Execution (Act):** Perform the actions you want to test. This could be running a specific query, killing a connection, or triggering a cluster sync.
+    *   **Verification (Assert):** Check that the outcome of the action is what you expected. This is done using the `ok()` macro.
+        *   `ok(condition, "description of the test");`
+        *   The `condition` is a boolean expression. If true, the test passes. If false, it fails.
+        *   The description should clearly state what is being tested.
+    *   **`diag("message")`:** Print diagnostic messages to stderr. This is useful for showing the test's progress or debugging failures. These messages are not counted as test results.
+    *   **Cleanup:** Restore the original state if necessary (e.g., `LOAD MYSQL VARIABLES FROM DISK`).
+    *   **`return exit_status();`:** Finally, return the overall test status. The test runner uses this exit code.
+
+## Key Principles for Good Tests
+
+### 1. Be Isolated and Self-Contained
+
+A test should not depend on the state left behind by other tests.
+
+*   **GOOD:** Start by deleting any existing configuration relevant to your test.
+    ```cpp
+    // test_firewall-t.cpp
+    MYSQL_QUERY(mysqladmin, "delete from mysql_firewall_whitelist_users");
+    MYSQL_QUERY(mysqladmin, "delete from mysql_firewall_whitelist_rules");
+    MYSQL_QUERY(mysqladmin, "load mysql firewall to runtime");
+    ```
+*   **GOOD:** If you modify global variables, reload them from disk at the end of the test.
+    ```cpp
+    // test_firewall-t.cpp
+    MYSQL_QUERY(mysqladmin, "load mysql variables from disk");
+    MYSQL_QUERY(mysqladmin, "load mysql variables to runtime");
+    ```
+
+### 2. Verify State Through the Admin Interface
+
+The most reliable way to check ProxySQL's internal state is by querying the `stats` and `runtime` tables.
+
+*   **GOOD:** To check if a connection was created, query `stats_mysql_connection_pool`.
+    ```cpp
+    // test_connection_annotation-t.cpp
+    MYSQL_QUERY(proxysql_admin, "SELECT ConnUsed, ConnFree FROM stats.stats_mysql_connection_pool WHERE hostgroup=1");
+    // ... compare results before and after
+    ```
+*   **GOOD:** To check if a query hit the cache, query `stats_mysql_query_digest`.
+    ```cpp
+    // test_query_cache_soft_ttl_pct-t.cpp
+    const string STATS_QUERY_DIGEST =
+    	"SELECT hostgroup, SUM(count_star) FROM stats_mysql_query_digest "
+    	"WHERE digest_text = 'SELECT SLEEP(?)' GROUP BY hostgroup";
+    ```
+
+### 3. Handle Asynchronicity
+
+Many operations in ProxySQL are asynchronous (e.g., connection killing, cluster synchronization). Your test must account for this.
+
+*   **GOOD:** Use a polling loop with a timeout to wait for a condition to become true. This is more robust than a fixed `sleep()`.
+    ```cpp
+    // test_cluster1-t.cpp
+    int module_in_sync(...) {
+        while (i < num_retries && rc != 1) {
+            // ... query stats_proxysql_servers_checksums and check if all nodes have the same checksum ...
+            sleep(1);
+            i++;
+        }
+        return (rc == 1 ? 0 : 1); // Return 0 on success
+    }
+    ```
+*   **ACCEPTABLE:** For simple cases where an action is expected to be fast, a short `sleep()` can be used.
+    ```cpp
+    // kill_connection-t.cpp
+    std::string s = "KILL CONNECTION " + std::to_string(mythreadid[j]);
+    MYSQL_QUERY(mysql, s.c_str());
+    sleep(1); // Give ProxySQL a moment to process the kill
+    int rc = run_q(other_mysql_conn, "DO 1");
+    ok(rc != 0, "Connection should be killed");
+    ```
+
+### 4. Use Helper Functions and Macros
+
+For complex or repetitive tasks, use helpers to make your test more readable and maintainable.
+
+*   **GOOD:** The `test_cluster1-t.cpp` test defines `trigger_sync_and_check` to encapsulate the entire logic for testing one module's synchronization.
+*   **GOOD:** The `pgsql-basic_tests-t.cpp` test defines a `PQEXEC` macro to wrap `PQexec` and add error checking, reducing boilerplate.
+
+### 5. Structure Complex Tests Clearly
+
+For features that require multiple steps or scenarios (like the REST API or PostgreSQL protocol tests), break the test into smaller functions.
+
+*   **GOOD:** The `pgsql-basic_tests-t.cpp` has separate functions like `test_simple_query`, `test_insert_query`, `test_transaction_commit`, etc. This makes it easy to see what is being tested and to debug failures.
+*   **GOOD:** The `reg_test_3223-restapi_return_codes-t.cpp` test defines its test cases in data structures (`std::vector` of structs) and then iterates over them. This data-driven approach is excellent for testing many variations of an input.
+
+## Example Template
+
+Here is a basic template to get you started.
+
+```cpp
+#include <string>
+#include <vector>
+#include <cstdio>
+
+#include "mysql.h"
+#include "tap.h"
+#include "command_line.h"
+#include "utils.h"
+
+int main(int argc, char** argv) {
+    // 1. Declare the number of tests you will run.
+    plan(3);
+
+    CommandLine cl;
+    if (cl.getEnv()) {
+        diag("Failed to get the required environmental variables.");
+        return exit_status();
+    }
+
+    // 2. Establish connections.
+    MYSQL* admin = mysql_init(NULL);
+    if (!mysql_real_connect(admin, cl.host, cl.admin_username, cl.admin_password, NULL, cl.admin_port, NULL, 0)) {
+        diag("Failed to connect to admin interface: %s", mysql_error(admin));
+        return exit_status();
+    }
+
+    MYSQL* client = mysql_init(NULL);
+    if (!mysql_real_connect(client, cl.host, cl.username, cl.password, NULL, cl.port, NULL, 0)) {
+        diag("Failed to connect to client interface: %s", mysql_error(client));
+        mysql_close(admin);
+        return exit_status();
+    }
+
+    // 3. Arrange: Set up the test environment.
+    diag("Setting up test: creating a new query rule.");
+    MYSQL_QUERY(admin, "DELETE FROM mysql_query_rules WHERE rule_id=999");
+    MYSQL_QUERY(admin, "INSERT INTO mysql_query_rules (rule_id, active, match_pattern, destination_hostgroup) VALUES (999, 1, '^SELECT 123', 1)");
+    MYSQL_QUERY(admin, "LOAD MYSQL QUERY RULES TO RUNTIME");
+
+    // 4. Act & Assert: Run the test and verify the outcome.
+    diag("Running a query that should match the rule.");
+    int rc = mysql_query(client, "SELECT 123");
+    ok(rc == 0, "Query 'SELECT 123' should execute successfully.");
+
+    // Verify state via statistics
+    MYSQL_QUERY(admin, "SELECT hits FROM stats_mysql_query_rules WHERE rule_id=999");
+    MYSQL_RES* res = mysql_store_result(admin);
+    ok(res && mysql_num_rows(res) == 1, "Rule 999 should exist in stats.");
+    if (res && mysql_num_rows(res) == 1) {
+        MYSQL_ROW row = mysql_fetch_row(res);
+        ok(atoi(row[0]) == 1, "Rule 999 should have exactly 1 hit.");
+    }
+    mysql_free_result(res);
+
+
+    // 5. Cleanup
+    diag("Cleaning up test rule.");
+    MYSQL_QUERY(admin, "DELETE FROM mysql_query_rules WHERE rule_id=999");
+    MYSQL_QUERY(admin, "LOAD MYSQL QUERY RULES TO RUNTIME");
+
+
+    // 6. Close connections and exit.
+    mysql_close(admin);
+    mysql_close(client);
+
+    return exit_status();
+}
+```