out_azure_kusto: Added workload identity support

[tanmaya-panda1]

This pull request introduces support for a new authentication method, "Workload Identity," in the Azure Kusto plugin. It includes updates to the configuration, authentication logic, and token management to accommodate this new method while maintaining compatibility with existing authentication types. Additionally, minor improvements and fixes are made to logging and code readability.

Example configuration for this change:
[OUTPUT]
name azure_kusto
match *
tenant_id xxxx
client_id xxxxx
ingestion_endpoint xxxx
auth_type workload_identity
database_name xxxx
table_name xxxxx
ingestion_endpoint_connect_timeout 600
Retry_Limit 5
buffering_enabled On
upload_file_size 10M
upload_timeout 2m
unify_tag On

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[tanmaya-panda1]

docs PR : fluent/fluent-bit-docs#1631

[tanmaya-panda1]

Debug logs of the feature

[2025/05/09 05:28:34] [ info] [azure workload identity] inside flb_azure_workload_identity_token_get [2025/05/09 05:28:34] [ info] [azure workload identity] after read token from file xxxxx [2025/05/09 05:28:34] [debug] [http_client] not using http_proxy for header [2025/05/09 05:28:34] [debug] [azure workload identity] Sending request body (len=1859): client_id=xxxx&grant_type=client_credentials&client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer&client_assertion=xxxxxx&scope=https://help.kusto.windows.net/.default [2025/05/09 05:28:34] [debug] [azure workload identity] HTTP Status=200 [2025/05/09 05:28:34] [debug] [azure workload identity] token exchange successful [2025/05/09 05:28:34] [ info] [azure workload identity] access token retrieved successfully [2025/05/09 05:28:34] [debug] [output:azure_kusto:azure_kusto.0] Workload identity token retrieved successfully [2025/05/09 05:28:34] [debug] [http_client] not using http_proxy for header [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] Kusto ingestion command request http_do=0, HTTP Status: 200 [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] Kusto ingestion command HTTP response payload: xxxxxxx [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: TempStorage [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: TempStorage [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: IngestionsStatusTable [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] parsed 2 blob resources and 8 queue resources [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] before getting upstream connection [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] Logging attributes of flb_azure_kusto_resources: [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] blob_ha: (nil) [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] queue_ha: (nil) [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] load_time: 0 [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] execute_ingest_csl_command -- async flag is 0 [2025/05/09 05:28:35] [debug] [upstream] KA connection #104 to ingest-xxxxx.kusto.windows.net:443 has been assigned (recycled) [2025/05/09 05:28:35] [debug] [http_client] not using http_proxy for header [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] Kusto ingestion command request http_do=0, HTTP Status: 200 [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] Kusto ingestion command HTTP response payload: {"Tables":[{"TableName":"Table_0","Columns":[{"ColumnName":"AuthorizationContext","DataType":"String","ColumnType":"string"}],"Rows":[["xxxxxxx"]]}]} [2025/05/09 05:28:35] [debug] [upstream] KA connection #104 to ingest-xxxxxx.kusto.windows.net:443 is now available [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] parsed kusto identity token [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] [construct_request_buffer] size of buffer file read 7288218 [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] [construct_request_buffer] size of new_data 10122192 [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] [construct_request_buffer] final increased 10122192 [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] enabled payload gzip compression [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] inside blob after upstream ha node get [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] azure_kusto_create_blob -- async flag is 0 [2025/05/09 05:28:35] [debug] [output:azure_kusto:azure_kusto.0] inside blob after upstream ha node get :: setting ingestion timeout [2025/05/09 05:28:36] [debug] [output:azure_kusto:azure_kusto.0] inside blob after upstream ha node get :: after getting connection [2025/05/09 05:28:36] [debug] [output:azure_kusto:azure_kusto.0] inside blob before create blob uri [2025/05/09 05:28:36] [debug] [output:azure_kusto:azure_kusto.0] created blob uri https://xxxxx.blob.core.windows.net/xxxxxxx [2025/05/09 05:28:36] [ info] [output:azure_kusto:azure_kusto.0] azure_kusto: before calling azure storage api :: value of set io_timeout is 60 [2025/05/09 05:28:36] [debug] [output:azure_kusto:azure_kusto.0] uploading payload to blob uri: https://xxxxx.blob.core.windows.net/xxxxxx [2025/05/09 05:28:36] [debug] [http_client] not using http_proxy for header [2025/05/09 05:28:37] [debug] [output:azure_kusto:azure_kusto.0] kusto blob upload request http_do=0, HTTP Status: 201 [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] created queue uri /aggregatorinput-secured/messages?xxxxx [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] uuid :: 57759dd7-ac0f-c324-83ae-e2181e1f5a61 [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] blob uri :: https://xxxxx.blob.core.windows.net/xxxxxx [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] payload size :: 273989 [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] database_name :: xxx [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] table name :: xxxx [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] created ingestion message: {"Id": "57759dd7-ac0f-c324-83ae-e2181e1f5a61", "BlobPath": "https://xxxxxx.blob.core.windows.net/20250509-ingestdata-e5c334ee145d4b4-0/flb__e2e__FluentBit__7TVXnUkYay3O9PjtKiBpNnv7FhNsTjMO1wAegI0e6TS5yZm85LlIXGGsMhLtrn5i__1746768515684__20250509052835__15474572-10fc-b60e-a274-0f667f82d218.multijson.gz?xxxxxxxxxx", "RawDataSize": 273989, "DatabaseName": "e2e", "TableName": "FluentBit", "ClientVersionForTracing": "Kusto.Fluent-Bit:4.0.2", "ApplicationForTracing": "Kusto.Fluent-Bit", "AdditionalProperties": { "format": "multijson", "authorizationContext": "xxxxxxx", "jsonMappingReference": "" }} [2025/05/09 05:28:38] [debug] [http_client] not using http_proxy for header [2025/05/09 05:28:38] [debug] [output:azure_kusto:azure_kusto.0] kusto queue request http_do=0, HTTP Status: 201 [2025/05/09 05:28:38] [debug] [task] created task=0x7f3d7dd7eea0 id=0 OK

[whober0521]

@leonardo-albertovich , @edsiper , @koleini , @fujimotos , could you please review this PR?