Fatal Error in DB Workload

[Tyler-Ouellette]

The DB workload is running into errors:

{"level":"info","ts":"2025-06-29T23:21:02.946Z","logger":"injection-startup","msg":"creating standalone runner"}
{"level":"info","ts":"2025-06-29T23:21:02.946Z","logger":"injection-startup","msg":"checking envvars"}
{"level":"info","ts":"2025-06-29T23:21:02.946Z","logger":"injection-startup","msg":"envvars checked","env":{"failurePolicy":"silent","installerUrl":"","installerFlavor":"multidistro","installVersion":"1.317.39.20250627-183006","installPath":"/opt/dynatrace/oneagent-paas","k8NodeName":"knackstedt-asus-tuf","k8PodName":"db-75c5cd7bcd-vnl5j","k8BasePodUID":"bfb80dfe-a654-43ff-96fa-f58dae7b940e","k8BasePodName":"db-75c5cd7bcd","k8Namespace":"easytrade","k8ClusterID":"e4026976-5c6d-4f24-919e-e6ce77106012","k8sClusterName":"andrews-cluster","k8sClusterEntityID":"KUBERNETES_CLUSTER-FE2EB0776F07D48E","workloadKind":"deployment","workloadName":"db","workloadAnnotations":{},"installerTech":["all"],"containers":[{"name":"db","image":"europe-docker.pkg.dev/dynatrace-demoability/docker/easytrade/db:latest"}],"oneAgentInjected":true,"metadataEnrichmentInjected":true}}
{"level":"info","ts":"2025-06-29T23:21:02.947Z","logger":"injection-startup","msg":"read secret from filesystem"}
{"level":"info","ts":"2025-06-29T23:21:02.947Z","logger":"injection-startup","msg":"contents of secret config","content":{"monitoringNodes":{"knackstedt-asus-tuf":"SPRINT-ID"},"apiUrl":"https://.sprint.dynatracelabs.com/api","apiToken":"","paasToken":"","proxy":"","noProxy":"","networkZone":"us.east-1.ohio.andrews.house","oneAgentNoProxy":"andrews-cluster-activegate.dynatrace","tenantUUID":"SPRINT-ID","hostGroup":"k8s","initialConnectRetry":-1,"skipCertCheck":false,"hasHost":true,"enforcementMode":true,"csiMode":true,"readOnlyCSIDriver":false}}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"read certificates from filesystem","content":"","filename":"agcerts"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"read certificates from filesystem","content":"","filename":"trustedcas"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"creating dtclient"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"dtclient created successfully"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"standalone runner created successfully"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"standalone init started"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"setting host tenant"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"host tenant set to TenantUUID"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"successfully set host tenant","hostTenant":"sprint-ID"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"configuring standalone OneAgent"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"setting ld.so.preload"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"created file","filePath":"/mnt/share/ld.so.preload","content":"/opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"creating container configuration files"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"creating conf file for container","container":{"name":"db","image":"europe-docker.pkg.dev/dynatrace-demoability/docker/easytrade/db:latest"}}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"adding k8s cluster id"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"adding k8s node name"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"created file","filePath":"/mnt/share/container_db.conf","content":"[container]\ncontainerName db\nimageName europe-docker.pkg.dev/dynatrace-demoability/docker/easytrade/db:latest\nk8s_fullpodname db-75c5cd7bcd-vnl5j\nk8s_poduid bfb80dfe-a654-43ff-96fa-f58dae7b940e\nk8s_containername db\nk8s_basepodname db-75c5cd7bcd\nk8s_namespace easytrade\nk8s_cluster_id e4026976-5c6d-4f24-919e-e6ce77106012\nk8s_node_name knackstedt-asus-tuf\n[host]\ntenant SPRINTID\nisCloudNativeFullStack true\n"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"propagating tls certificates to agent"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"created file","filePath":"/mnt/share/custom.pem"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"creating enrichment files"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"created file","filePath":"/tmp/enrichment/dt_metadata_db.json","content":"{"dt.entity.kubernetes_cluster":"KUBERNETES_CLUSTER-FE2EB0776F07D48E","dt.kubernetes.cluster.id":"e4026976-5c6d-4f24-919e-e6ce77106012","dt.kubernetes.workload.kind":"deployment","dt.kubernetes.workload.name":"db","k8s.cluster.name":"andrews-cluster","k8s.cluster.uid":"e4026976-5c6d-4f24-919e-e6ce77106012","k8s.container.name":"db","k8s.namespace.name":"easytrade","k8s.node.name":"knackstedt-asus-tuf","k8s.pod.name":"db-75c5cd7bcd-vnl5j","k8s.pod.uid":"bfb80dfe-a654-43ff-96fa-f58dae7b940e","k8s.workload.kind":"deployment","k8s.workload.name":"db"}"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"created file","filePath":"/tmp/enrichment/dt_metadata_db.properties","content":"k8s.container.name=db\nk8s.pod.uid=bfb80dfe-a654-43ff-96fa-f58dae7b940e\nk8s.namespace.name=easytrade\nk8s.cluster.name=andrews-cluster\nk8s.cluster.uid=e4026976-5c6d-4f24-919e-e6ce77106012\nk8s.workload.kind=deployment\nk8s.workload.name=db\ndt.kubernetes.workload.kind=deployment\nk8s.pod.name=db-75c5cd7bcd-vnl5j\nk8s.node.name=knackstedt-asus-tuf\ndt.entity.kubernetes_cluster=KUBERNETES_CLUSTER-FE2EB0776F07D48E\ndt.kubernetes.cluster.id=e4026976-5c6d-4f24-919e-e6ce77106012\ndt.kubernetes.workload.name=db\n"}
{"level":"info","ts":"2025-06-29T23:21:03.042Z","logger":"injection-startup","msg":"standalone init completed"}
Setting up database and tables
The inotify_init attempt failed with File: signals.cpp:569 [Status: 0xC000011F Operation unsuccessful errno = 0x18(24) Too many open files]
This program has encountered a fatal error and cannot continue running at Tue Jul 8 19:48:49 2025

The following diagnostic information is available:

     Reason: 0x00000003
     Status: 0x00000018
    Message: false
      Stack:
             IP               Function
             ---------------- --------------------------------------
             00005a5cc83f47f1 std::__1::bad_function_call::~bad_function_call()+0x97191
             00005a5cc83f41b6 std::__1::bad_function_call::~bad_function_call()+0x96b56
             00005a5cc83462df std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char>>::str() const+0x1c8f
             00005a5cc83f6e94 std::__1::bad_function_call::~bad_function_call()+0x99834
             00007e104331dac3 pthread_condattr_setpshared+0x513
    Process: 8 - sqlservr
     Thread: 13
Instance Id: 574146cc-0106-469b-bf73-a0d8580f2486
   Crash Id: f25805f5-0286-4240-bb96-235a293b9110
Build stamp: 9da4a512b2a09348c40cfe9ff7f072a7fa572ee5a1a90da8524b94fc4dd72186

Distribution: Ubuntu 22.04.4 LTS
Processors: 24
Total Memory: 134968467456 bytes
Timestamp: Tue Jul 8 19:48:49 2025
Last errno: 24
Last errno text: Too many open files
Capturing a dump of 8
sqlservr: utils.cpp:561: uintptr_t GetEndOfLibOSVmRange(): Assertion `GetGlobals()->GetStatics()->IsLibOSVmmRangeEndValid()' failed.
2025-07-08 19:48:50.71 Server The licensing PID was successfully processed. The new edition is [Express Edition].
Changed database context to 'TradeManagement'.

(3 rows affected)
RecoveryUnit::Shutdown. IsOnline: 02025-07-08 19:48:54.64 spid53 [5]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0

(105 rows affected)
RecoveryUnit::Shutdown. IsOnline: 02025-07-08 19:48:55.74 spid53 [5]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0

(105 rows affected)
RecoveryUnit::Shutdown. IsOnline: 02025-07-08 19:48:56.75 spid53 [5]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0

(3 rows affected)
2025-07-08 19:48:57.40 spid61s Attribute synchronization initialized

(15 rows affected)
RecoveryUnit::Shutdown. IsOnline: 02025-07-08 19:48:59.63 spid62 [5]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
RecoveryUnit::Shutdown. IsOnline: 02025-07-08 19:49:02.12 spid62 [5]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
Database 'TradeManagement' is being recovered. Waiting until recovery is finished.
ms].) ADR-enabled

To Reproduce
Steps to reproduce the behavior:
Deploy the manifest using europe-docker.pkg.dev/dynatrace-demoability/docker/easytrade/db:latest

Additional context
Add any other context about the problem here.

[Tyler-Ouellette]

You can contact me on slack if you have questions