Add workers.celery.securityContexts & workers.kubernetes.securityContexts

[Miretpl]

related: #28880

This PR introduces two new fields: workers.celery.securityContexts and workers.kubernetes.securityContexts. The workers.securityContexts field is not deprecated, as it is used by two worker types. Now it will be possible to use dedicated commands for both Celery and Kubernetes workers.

Despite adding new sections, the following things were done:

1. Update and separate tests in security/test_security_context.py
2. Add missing overwrite tests for securityContext/securityContexts in security/test_security_context.py and airflow_aux/test_pod_template_file.py
3. Rewrite the airflowPodSecurityContext and containerSecurityContext functions in _helpers.yaml to adjust them to reading nested securityContexts sections
4. Remove unneeded if for workers.celery.persistence.securityContexts section

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[Miretpl]

Adding a little more context to additional changes:

1. Points 1 and 2 are done, because of point 3 - to make sure that the changes made to functions are not breaking the logic before
2. Point 3 for simplification of future logic related to all securityContexts sections, which we will have in other, not yet done, sections in workers.celery/workers.kubernetes

+ I believe that this PR might be easier to review commit by commit

[jscheffl]

This time you did not mark the (non celery/k8s) workers parameters as deprecated - is this by intend? Because now both can be specified? Also no newsfragment about this?

[Miretpl]

Yes.

The idea is, if some workers section is used in both places (worker-deployment and pod-template-file), we want to give an easy option to overwrite the values for one, but if someone wants to specify the same config for both, I think it should be possible (maybe with an exception for resources as having the same resources for celery workers and kubernetes executor pods would be a bad idea).

Regarding the newsfragment, until now I only added them when the change was changing the behaviour of the chart, so e.g. in cases like:

{{ workers.celery.example | default workers.example }}

where example would be e.g. 1 by default, but if the default was unset or null, I didn't add the newsfragment (only the change in NOTES if the value was deprecated).

[jscheffl]

Yes.

The idea is, if some workers section is used in both places (worker-deployment and pod-template-file), we want to give an easy option to overwrite the values for one, but if someone wants to specify the same config for both, I think it should be possible (maybe with an exception for resources as having the same resources for celery workers and kubernetes executor pods would be a bad idea).

Okay, this makes sense. Then also no parameter deprecated, also no newsfragment needed. That is fine.

But can you please add a note to the new/old parameters describing the behavior of "one or the other" and the order of selection to both parameters please? Otherwise users will be confused (like me :-D)

[Miretpl]

Ahhh, I forgot to add it, sorry. I will add it in a couple of minutes

Also, maybe at this time the network error in tests will resolve itself :D

[Miretpl]

Clarifying comments added - let me know if it's good now

[jscheffl]

Thanks for the comments, looks good!

[Miretpl]

Small nit - I added the Kubernetes docSection to the newly added securityContexts. They should be rather there and not under the Workers section

[henry3260]

FYI: The S3 error is unrelated to your changes. I opened an issue to track this fix. Hopefully, that solves it.

[jscheffl]

Now a merge conflict from the other PR :-(