feat(annotations): add custom annotation prefix support for split horizon DNS

[lexfrei]

What does it do?

This PR adds a new --annotation-prefix flag that allows customizing the annotation prefix used by external-dns. This enables running multiple external-dns instances against the same Kubernetes resources, with each instance processing only its designated set of annotations.

Motivation

There are several real-world scenarios where you need multiple external-dns instances writing to different DNS servers or zones from the same Kubernetes resources:

1. Split Horizon DNS (Internal vs External)

A single Service needs to be accessible both from within the company's internal network and from the public internet, with different DNS records:

• Internal DNS: api.internal.company.com → internal LoadBalancer IP
• External DNS: api.company.com → public LoadBalancer IP
• Both annotations on the same Service, processed by different external-dns instances

2. Multiple API Gateways

When running multiple ingress controllers or API gateways (e.g., Kong for public APIs, internal gateway for private APIs):

• Public gateway external-dns instance processes public.company.io/hostname annotations
• Internal gateway external-dns instance processes internal.company.io/hostname annotations
• Each gateway has its own DNS zone and provider

3. Multi-Region DNS with Different Providers

• Region A external-dns writes to AWS Route53 with aws.company.io/ prefix
• Region B external-dns writes to Cloudflare with cloudflare.company.io/ prefix
• Same Service definitions deployed across regions, each processes its own annotations

4. Migration Scenarios

When migrating from one DNS provider to another:

• Old provider instance uses legacy.company.io/ prefix
• New provider instance uses external-dns.alpha.kubernetes.io/ prefix
• Both run simultaneously during migration period

5. Corporate Network Architecture

When additional proxying is implemented outside Kubernetes (e.g., hardware load balancers, CDN):

• Internal external-dns writes to internal DNS for direct cluster access
• External external-dns writes to public DNS pointing to external proxies
• No need to duplicate Service definitions

Implementation Details

Changes:

• Added AnnotationPrefix field to Config with validation (must end with /)
• Converted annotation constants to package-level variables that can be reconfigured
• Added SetAnnotationPrefix() function to rebuild all annotation keys at runtime
• Integrated annotation prefix configuration in controller startup
• Updated Helm chart with annotationPrefix parameter
• Added comprehensive split horizon DNS documentation
• Updated FAQ with annotation prefix examples

Backward Compatibility:

• Default prefix remains external-dns.alpha.kubernetes.io/
• No changes required for existing deployments
• Fully backward compatible with all existing configurations

Testing:

• All existing unit tests pass
• Added new unit tests for annotation prefix validation and reconfiguration
• Tested in real Kubernetes cluster with multiple instances (dry-run mode)
• Verified no cross-contamination between instances with different prefixes

Example Usage

# Internal DNS instance
--annotation-prefix=internal.company.io/

# Service with dual annotations
apiVersion: v1
kind: Service
metadata:
  annotations:
    # Internal DNS instance (Unifi provider)
    internal.company.io/target: "10.0.1.50"
    
    # External DNS instance (Cloudflare provider)
    external-dns.alpha.kubernetes.io/target: "203.0.113.10"

More

• Yes, this PR title follows Conventional Commits
• Yes, I added unit tests
• Yes, I updated end user documentation accordingly

---

Note: This PR was implemented by Claude Code AI assistant and thoroughly reviewed, tested, and validated in a real Kubernetes cluster environment before submission by the human maintainer.

[k8s-ci-robot]

Welcome @lexfrei!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @lexfrei. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ivankatliarchuk]

Same behavior already exist --annotation-filter.

Hard to see a reason for very similar flag.

[ivankatliarchuk]

Another option, you could use FQDN https://github.com/kubernetes-sigs/external-dns/blob/master/docs/advanced/fqdn-templating.md#conditional-templating-combined-with-annotations-processing if fine grained behavior is prefered

[lexfrei]

Thank you for the feedback!

I understand the confusion. Let me clarify the difference between --annotation-filter, FQDN templating, and --annotation-prefix:

--annotation-filter vs --annotation-prefix:

• --annotation-filter filters resources by annotation values (e.g., external-dns.alpha.kubernetes.io/controller: internal)
• --annotation-prefix allows different instances to read different annotation keys (e.g., internal.company.io/hostname vs external-dns.alpha.kubernetes.io/hostname)

The key difference: With --annotation-filter, both instances still compete for the same annotation keys. With --annotation-prefix, each instance reads completely different annotations from the same resource.

My specific use case that FQDN templating cannot solve:

I need to expose api.service.com with:

• Same hostname: api.service.com (both internal and external clients use the same FQDN)
• Different targets:
  • External DNS (Cloudflare): api.service.com → 203.0.113.10 (public IP)
  • Internal DNS (Unifi): api.service.com → 10.0.1.50 (internal IP)

Why? Internal clients shouldn't be routed through the internet (performance, cost, resilience if external connectivity fails).

Why FQDN templating doesn't work:

• FQDN templating creates different hostnames (e.g., api.internal.company.com vs api.company.com)
• I need the same hostname with different targets
• Per documentation: "If --fqdn-template is specified, ExternalDNS ignores any external-dns.alpha.kubernetes.io/hostname annotations"
• FQDN templating cannot change the target, only the hostname

Why --annotation-filter doesn't work:
Both instances would read the same annotations:

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: api.service.com
    external-dns.alpha.kubernetes.io/target: ???  # Can't specify two different targets

With --annotation-prefix it works cleanly:

# values.yaml for third-party Helm chart (e.g., ArgoCD, Grafana, etc.)
service:
  annotations:
    # Internal DNS instance (Unifi provider)
    internal.company.io/hostname: api.service.com
    internal.company.io/target: "10.0.1.50"
    
    # External DNS instance (Cloudflare provider)
    external-dns.alpha.kubernetes.io/hostname: api.service.com
    external-dns.alpha.kubernetes.io/target: "203.0.113.10"

# Internal instance
external-dns --annotation-prefix=internal.company.io/ --provider=rfc2136 ...

# External instance  
external-dns --provider=cloudflare ...

Additional real-world benefit:
When deploying third-party Helm charts (ArgoCD, Grafana, any community chart), you can't modify the chart templates. Current workarounds are problematic:

• Fork the chart → maintenance burden
• Create duplicate resources externally → pollutes the system (extra manifests alongside Helm-managed resources)
• Use --annotation-filter with controller annotation → still can't have different targets for same hostname

With --annotation-prefix, you just configure values.yaml with both sets of annotations. Clean, simple, no resource duplication.

Does this clarify the use case?

[kashalls]

The current implementation of annotation filter requires me to have duplicated services/ingresses/httproutes for each DNS provider I utilize. Having this option would allow me to have one dedicated service and allow me to utilize multiple webhook provider annotations where they currently conflict (different providers can use the same key and conflict, requiring me to setup individual annotation filters for each separate service).

@lexfrei Please let me know if you need help with anything to move this forward.

[ivankatliarchuk]

This is a duplicate of a functionality that is currently supported.

The described case fully supported with --fqdn-template you could write any queries. Or you could write your own webhook provider to add customized logic.

Example how to support public and private routing with https://kubernetes-sigs.github.io/external-dns/latest/docs/sources/traefik-proxy/#support-private-and-public-routing

[ivankatliarchuk]

(different providers can use the same key and conflict, requiring me to setup individual annotation filters for each separate service).

Kubernetes offers a well-established and predictable model for working with labels and selectors, as outlined in the official documentation. You can combine --annotation-filter with --label-filter to refine resource selection. Since adding annotations or labels to Kubernetes objects incurs virtually no overhead, this approach is both efficient and scalable.

Official docs https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/

For edge-cases there is fqdn-template flag

[kashalls]

This is a duplicate of a functionality that is currently supported.

The described case fully supported with --fqdn-template you could write any queries. Or you could write your own webhook provider to add customized logic.

Example how to support public and private routing with https://kubernetes-sigs.github.io/external-dns/latest/docs/sources/traefik-proxy/#support-private-and-public-routing

Maintaining two different ingress routes, creates several risks and problems related to consistency and overhead. While the --fqdn-template and webhook provider options offer a workaround, they are not a true solution for handling different public and private access rules for the same application. Even with webhook provider specific variables, this template would not be consistent across providers as the annotations contain a hardcoded webhook-* key.

I believe there was talks to phase out in-tree providers, but running two identical webhook providers would cause these annotations to clash. Supporting annotationPrefixes would support that movement by allowing users to deploy multiple instances of the provider, (provided they config it).

How can this be implemented with fdqn-template to support two separate providers with the same DNS Endpoints with different values in each without having to create two services just to make it available one or the other?

[ivankatliarchuk]

/hold

[ivankatliarchuk]

Kubernetes Design Philosophy: Simplicity & Predictability

Kubernetes emphasizes declarative configuration and predictable behavior. Label and annotation selectors are designed to be:

• Exact and deterministic – ensuring precise resource targeting.
• Simple to reason about – avoiding ambiguity and fuzzy matching.
• Clear in semantics – making it easier to audit and debug configurations.

Introducing partial matches (e.g., prefix or regex) adds unnecessary complexity and increases the risk of unintended behavior.

Recommended Alternatives

1. Use structured annotations or labels to clearly distinguish resources.
2. Leverage FQDN templating, which is already supported in ExternalDNS for edge-cases like this one. Documentation is available on the ExternalDNS website.

[ivankatliarchuk]

The PR is currently on hold. Let's wait to hear the perspectives of other maintainers and project owners before drawing conclusions.

[samip5]

As just a user of external-dns, this would make split-dns setups much more clean.

[gavinmcfall]

Thank you for pushing this forward, @lexfrei I want to add my support (from real-world usage) and also highlight how this helps us (and others) in split-horizon / hybrid DNS setups.

I run external-DNS together with kashall’s unifi webhook project and have faced exactly the pain points that this --annotation-prefix flag aims to address. Below are a few concrete observations / suggestions from my side:

---

Why I think this is valuable / needed

• Single resource, multiple DNS targets
  In many setups, you want one Kubernetes Service / Ingress / HTTPRoute to surface under the same hostname (or alias) to both internal and external clients, but pointing to different targets (internal IP vs external IP). With only --annotation-filter, you often resort to duplicating resources or complex selectors to “split” them.
  Kashall already described this well:

  “The current implementation of annotation filter requires me to have duplicated services/ingresses/httproutes for each DNS provider I utilize.”
  (source)

  Using a prefix means that two external-DNS instances can cleanly “see” different annotation keys on the same object without conflict.

• Better compatibility with Helm / upstream charts
  When deploying third-party charts you don't control, you often can’t easily change annotation keys inside the chart. The prefix approach lets you overlay additional annotations (internal vs external) without having to fork or duplicate resources. (source)

• Clear separation, no “annotation collisions”
  One of the objections I see is “this duplicates functionality of annotation filters / FQDN templating.” But in practice those tools don’t solve the same problem.
  As you clarified:

  “With --annotation-filter, both instances still compete for the same annotation keys. With --annotation-prefix, each instance reads completely different annotations from the same resource.”
  (source)

  FQDN templating is useful for adjusting hostnames; but it doesn't naturally support same hostname, different targets per instance.
  Your example in the PR is spot on:

  “I need to expose api.service.com with … internal DNS → 10.0.1.50, external DNS → 203.0.113.10.”
  (source)

  That use case simply can’t be handled by templating alone (since templating is about the name, not the target).

• Future-proofing for webhook / provider strategies
  As Kashall points out, when multiple webhook providers or external-DNS instances coexist, annotation clashes are a real concern. Prefixing helps isolate them cleanly:

  “Supporting annotationPrefixes would support that movement by allowing users to deploy multiple instances of the provider … without needing duplicate resources.”
  (source)

---

Suggestions / clarifications that might strengthen the PR

• Include one or two real minimal YAML examples (Service / Ingress) showing how you’d annotate with two prefixes, what the external-DNS invocations look like, and what the resulting DNS records would be. You already have one in the PR, but a side-by-side “before / after” comparison might help reviewers see the added value more tangibly.

• A short table comparing --annotation-filter, --fqdn-template, and --annotation-prefix with respect to hostname variation, target variation, resource duplication, compatibility with charts could clarify exactly where the gap is.

• Because this adds a new flag / behavior, it’d be good to document any edge cases or limitations:

  • What happens if a prefix is misconfigured (e.g. no trailing slash)?
  • What if a resource has both default and prefixed annotations—could there be conflicts or unintended fallback behaviors?
  • How does this interact with existing annotation constants and functions in the code (ensuring no regressions)?

• Maybe include a note in the PR / docs that existing users (who don’t use prefixes) continue to work unchanged — i.e. default remains external-dns.alpha.kubernetes.io/, so this is backward compatible. (source)

---

My endorsement

From my own deployments (external-DNS + the unifi webhook), I’d definitely use this feature. It would simplify my manifests, avoid resource duplication, and let multiple external-DNS instances coexist cleanly in the same cluster. To me, this is a meaningful usability and architectural improvement.

Happy to help test or provide examples if that’s useful!

[NetBUG]

I'd like to respectfully support this PR and address the concerns raised about functionality overlap.

Regarding --annotation-filter vs --annotation-prefix:
The fundamental difference here is that --annotation-filter filters resources by annotation values (e.g., external-dns.alpha.kubernetes.io/controller: internal), while --annotation-prefix allows different instances to read different annotation keys from the same resource. With --annotation-filter, both instances still compete for the same annotation keys. With --annotation-prefix, each instance processes completely separate annotations on the same Kubernetes object. This is not duplicate functionality—it solves a different problem entirely.

Why --fqdn-template doesn't address this use case:
FQDN templating creates different hostnames (e.g., api.internal.company.com vs api.company.com), but the core requirement here is to have the same hostname resolve to different targets depending on whether the client is internal or external. This is the essence of split-horizon DNS. As documented, when --fqdn-template is specified, external-dns ignores external-dns.alpha.kubernetes.io/hostname annotations, and templating cannot change the target IP—only the hostname. The use case requires:

# Internal DNS: api.service.com → 10.0.1.50 (private IP)
# External DNS: api.service.com → 203.0.113.10 (public IP)

This cannot be achieved with FQDN templating alone, as it doesn't support same-hostname-different-targets scenarios.

Real-world impact on resource management:
A significant practical benefit is avoiding resource duplication when deploying third-party Helm charts (ArgoCD, Grafana, community charts, etc.). Currently, to achieve split-horizon DNS, users must either fork charts (maintenance burden), create duplicate Service/Ingress resources externally (pollutes the system), or use workarounds that still can't solve the same-hostname-different-targets requirement. With --annotation-prefix, you simply configure both sets of annotations in values.yaml—clean, simple, no duplication.

Addressing the "Kubernetes simplicity" concern:
This PR doesn't introduce partial matching or regex—it simply allows customizing the full, exact prefix (e.g., custom.io/ instead of external-dns.alpha.kubernetes.io/). This maintains deterministic, exact key matching that aligns perfectly with Kubernetes' declarative model. The implementation is fully backward compatible (default prefix unchanged) and adds minimal complexity while solving a legitimate architectural gap.

Conclusion:
This feature fills a real gap that cannot be addressed by existing flags. It enables true split-horizon DNS (same hostname, different targets) without resource duplication, particularly valuable for environments running multiple DNS providers or internal/external DNS separation. The three users who have already commented in support demonstrate genuine demand from the community for this functionality. I also would like to support the PR, and I fine it useful for some cases.

[ivankatliarchuk]

/hold cancel

[ivankatliarchuk]

/ok-to-test

[mloiseleur]

@lexfrei For the Chart, would you please also update the changelog in the UNRELEASED section ?

For the docs, you'll need to add it this advanced page to mkdocs.yml.

Apart from that, it LVGTM. Thanks 👍

[mloiseleur]

/lgtm

[mloiseleur]

/easycla

[lexfrei]

Question about documentation discoverability

I noticed that CONTRIBUTING.md at the repository root doesn't reference the detailed developer guides in docs/contributing/ (dev-guide.md, chart.md, design.md, etc.).

While working on this PR, I treated CONTRIBUTING.md as the single entry point for contributors and didn't realize the detailed guides existed until later. Would a PR adding a reference to docs/contributing/ in CONTRIBUTING.md be accepted? This could improve discoverability for future contributors.

[mloiseleur]

While working on this PR, I treated CONTRIBUTING.md as the single entry point for contributors and didn't realize the detailed guides existed until later. Would a PR adding a reference to docs/contributing/ in CONTRIBUTING.md be accepted? This could improve discoverability for future contributors.

Yes, sure. It makes sense 👍

[stevehipwell]

Helm chart changes look good to me.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: stevehipwell
Once this PR has been reviewed and has the lgtm label, please ask for approval from ivankatliarchuk. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[szuecs]

From my side also lgtm and I let @ivankatliarchuk review because of the change request, but the fix lgtm.

@lexfrei I hope we don't see so often such big PRs, because honestly these are a hell to review. Even if AI makes it simple to write a lot of code, it's not for free and likely contains a lot of bugs, so better less code. 46 files is too huge for the next time.

[lexfrei]

@szuecs Agreed! I manually review, test, and often edit everything by hand - so I understand the pain you're describing. Lesson learned - I'll keep PRs smaller next time. This one grew beyond my initial expectations. Thanks for reviewing! 🙏

[ivankatliarchuk]

I'll try to execute some tests against my labs this week

[ivankatliarchuk]

From first look, this not just add support for a use case like split Horizon DNS, but overrides default annotation for external-dns. So before we use to have external-dns.alpha.kubernetes.io/-----xxxx----, now expectation is external-dns to work with any annotation prefix

I'm not sure if blast radius of this change is well understood.

@szuecs are we confident that this what we planning to change?

[lexfrei]

In addition to gochecknoinits, I suggest enabling goconst (with -min-occurrences: 2) to automatically catch hardcoded string duplications like these. Happy to file issues if needed.

[ivankatliarchuk]

Yes. Create a PR for linter.

/lgtm