ingress-nginx Retirement and IngressNightmare: Comparing Replacements and Switching k3s to Traefik

2026-10-19 08:52 (110 minutes ago)

If you leave microk8s running as installed, it is easy to assume that snap's automatic updates keep it current. They do not. A snap tracks the channel chosen at install time (for example 1.24/stable) and only installs patch releases within that channel. The minor version does not change unless you run snap refresh --channel= by hand.

$ snap list microk8s
Name      Version   Rev   Tracking     Publisher    Notes
microk8s  v1.24.17  5872  1.24/stable  canonical**  classic

Upstream support for 1.24 ended in July 2023. When upgrading an old microk8s or k3s, the Ingress controller turns out to be a bigger problem than Kubernetes itself, because ingress-nginx, which is what the microk8s ingress addon runs and which has been widely used with k3s, is no longer maintained.

The end of ingress-nginx maintenance

The ingress-nginx discussed here is kubernetes/ingress-nginx, developed by the Kubernetes community. It is a different product from nginx/kubernetes-ingress, developed by F5 (NGINX). The names are similar, so care is needed.

On November 11, 2025, the official Kubernetes blog announced the retirement in Ingress NGINX Retirement: What You Need to Know. The main points are:

  • Best-effort maintenance continues until March 2026. After that there are no releases, no bug fixes and no security fixes
  • Existing container images and Helm charts are not deleted, so deployments that run today will not stop
  • Users are advised to move to a Gateway API implementation or another Ingress controller

The final release is controller-v1.15.1 from March 19, 2026, supporting Kubernetes 1.31 to 1.35. 1.36 is not part of the official test matrix.

Why it was retired

The reasons are a lack of maintainers and security debt that comes from the design.

  1. Despite its large user base, the project was maintained for years by one or two people volunteering outside working hours. Calls for new maintainers found no one
  2. A distinctive feature was the ability to write nginx configuration directly from annotations such as server-snippet and configuration-snippet. Because of this, anyone who could create an Ingress could take over the controller. The controller has permission to read Secrets in every namespace
  3. A real fix required a rewrite. A Gateway API based successor, InGate, was planned, but it also failed to attract contributors and was discontinued

Security problems

IngressNightmare (CVE-2025-1974 and related)

This is a set of vulnerabilities disclosed by Wiz Research in March 2025. CVE-2025-1974 has a CVSS score of 9.8.

  • The entry point is the ingress-nginx admission webhook, which validates an Ingress before it is created. It is reachable from the Pod network without authentication
  • An attacker sends a crafted Ingress to the webhook and injects nginx configuration through annotations (CVE-2025-1097, CVE-2025-1098, CVE-2025-24514 and others)
  • To validate it, the webhook runs nginx -t against the generated configuration. The injected configuration is read at that point, a shared library uploaded separately is loaded, and arbitrary code runs inside the controller
  • Because the controller can read Secrets in every namespace, this leads to a takeover of the whole cluster

The fixed versions are v1.11.5 and v1.12.1. Wiz estimated that about 40% of cloud environments were affected.

Earlier vulnerabilities of the same kind

Injection caused by annotation values being embedded directly into the nginx configuration had been found repeatedly before.

CVE Description
CVE-2021-25742 Secrets could be read through *-snippet annotations
CVE-2022-4886 Path validation could be bypassed
CVE-2023-5043 / CVE-2023-5044 Configuration injection through configuration-snippet / permanent-redirect

Closing individual holes does not stop the next hole of the same kind from being found. This design problem is one of the reasons for the retirement.

What the distributions did

In its 1.35 release notes, microk8s replaced the NGINX ingress addon with Traefik. The 1.35 / 1.36 addon script (addons/ingress/enable in canonical/microk8s-core-addons) only installs Traefik with Helm and has no option to choose NGINX. There is no NGINX addon among the community addons either.

k3s has always bundled Traefik as its Ingress controller.

Upstream Kubernetes support ends on the following dates. At the time of writing (October 2026), 1.34 has about one week left.

Version End of support
1.34 2026-10-27
1.35 2027-02-28
1.36 2027-06-28

An upgrade started now should target 1.35 or later. Once microk8s is on 1.35 or later, ingress-nginx can no longer be used through the addon.

Comparing the replacements

These are the candidates.

Option Description Drawbacks
Traefik (bundled with the distribution) Default in microk8s / k3s. Updated together with the distribution Raw nginx configuration cannot be written. Anything equivalent to server-snippet is limited to what Middleware can express. The default entrypoint readTimeout is 60 seconds
F5 NGINX Ingress Controller (OSS edition) nginx under the hood. Apache-2.0. v5.7.0 is tested on Kubernetes 1.30 to 1.37 Annotations are a different specification with the nginx.org/ prefix, so every Ingress must be rewritten. Installed and upgraded by yourself with Helm. Some features require the paid NGINX Plus
Self-install the old ingress-nginx v1.15.1 Existing annotations keep working Vulnerabilities are not fixed. 1.36 is out of scope
Stay on 1.35 No rewriting Support for 1.35 ends in February 2027. This only postpones the problem
Install nginx directly on the host nginx from apt terminates TLS and forwards to NodePorts Routing configuration lives outside Kubernetes. A fixed NodePort must be assigned and tracked for each service

F5 NGINX Ingress Controller annotations map to ingress-nginx as follows (annotation reference).

ingress-nginx F5 NGINX Ingress Controller
proxy-body-size nginx.org/client-max-body-size
proxy-read-timeout etc. nginx.org/proxy-read-timeout etc.
proxy-buffer-size nginx.org/proxy-buffer-size
server-snippet nginx.org/server-snippets (the controller needs -enable-snippets)
rewrite-target (regex) nginx.org/rewrite-target + nginx.org/path-regex
upstream-vhost nginx.org/upstream-vhost (variables are not allowed)
proxy-redirect-off No direct equivalent. Use proxy-redirect-from / -to instead

The future standard

There is no single standard controller that succeeds ingress-nginx. What becomes standard is the API: Gateway API. The Ingress API is feature-frozen (not deprecated), and new features go only into Gateway API. Traefik, NGINX Gateway Fabric, Envoy Gateway, Istio, Cilium and others implement Gateway API.

In terms of the default for lightweight distributions, the answer is Traefik. Both k3s and microk8s 1.35 and later now default to Traefik.

F5 NGINX Ingress Controller is a strong choice when you want to keep nginx behavior. However, F5's product for Gateway API is a separate one, NGINX Gateway Fabric, so moving to Gateway API later means switching products. With Traefik, the same controller can move to Gateway API.

In terms of operating cost, Traefik, which is updated together with the distribution, has the advantage. The following steps switch a k3s server that had disabled the bundled Traefik with --disable=traefik and ran ingress-nginx separately over to Traefik.

Migrating on k3s

Upgrading Kubernetes

k3s is upgraded by running the install script again with a different channel. Minor versions are raised one at a time.

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.34 \
  INSTALL_K3S_EXEC="--disable=traefik --disable=servicelb" \
  sh -s - --write-kubeconfig-mode 644

Pass all the arguments used at install time. Any argument you leave out is removed from the systemd unit file. Back up /var/lib/rancher/k3s/server, /etc/rancher/k3s and the k3s binary beforehand.

ingress-nginx v1.13 supports Kubernetes up to 1.33, so switch the ingress once the server is on 1.34.

Removing ingress-nginx

For ingress-nginx installed from the upstream baremetal manifest, resources outside the namespace have to be deleted individually.

kubectl delete validatingwebhookconfiguration ingress-nginx-admission
kubectl delete ns ingress-nginx
kubectl delete clusterrole ingress-nginx ingress-nginx-admission
kubectl delete clusterrolebinding ingress-nginx ingress-nginx-admission
kubectl delete ingressclass nginx

Enabling the bundled Traefik

The Traefik Helm chart bundled with k3s 1.34 is 40.1.5 (the Traefik v3.7 series). As mentioned above, the default entrypoint readTimeout in Traefik v3 is 60 seconds. To keep long requests that had proxy-read-timeout: "1800" under ingress-nginx from being cut off, extend the entrypoint timeouts with a HelmChartConfig. The key locations can be checked in the bundled chart's values.yaml.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        transport:
          respondingTimeouts:
            readTimeout: 1800s
            writeTimeout: 1800s
      websecure:
        transport:
          respondingTimeouts:
            readTimeout: 1800s
            writeTimeout: 1800s

Placing this file in /var/lib/rancher/k3s/server/manifests/ makes k3s apply it automatically. Then reinstall without --disable=traefik --disable=servicelb.

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.34.12+k3s1 \
  sh -s - --write-kubeconfig-mode 644

Even without a bare-metal load balancer, enabling servicelb (klipper-lb) starts a small Pod that receives 80/443 through hostPort, and the node's IP is assigned to Traefik's LoadBalancer Service.

After applying, you can confirm that Traefik's startup arguments include --entryPoints.websecure.transport.respondingTimeouts.readTimeout=1800s and the like. Behavior with long requests has not been verified.

Results and caveats

  • Traefik handles an Ingress without annotations with no rewriting. TLS also uses the certificate in the existing Secret referenced by the Ingress as is
  • Traefik's IngressClass is created as the default, so Traefik also handles Ingresses without ingressClassName. In an environment where the ingress-nginx IngressClass was not marked as default, class-less Ingresses that had been ignored become active after the switch
  • The bundled Traefik version is the same (3.7.13) from k3s 1.34 to 1.36, and the HelmChartConfig timeout settings are kept when upgrading to 1.35 and 1.36

Caveats when migrating microk8s

Annotations specific to ingress-nginx are handled in Traefik as follows.

Annotation Handling in Traefik
proxy-body-size Not needed, since Traefik has no limit by default
proxy-read-timeout etc. Covered by the entrypoint timeout settings
server-snippet Rewrite as Middleware
rewrite-target (regex) Rewrite as ReplacePathRegex Middleware
upstream-vhost / proxy-redirect-off Rewrite as Headers Middleware

Traefik v3.5 and later include a compatibility provider (kubernetesIngressNGINX) that interprets ingress-nginx annotations. Which annotations it supports has not been verified.

The official microk8s procedure is to snap refresh one minor version at a time.

sudo snap refresh microk8s --channel=1.25/stable
microk8s status --wait-ready

Addons and the CNI (calico) are not updated automatically. calico installed in the 1.24 era is the 3.21 series, which does not support Kubernetes 1.25 or later, so the manifest bundled with the snap has to be reapplied partway through. In addition, args/kubelet of older microk8s may still contain --container-runtime=remote, which was removed in kubelet 1.27. snap revert only goes back one step, so take a YAML dump of all resources and a microk8s dbctl backup beforehand.

Because the ingress addon becomes Traefik in 1.35, it is better to split the work into two stages.

  1. Upgrade to 1.34 while keeping the ingress-nginx addon. Deal with API removals (such as PodSecurityPolicy in 1.25) and the calico update here
  2. Rewrite the Ingress annotations for Traefik, then upgrade 1.35 → 1.36

Going from 1.24 to 1.34 takes 10 steps. At 10 to 15 minutes per step for the refresh and checks, that is a 2 to 3 hour job.

Please rate this article (No signup or login required)
Currently unrated
The author runs the application development company Cyberneura.
We look forward to discussing your development needs.

Categories

Archive