On this page
Kubernetes 1.29, released on 13 December 2023, turned on native sidecar containers by default. The feature arrived as alpha in 1.28 behind the SidecarContainers feature gate and is beta in 1.29. Amazon EKS added 1.29 on 23 January 2024, and AKS has had 1.29.0 in preview since its 7 February release, so many teams can now use it in a managed cluster for the first time.
The change is small in YAML and large in behaviour. A sidecar is now an init container with restartPolicy: Always. Kubernetes starts it before the main containers, keeps it running, stops it after them, and no longer lets it hold a Job open. This article runs seven scenarios on kind clusters and reports what actually happened, including two cases where the new behaviour can still lose data, and one where the same manifest on a 1.28 cluster leaves the Pod stuck forever.
The problem sidecars had before 1.29
A sidecar used to be an ordinary second container in the Pod: a log shipper, a service mesh proxy, a Cloud SQL or database proxy, a secrets agent. Kubernetes treated it as an equal of the application container, which caused three long-standing problems:
- Jobs never finished. A Job’s Pod completes only when all its containers have exited. The main container finishes; the log shipper does not; the Job hangs. Teams worked around it with shared files,
curlcalls to a proxy’s quit endpoint, or wrapper scripts. - No start order. The application could start before its proxy was ready and fail its first outbound calls.
- No stop order. On deletion every container got SIGTERM at once, so a proxy could exit while the application was still draining requests through it.
The first problem is easy to reproduce. This Job runs a five-second task next to a log shipper written as a normal container:
# The pre-1.29 pattern: the log shipper is just another container.
apiVersion: batch/v1
kind: Job
metadata:
name: classic-sidecar
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: main
image: busybox:1.36.1
command: ["sh", "-c", "echo working; sleep 5; echo done"]
- name: log-shipper
image: busybox:1.36.1
command: ["sh", "-c", "while true; do sleep 1; done"]After 60 seconds the Job had no condition at all. The main container had completed, but the Pod was still Running because the log shipper was:
[v1.29/job-classic] server v1.29.2
job_condition: none after 60 s
seconds_to_condition: None
pod_phase: Running
log-shipper main reason=None exit=None restarts=0
main main reason=Completed exit=0 restarts=0The 1.29 model
A native sidecar is declared in initContainers, with one new field. Init containers normally run one after another and must exit before the next starts. With restartPolicy: Always, the kubelet starts the container, considers it done as soon as it has started (or, if it has a startup probe, once the probe passes) and moves on, but keeps it running and restarts it if it exits:
# Kubernetes 1.29: the log shipper is an init container with restartPolicy: Always.
apiVersion: batch/v1
kind: Job
metadata:
name: native-sidecar
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
initContainers:
- name: log-shipper
image: busybox:1.36.1
restartPolicy: Always
command: ["sh", "-c", "while true; do sleep 1; done"]
containers:
- name: main
image: busybox:1.36.1
command: ["sh", "-c", "echo working; sleep 5; echo done"]%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E'}}}%%
sequenceDiagram
participant K as kubelet
participant S as Sidecar (init, restartPolicy Always)
participant M as Main container
K->>S: start
Note over K,S: wait for the startup probe, if any
K->>M: start
M-->>K: exits (Job finished)
K->>S: SIGTERM
S-->>K: exits, or SIGKILL when the grace period runs out
Note over K: Pod Succeeded, Job CompleteYou can check whether the gate is on from the API server’s metrics, with kubectl get --raw /metrics. The 1.28.7 test cluster reported the alpha gate off:
kubernetes_feature_enabled{name="SidecarContainers",stage="ALPHA"} 0and the 1.29.2 cluster reported the beta gate on:
kubernetes_feature_enabled{name="SidecarContainers",stage="BETA"} 1Test setup
Every scenario ran on kind v0.22.0, released on 14 February 2024, with its default node image kindest/node:v1.29.2 pinned by digest. The 1.28 scenario used kindest/node:v1.28.7. kubectl was v1.29.2 and every container used busybox:1.36.1, loaded into the cluster before the tests so image pulls did not affect timings. A Python harness applied each manifest, waited, read container states from the API and deleted what it created.
Status timestamps from the API have one-second resolution, which is too coarse to see a shutdown order. So in the two shutdown scenarios each container appends a line to a file on the node when it gets SIGTERM and when it exits. The timestamp is the node’s /proc/uptime, one clock for every container on the node. This is the second sidecar’s definition; the others differ only in name and in how long they take to stop:
- name: second-sidecar
image: busybox:1.36.1
restartPolicy: Always
volumeMounts: [{name: events, mountPath: /events}]
command: ["sh", "-c", "log() { echo \"$(cut -d' ' -f1 /proc/uptime) second-sidecar $1\" >> /events/log; }; trap 'log TERM; sleep 1; log exit; exit 0' TERM; log start; while true; do sleep 0.2; done"]Test results

Jobs complete, after the sidecar stops. With the log shipper as a native sidecar the Job reached Complete. It took 40 seconds for a five-second task, and the sidecar’s exit code shows why:
[v1.29/job-native] server v1.29.2
job_condition: Complete
seconds_to_condition: 40.3
pod_phase: Succeeded
log-shipper init reason=Error exit=137 restarts=0
main main reason=Completed exit=0 restarts=0When the main container finished, the kubelet sent the sidecar SIGTERM. The sidecar was a shell loop running as process 1 with no SIGTERM handler, and Linux does not deliver default-action signals to process 1 in a container, so it ignored the signal. The kubelet waited out the 30-second default grace period and then killed it, exit code 137. The Job still counted as Complete; only the main container’s result matters. But every run of that Job now takes 30 seconds longer than it should. Real log shippers and proxies handle SIGTERM; check yours does, or set a shorter terminationGracePeriodSeconds on Jobs.
Start order works, with a startup probe. The proxy in this scenario creates a ready file after eight seconds, and its startup probe checks for it every second:
# The proxy takes 8 seconds to become ready. Does main wait for it?
apiVersion: v1
kind: Pod
metadata:
name: startup-order
spec:
restartPolicy: Never
initContainers:
- name: proxy
image: busybox:1.36.1
restartPolicy: Always
command: ["sh", "-c", "sleep 8; touch /tmp/ready; while true; do sleep 1; done"]
startupProbe:
exec:
command: ["test", "-f", "/tmp/ready"]
periodSeconds: 1
failureThreshold: 30
containers:
- name: main
image: busybox:1.36.1
command: ["sh", "-c", "sleep 3600"]The main container started 9 seconds after the proxy, so the kubelet waited for the probe. Without a startup probe the kubelet moves on as soon as the sidecar’s process has started, which is not the same as ready to serve.
Shutdown is ordered. When the Pod was deleted, the main container got SIGTERM first. Only after it had exited did the second sidecar get SIGTERM, and only after that one had exited did the first sidecar get it:
[v1.29/termination-order] server v1.29.2
delete_returned_after_s: 6.0
0.00s main TERM
3.01s main exit
3.05s second-sidecar TERM
4.05s second-sidecar exit
4.09s first-sidecar TERM
5.09s first-sidecar exit
final first-sidecar exit=0 reason=Completed
final second-sidecar exit=0 reason=Completed
final main exit=0 reason=Completed
The order is the reverse of declaration, so declare sidecars in dependency order: something every other container needs, such as a network proxy, goes first; a log shipper that should outlive the proxy’s own logs goes after it.
The grace period is shared, and sidecars come last. In the fifth scenario the main container ignores SIGTERM, the grace period is 10 seconds, and the log shipper needs five seconds to flush:
[v1.29/grace-squeeze] server v1.29.2
delete_returned_after_s: 13.0
0.00s main TERM-ignored
10.00s log-shipper TERM
final log-shipper exit=137 reason=Error
final main exit=137 reason=ErrorThe sidecar received SIGTERM only at 10 seconds, when the kubelet killed the main container at the end of the grace period. It never logged “flushed” and ended with exit code 137. The grace period covers the whole Pod, not each container, and sidecars get whatever is left. A main container that drains slowly can use up the time a log shipper needs to send the last lines, which are usually the ones that explain why the Pod was being stopped.
Size the grace period as a sum, not a maximum. Take the main container’s drain time, including any preStop hook, because hooks run inside the same grace period. Then add each sidecar’s own shutdown time, because the sidecars stop one after another. A web container that drains for 20 seconds, a proxy that needs 2 and a log shipper that needs 5 want at least 27 seconds, so the 30-second default leaves only three to spare. Keeping the main container’s drain short is usually cheaper than raising the grace period for every Pod.
Sidecars restart even when the Pod does not. The sixth scenario’s sidecar exits with code 1 after three seconds, in a Job whose Pod has restartPolicy: Never. The kubelet restarted it 2 times while the main container ran, and the Job completed:
[v1.29/sidecar-crash] server v1.29.2
job_condition: Complete
seconds_to_condition: 29.7
pod_phase: Succeeded
log-shipper init reason=Error exit=1 restarts=2
main main reason=Completed exit=0 restarts=0That is the point of restartPolicy: Always on the container, and it is better behaviour than before, when a crashed log shipper stayed dead for the rest of the Job. It also means a broken sidecar does not fail the Job, so alert on sidecar restart counts.
The 1.28 trap
The last scenario applied the native sidecar Job to a 1.28.7 cluster, where the SidecarContainers gate is off by default. The API server accepted the manifest without an error or a warning, dropped the restartPolicy field, and stored an ordinary init container:
[v1.28/native-on-old] server v1.28.7
pod_phase: Pending
stored_restartPolicy: (field dropped)
job_condition_after_45s: None
log-shipper init reason=None exit=None restarts=0
main main reason=PodInitializing exit=None restarts=0An ordinary init container must exit before the main containers start. A log shipper never exits, so the Pod stayed Pending with the main container waiting in PodInitializing. A Job like this sits there until someone notices. This is the standard Kubernetes rule for fields behind a disabled feature gate: they are dropped, not rejected. In practice it means a Helm chart or manifest that uses native sidecars must not reach a cluster older than 1.29, or a 1.28 cluster without the gate. Pin the chart’s kubeVersion to >=1.29.0-0, or check kubectl version in the pipeline, so the deploy fails loudly instead of hanging.
The gate has to be on in the API server and in the kubelet on the node that runs the Pod. During an upgrade, the control plane is usually upgraded before the nodes. Until every node pool is on 1.29, schedule sidecar workloads only onto upgraded nodes, or wait for the upgrade to finish.
Resource requests and service meshes
Native sidecars also change how the scheduler counts a Pod’s resources. An ordinary init container’s request only matters during initialisation, so the Pod’s effective request is the larger of the biggest init container and the sum of the app containers. A native sidecar runs alongside the app containers, so its request is added to that sum. During initialisation, each later init container’s request is counted together with the sidecars declared before it. In practice: a proxy that requested 100m CPU as an ordinary container still costs 100m as a native sidecar, but an init container placed after it now needs room for both.
Most sidecars in production are not written by hand; a service mesh or an agent’s mutating webhook injects them. Those keep the old behaviour until the injector puts the container in initContainers. Istio has had experimental support since the 1.19 release in 2023, behind the ENABLE_NATIVE_SIDECARS setting on istiod, which is off by default. Check what your mesh and agents do before expecting the start and stop order described here; a Pod can mix both kinds, and only the native ones get the new lifecycle.
Migrating an existing sidecar
- Move the container from
containerstoinitContainersand addrestartPolicy: Always. Put it after any ordinary init containers that it depends on, and before any it serves. - Add a startup probe if the main container must not start before the sidecar is ready, such as a proxy that must accept connections first.
- Delete the workarounds: shared-file signals, quit endpoints called from the main container, wrapper scripts that kill the sidecar.
- Check the sidecar handles SIGTERM and exits promptly. If it runs as process 1 through a shell, use
execor an init such as tini so the signal reaches it. - Review
terminationGracePeriodSeconds: it must cover the main container’s drain time plus every sidecar’s shutdown time, one after another. - Sidecar injectors from service meshes and agents decide where the container goes. Until yours supports native sidecars, injected sidecars keep the old behaviour.
Decision framework
| Situation | Recommendation |
|---|---|
| Jobs or CronJobs with a log shipper or proxy | Move to native sidecars on 1.29; this is the clearest win |
| Long-running Deployments with a mesh proxy | Worth it for start and stop order; wait for your mesh’s support if it injects the proxy |
| Clusters still on 1.28 | Do not ship native sidecar manifests there; the Pod hangs in Init |
| Sidecar that needs time to flush on shutdown | Raise terminationGracePeriodSeconds to cover main plus sidecars, and keep main’s drain short |
| Sidecar that must be ready before the app starts | Give it a startup probe |
| Sidecar that crashes now and then | Native sidecars restart it; alert on its restart count, because the Job will not fail |
All seven scenarios ran in one GitHub Actions run (run) on kind v0.22.0 with node images v1.29.2 and v1.28.7, kubectl v1.29.2 and busybox 1.36.1. The outputs above are copied from that run’s summary file and metric files. kind only, one node: no managed service, no service mesh, and no cluster with mixed control-plane and node versions was tested. Release dates come from the Kubernetes, AWS and AKS announcements.
Key takeaways
- In Kubernetes 1.29 a sidecar is an init container with restartPolicy: Always; the feature is beta and on by default.
- Jobs with native sidecars complete, but only after the sidecar exits, so a sidecar that ignores SIGTERM adds the full grace period to every run.
- Start order needs a startup probe; stop order is automatic, main containers first and then sidecars in reverse order of declaration.
- The grace period is shared and sidecars come last; a slow main container can leave a log shipper no time to flush.
- On 1.28 with the gate off, the API silently drops restartPolicy and the Pod waits in Init forever.
If you have moved a mesh proxy or log shipper to a native sidecar and hit something not covered here, let me know in the comments below.
The complete sample for this article is on GitHub: nithinmohantk/blog-samples-working-repo/k8s-129-native-sidecars. Clone it with git clone https://github.com/nithinmohantk/blog-samples-working-repo.git and open the k8s-129-native-sidecars folder. Its README says how to run it and what has been verified.
Glossary
- Sidecar container
- A helper container that runs alongside the application container in the same Pod, such as a log shipper or proxy.
- Native sidecar
- An init container with restartPolicy: Always, supported from Kubernetes 1.28 (alpha) and on by default from 1.29 (beta).
- Init container
- A container that runs before the Pod’s main containers; ordinary init containers must exit successfully first.
- SidecarContainers
- The feature gate that controls native sidecars; it must be on in the API server and the kubelet.
- Feature gate
- A named switch that turns a Kubernetes feature on or off; alpha gates are off by default, beta gates usually on.
- Startup probe
- A check the kubelet runs until it passes; for a native sidecar it decides when the next container may start.
- terminationGracePeriodSeconds
- How long the kubelet waits after SIGTERM before killing the Pod’s containers; 30 seconds by default.
- SIGTERM
- The signal asking a process to stop; SIGKILL follows if it has not stopped when the grace period ends.
- Exit code 137
- 128 plus 9: the process was killed with SIGKILL.
- kind
- Kubernetes in Docker, a tool that runs a cluster’s nodes as Docker containers.
- Job
- A Kubernetes workload that runs Pods to completion and records whether they succeeded.
References & Further Reading
- → Kubernetes v1.29: Mandala— Release announcement, 13 December 2023
- → Sidecar Containers (v1.29 documentation)— The feature gate, start order and termination order
- → KEP-753: Sidecar Containers— Design and graduation history
- → Amazon EKS and Amazon EKS Distro now support Kubernetes version 1.29— What's New, 23 January 2024
- → AKS release 2024-02-07— AKS 1.29.0 in preview
- → kind v0.22.0— Release of 14 February 2024 used for the tests
- → Sample code for this article— Manifests, harness, results and tests
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.