Why Is My Kubernetes Service Not Reachable When the Pod Is Running?
· 9 min read · StackMonsters
Your frontend calls http://backend and gets a connection error. kubectl get pods says every backend Pod is Running. Both outputs are accurate, and they don't contradict each other.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
backend-55bdcddbfd-fpv7q 0/1 Running 0 4m
backend-55bdcddbfd-pts8h 0/1 Running 0 4m
frontend 1/1 Running 0 4m
$ kubectl exec frontend -- curl -sS http://backend
curl: (7) Failed to connect to backend port 80 after 1 ms: Couldn't connect to server
command terminated with exit code 7Running only tells you the containers were created and at least one process is running. Whether a Service will send traffic to the Pod, and whether the process accepts it, are separate questions with separate answers.
The Path a Request Takes Through a Service
A request from one Pod to a Service crosses four links. Each one can break on its own:
frontend Pod
│ GET http://backend DNS: backend → ClusterIP 10.96.128.213
▼
Service "backend" selector: app=backend
│ port 80 → targetPort 8080
│ EndpointSlice controller: every Pod whose labels match the selector
▼
EndpointSlice backend-zq4f4 10.42.0.8 ready: true
│ 10.42.0.9 ready: false
│ Service proxy (kube-proxy): routes to ready endpoints only
▼
backend Pod 10.42.0.8 a process must accept connections on 10.42.0.8:8080- Selector → labels. The Service's
spec.selectormust match the labels on the Pods. If it matches nothing, the EndpointSlice is empty. - Readiness. A selected Pod is listed in the EndpointSlice, but traffic only goes to it while its
Readycondition is true. A failing readiness probe keeps a Running Pod out of rotation. - targetPort. The Service forwards to
targetPorton the Pod IP. If nothing listens there, the connection is refused. - Listening address. Service traffic arrives on the Pod's IP address. A process bound to
127.0.0.1only accepts connections from inside its own Pod.
The first two produce a Service with no ready endpoints. The last two produce a Service with ready endpoints that refuse the connection. The client error can be identical in both cases, so the EndpointSlice is what tells them apart.
How Each Link Breaks, and What It Looks Like
A selector that matches nothing produces an empty EndpointSlice
The EndpointSlice controller watches every Service with a selector and records the Pods that match it. When no Pod matches, the slice exists but has no endpoints, and kubectl prints <unset>:
$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zq4f4 IPv4 <unset> <unset> 6mCompare the selector with the labels the Pods actually carry:
kubectl get svc backend -o jsonpath='{.spec.selector}'
kubectl get pods --show-labelsA selector matches only when every key-value pair in it is present on the Pod. app=backend does not match app=backend-api, and a selector with two labels does not match a Pod that has only one of them.
Fix whichever side is wrong. If it is the Deployment, change the Pod template's labels and its spec.selector together. A Deployment's selector is immutable after creation, so in practice that means recreating the Deployment. Changing the Service is usually simpler:
kubectl patch service backend -p '{"spec":{"selector":{"app":"backend-api"}}}'A Running Pod can have zero ready endpoints
A readiness probe that fails keeps the Pod's Ready condition false. Note the READY column:
$ kubectl get pods -l app=backend
NAME READY STATUS RESTARTS AGE
backend-55bdcddbfd-fpv7q 0/1 Running 0 2m
backend-55bdcddbfd-pts8h 0/1 Running 0 2mThe probe's failures are recorded as events on the Pod:
$ kubectl describe pod backend-55bdcddbfd-fpv7q
...
Readiness: http-get http://:8080/ready delay=0s timeout=1s period=5s #success=1 #failure=3
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 4s (x24 over 2m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 404The EndpointSlice output can mislead you here. Unlike the older Endpoints API, an EndpointSlice keeps not-ready Pods in the list and marks them with conditions.ready: false. kubectl get endpointslices prints every address regardless of readiness:
$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-zq4f4 IPv4 8080 10.42.0.9,10.42.0.8 3mTo see readiness, print the conditions:
kubectl get endpointslices -l kubernetes.io/service-name=backend \
-o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\t"}{.conditions.ready}{"\n"}{end}'10.42.0.9 false
10.42.0.8 falseThe fix depends on why the probe fails. Here it requested a path the application doesn't serve. Other common causes are a probe that runs before the application finishes starting (use a startup probe or a longer initialDelaySeconds), and a readiness check that depends on a downstream service that is itself down.
Endpoints that exist but refuse connections
When the slice has ready endpoints and the connection still fails, routing is working. The packet reaches the Pod, and nothing accepts it. Take the Service out of the picture and connect to a Pod IP on the target port from another Pod:
kubectl get pods -o wide # find a backend Pod IP
kubectl exec frontend -- curl -sS -m 2 http://10.42.0.8:8080If that fails too, the problem is in the backend Pod. There are two usual causes.
The process listens on a different port than targetPort. containerPort in the Pod spec is informational: it doesn't make the process listen there. targetPort has to match the port the application actually opened, which is usually in its startup logs or configuration:
$ kubectl get svc backend -o jsonpath='{.spec.ports}'
[{"port":80,"protocol":"TCP","targetPort":80}]
$ kubectl logs deploy/backend
2026-10-10T09:00:17Z INFO monster-market-api v1 starting
2026-10-10T09:00:17Z INFO listening on :8080The process listens on 127.0.0.1. Many frameworks default to loopback, or read a HOST setting that was right on a developer laptop. Inside a Pod, loopback is shared only by the containers of that Pod. Service traffic, traffic from other Pods and the kubelet's HTTP probes all arrive on the Pod IP, so they are refused. The application must listen on 0.0.0.0 (or the Pod IP).
If the client is outside the cluster
A ClusterIP Service is reachable only from inside the cluster. Running curl 10.96.128.213 from your laptop fails even when every link above is healthy. To reach the Service from outside, use kubectl port-forward for a quick test, or expose it through a NodePort or LoadBalancer Service, or through an Ingress or Gateway. See Service types.
A Debugging Order That Doesn't Depend on Guessing
Work from the client toward the process. Each step separates two kinds of failure:
1. Does the name resolve? curl: (6) Could not resolve host
→ wrong name or namespace (use backend.<namespace>), or DNS
2. Does the Service select Pods? EndpointSlice ENDPOINTS <unset>
→ compare spec.selector with Pod labels
3. Are the selected Pods ready? conditions.ready: false, READY 0/1
→ kubectl describe pod: probe events
4. Does a Pod accept on targetPort? curl <pod-ip>:<targetPort> from another Pod fails
→ targetPort vs. the port the process opened, and its bind address
5. Is the client inside the cluster? ClusterIP from outside never works
→ port-forward, NodePort, LoadBalancer, IngressTwo caveats apply to the symptoms. With kube-proxy in iptables or nftables mode, a Service with no ready endpoints rejects connections immediately, so clients see "connection refused". Other Service proxies, such as IPVS mode or eBPF-based CNIs, may drop the packets instead, so the client times out. And a NetworkPolicy can block traffic even when every link above is healthy, usually showing up as a timeout. The upstream guide Debug Services covers the DNS and kube-proxy steps in more depth.
Two Faults at Once
Real incidents often have more than one cause, and fixing the first one changes the symptom without fixing the request. Take a Service whose selector requires tier=web while the Pods are labelled tier=api, and whose targetPort is 80 while the process listens on 8080.
The first check shows an empty slice, which points at the selector:
$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-7xk2p IPv4 <unset> <unset> 5mAfter the selector is fixed, the slice fills with ready endpoints, and the request still fails:
$ kubectl patch service backend -p '{"spec":{"selector":{"app":"backend","tier":"api"}}}'
service/backend patched
$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-7xk2p IPv4 80 10.42.0.8,10.42.0.9 5m
$ kubectl exec frontend -- curl -sS -m 2 http://backend
curl: (7) Failed to connect to backend port 80 after 1 ms: Couldn't connect to server
command terminated with exit code 7The symptom moved from "no endpoints" to "endpoints that refuse connections". The slice now reports port 80, and the logs show the process listening on 8080, so the remaining fault is targetPort:
kubectl patch service backend -p '{"spec":{"ports":[{"port":80,"targetPort":8080}]}}'Re-checking the EndpointSlice after every change is what keeps the second fault from looking like the first fix didn't work.
Recap
- Running and Ready are different. Running is the Pod phase; Ready is a condition controlled by readiness probes, and only Ready Pods receive Service traffic.
- The EndpointSlice tells you which half failed. No endpoints, or none ready, means a selector or readiness problem. Ready endpoints that refuse connections mean a port or bind-address problem in the Pod.
kubectl get endpointsliceslists not-ready addresses too. Checkconditions.readybefore concluding the endpoints are healthy.- Test the layer below. Curl a Pod IP on
targetPortfrom another Pod to take the Service out of the picture. - Applications in Pods must listen on
0.0.0.0. Not on127.0.0.1, andport-forwardwon't tell you otherwise.
