Blog

Why Is My Kubernetes Service Not Reachable When the Pod Is Running?

· 9 min read · StackMonsters

Your frontend calls http://backend and gets a connection error. kubectl get pods says every backend Pod is Running. Both outputs are accurate, and they don't contradict each other.

console
$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
backend-55bdcddbfd-fpv7q   0/1     Running   0          4m
backend-55bdcddbfd-pts8h   0/1     Running   0          4m
frontend                   1/1     Running   0          4m

$ kubectl exec frontend -- curl -sS http://backend
curl: (7) Failed to connect to backend port 80 after 1 ms: Couldn't connect to server
command terminated with exit code 7

Running only tells you the containers were created and at least one process is running. Whether a Service will send traffic to the Pod, and whether the process accepts it, are separate questions with separate answers.

The Path a Request Takes Through a Service

A request from one Pod to a Service crosses four links. Each one can break on its own:

text
frontend Pod
   │  GET http://backend          DNS: backend → ClusterIP 10.96.128.213
   ▼
Service "backend"                 selector: app=backend
   │                              port 80 → targetPort 8080
   │  EndpointSlice controller: every Pod whose labels match the selector
   ▼
EndpointSlice backend-zq4f4       10.42.0.8   ready: true
   │                              10.42.0.9   ready: false
   │  Service proxy (kube-proxy): routes to ready endpoints only
   ▼
backend Pod 10.42.0.8             a process must accept connections on 10.42.0.8:8080
  1. Selector → labels. The Service's spec.selector must match the labels on the Pods. If it matches nothing, the EndpointSlice is empty.
  2. Readiness. A selected Pod is listed in the EndpointSlice, but traffic only goes to it while its Ready condition is true. A failing readiness probe keeps a Running Pod out of rotation.
  3. targetPort. The Service forwards to targetPort on the Pod IP. If nothing listens there, the connection is refused.
  4. Listening address. Service traffic arrives on the Pod's IP address. A process bound to 127.0.0.1 only accepts connections from inside its own Pod.

The first two produce a Service with no ready endpoints. The last two produce a Service with ready endpoints that refuse the connection. The client error can be identical in both cases, so the EndpointSlice is what tells them apart.

How Each Link Breaks, and What It Looks Like

A selector that matches nothing produces an empty EndpointSlice

The EndpointSlice controller watches every Service with a selector and records the Pods that match it. When no Pod matches, the slice exists but has no endpoints, and kubectl prints <unset>:

console
$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME            ADDRESSTYPE   PORTS     ENDPOINTS   AGE
backend-zq4f4   IPv4          <unset>   <unset>     6m

Compare the selector with the labels the Pods actually carry:

bash
kubectl get svc backend -o jsonpath='{.spec.selector}'
kubectl get pods --show-labels

A selector matches only when every key-value pair in it is present on the Pod. app=backend does not match app=backend-api, and a selector with two labels does not match a Pod that has only one of them.

Fix whichever side is wrong. If it is the Deployment, change the Pod template's labels and its spec.selector together. A Deployment's selector is immutable after creation, so in practice that means recreating the Deployment. Changing the Service is usually simpler:

bash
kubectl patch service backend -p '{"spec":{"selector":{"app":"backend-api"}}}'

A Running Pod can have zero ready endpoints

A readiness probe that fails keeps the Pod's Ready condition false. Note the READY column:

console
$ kubectl get pods -l app=backend
NAME                       READY   STATUS    RESTARTS   AGE
backend-55bdcddbfd-fpv7q   0/1     Running   0          2m
backend-55bdcddbfd-pts8h   0/1     Running   0          2m

The probe's failures are recorded as events on the Pod:

console
$ kubectl describe pod backend-55bdcddbfd-fpv7q
...
    Readiness:      http-get http://:8080/ready delay=0s timeout=1s period=5s #success=1 #failure=3
...
Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Warning  Unhealthy  4s (x24 over 2m)   kubelet  Readiness probe failed: HTTP probe failed with statuscode: 404

The EndpointSlice output can mislead you here. Unlike the older Endpoints API, an EndpointSlice keeps not-ready Pods in the list and marks them with conditions.ready: false. kubectl get endpointslices prints every address regardless of readiness:

console
$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME            ADDRESSTYPE   PORTS   ENDPOINTS             AGE
backend-zq4f4   IPv4          8080    10.42.0.9,10.42.0.8   3m

To see readiness, print the conditions:

bash
kubectl get endpointslices -l kubernetes.io/service-name=backend \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\t"}{.conditions.ready}{"\n"}{end}'
console
10.42.0.9	false
10.42.0.8	false

The fix depends on why the probe fails. Here it requested a path the application doesn't serve. Other common causes are a probe that runs before the application finishes starting (use a startup probe or a longer initialDelaySeconds), and a readiness check that depends on a downstream service that is itself down.

Endpoints that exist but refuse connections

When the slice has ready endpoints and the connection still fails, routing is working. The packet reaches the Pod, and nothing accepts it. Take the Service out of the picture and connect to a Pod IP on the target port from another Pod:

bash
kubectl get pods -o wide                                   # find a backend Pod IP
kubectl exec frontend -- curl -sS -m 2 http://10.42.0.8:8080

If that fails too, the problem is in the backend Pod. There are two usual causes.

The process listens on a different port than targetPort. containerPort in the Pod spec is informational: it doesn't make the process listen there. targetPort has to match the port the application actually opened, which is usually in its startup logs or configuration:

console
$ kubectl get svc backend -o jsonpath='{.spec.ports}'
[{"port":80,"protocol":"TCP","targetPort":80}]

$ kubectl logs deploy/backend
2026-10-10T09:00:17Z INFO  monster-market-api v1 starting
2026-10-10T09:00:17Z INFO  listening on :8080

The process listens on 127.0.0.1. Many frameworks default to loopback, or read a HOST setting that was right on a developer laptop. Inside a Pod, loopback is shared only by the containers of that Pod. Service traffic, traffic from other Pods and the kubelet's HTTP probes all arrive on the Pod IP, so they are refused. The application must listen on 0.0.0.0 (or the Pod IP).

If the client is outside the cluster

A ClusterIP Service is reachable only from inside the cluster. Running curl 10.96.128.213 from your laptop fails even when every link above is healthy. To reach the Service from outside, use kubectl port-forward for a quick test, or expose it through a NodePort or LoadBalancer Service, or through an Ingress or Gateway. See Service types.

A Debugging Order That Doesn't Depend on Guessing

Work from the client toward the process. Each step separates two kinds of failure:

text
1. Does the name resolve?            curl: (6) Could not resolve host
                                     → wrong name or namespace (use backend.<namespace>), or DNS
2. Does the Service select Pods?     EndpointSlice ENDPOINTS <unset>
                                     → compare spec.selector with Pod labels
3. Are the selected Pods ready?      conditions.ready: false, READY 0/1
                                     → kubectl describe pod: probe events
4. Does a Pod accept on targetPort?  curl <pod-ip>:<targetPort> from another Pod fails
                                     → targetPort vs. the port the process opened, and its bind address
5. Is the client inside the cluster? ClusterIP from outside never works
                                     → port-forward, NodePort, LoadBalancer, Ingress

Two caveats apply to the symptoms. With kube-proxy in iptables or nftables mode, a Service with no ready endpoints rejects connections immediately, so clients see "connection refused". Other Service proxies, such as IPVS mode or eBPF-based CNIs, may drop the packets instead, so the client times out. And a NetworkPolicy can block traffic even when every link above is healthy, usually showing up as a timeout. The upstream guide Debug Services covers the DNS and kube-proxy steps in more depth.

Two Faults at Once

Real incidents often have more than one cause, and fixing the first one changes the symptom without fixing the request. Take a Service whose selector requires tier=web while the Pods are labelled tier=api, and whose targetPort is 80 while the process listens on 8080.

The first check shows an empty slice, which points at the selector:

console
$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME            ADDRESSTYPE   PORTS     ENDPOINTS   AGE
backend-7xk2p   IPv4          <unset>   <unset>     5m

After the selector is fixed, the slice fills with ready endpoints, and the request still fails:

console
$ kubectl patch service backend -p '{"spec":{"selector":{"app":"backend","tier":"api"}}}'
service/backend patched

$ kubectl get endpointslices -l kubernetes.io/service-name=backend
NAME            ADDRESSTYPE   PORTS   ENDPOINTS             AGE
backend-7xk2p   IPv4          80      10.42.0.8,10.42.0.9   5m

$ kubectl exec frontend -- curl -sS -m 2 http://backend
curl: (7) Failed to connect to backend port 80 after 1 ms: Couldn't connect to server
command terminated with exit code 7

The symptom moved from "no endpoints" to "endpoints that refuse connections". The slice now reports port 80, and the logs show the process listening on 8080, so the remaining fault is targetPort:

bash
kubectl patch service backend -p '{"spec":{"ports":[{"port":80,"targetPort":8080}]}}'

Re-checking the EndpointSlice after every change is what keeps the second fault from looking like the first fix didn't work.

Recap

  • Running and Ready are different. Running is the Pod phase; Ready is a condition controlled by readiness probes, and only Ready Pods receive Service traffic.
  • The EndpointSlice tells you which half failed. No endpoints, or none ready, means a selector or readiness problem. Ready endpoints that refuse connections mean a port or bind-address problem in the Pod.
  • kubectl get endpointslices lists not-ready addresses too. Check conditions.ready before concluding the endpoints are healthy.
  • Test the layer below. Curl a Pod IP on targetPort from another Pod to take the Service out of the picture.
  • Applications in Pods must listen on 0.0.0.0. Not on 127.0.0.1, and port-forward won't tell you otherwise.