Labels and selectors let a Pod choose nodes. Taints work the other way around: a node refuses Pods unless they carry a matching toleration.
Taint a node
kubectl taint nodes peaks-node-3 dedicated=gpu:NoSchedule
kubectl describe node peaks-node-3 | grep TaintsTaints: dedicated=gpu:NoScheduleA taint has a key, an optional value and an effect:
| Effect | Means |
|---|---|
NoSchedule | new Pods without a toleration are not placed here |
PreferNoSchedule | avoided in scoring, used if nothing else fits |
NoExecute | new Pods are kept out and running ones without a toleration are evicted |
Remove it with a trailing dash: kubectl taint nodes peaks-node-3 dedicated=gpu:NoSchedule-.
Tolerate it
spec:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoScheduleA toleration matches a taint by key, value and effect. operator: Exists with no value matches any value for that key. A Pod may carry many tolerations, one per taint it is willing to accept.
Allowed is not attracted
A toleration lets the Pod use the tainted node. It does not send it there. A training job that must run on the GPU node needs both: a toleration for the taint and a nodeSelector for the node's label. That pairing, taint plus label, is how a pool of special nodes is reserved for the workloads that need it.
Reading it
0/3 nodes are available: 1 node(s) had untolerated taint {dedicated: gpu}, 2 node(s) didn't match Pod's node affinity/selector.Each node's reason is listed. kubectl describe pod prints the Pod's Tolerations:; every Pod already tolerates not-ready and unreachable for 300 seconds, which is how Pods survive a node's brief hiccup.