By Duffie Cooley, June 8, 2017
Originally published on the CoreOS blog (archived copy) . Re-published with the authors' permission.
kubectl enables system administrators to interact with Kubernetes using a well-designed command-line interface. The common interactions include running services, grabbing pod logs, or identifying nodes inside of a cluster. For most Kubernetes users, these interactions are second nature.
The kubectl tool has a number of advanced features that can empower users to filter data, aggregate data, or even forward cluster services to a local machine. These tools can be powerful for debugging production users or debugging your application. In our last post, we showed some shell tips for kubectl, and how kubeconfig contexts can provide flexibility for your cluster configurations. Let's explore another set of advanced examples that you can apply to deepen your knowledge of Kubernetes.
All of these tricks assume you have a working Kubernetes cluster and kubectl tool. If you don't have a Kubernetes cluster, we suggest Tectonic for AWS or bare metal and minikube for local laptop development.
All of these examples make use of the Kubernetes API to pull specific information about pods. One way to build a query is to grab all the relevant data and then figure out what the jsonpath expression would be. For example, run kubectl get pods --all-namespaces -o json to see all the data we have to filter on for the 'Filter Kubernetes pods by time' example.
If you don't have an application running, try creating pods labelled run=shop and start a service listening on port 80:
kubectl run shop --replicas=2 --image quay.io/coreos/example-app:v1.0 --port 80 --expose
Then you can see what we are doing with jsonpath.
For more on jsonpath, check out the Kubernetes.io jsonpath support documentation.
kubectl get pods --all-namespaces --sort-by='.metadata.creationTimestamp' -o jsonpath='{range .items[*]}{.metadata.name}, {.metadata.creationTimestamp}{"\n"}{end}'
Given a namespace "your-namespace" and a label query that identifies the pods you are interested in, you can get the logs for all of those pods. If the pod isn't unique, it will fetch the logs for each pod in parallel.
ns='<your-namespace>' label='<yourkey>=<yourvalue>' kubectl get pods -n $ns -l $label -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -I {} kubectl -n $ns logs {}
Given a namespace "your-namespace" and a label query that identifies the pods you are interested in connecting to, this command will connect to the first pod by name. Ensure that you replace 8080 with your pod's port.
ns='<your-namespace>' label='<yourkey>=<yourvalue>' kubectl -n $ns get pod -l $label -o jsonpath='{.items[1].metadata.name}' | xargs -I{} kubectl -n $ns port-forward {} 8080:80
By combining jq with kubectl JSON output, you can make complex queries, like filtering all resources by their create date.
Often, high level statistics can help in debugging. This command will count all of the pods on each node:
kubectl get pods --all-namespaces -o json | jq '.items[] | .spec.nodeName' -r | sort | uniq -c
You can use label queries on nodes. This is often used in configuring deployments that need certain constraints. For more info on selectors, see: kubectl explain deployment.spec.selector
kubectl get nodes -l 'master' or kubectl get nodes -l '!master'
You can list all labels with the --show-labels argument to any Kubernetes object. kubectl get nodes --all-namespaces --show-labels
This will generate a JSON document that has a Kubernetes node name, and then a list of all of the pod names running on the node. This is very useful for debugging placement or load issues.
kubectl get pods --all-namespaces -o json | jq '.items | map({podName: .metadata.name, nodeName: .spec.nodeName}) | group_by(.nodeName) | map({nodeName: .[0].nodeName, pods: map(.podName)})'
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.status.addresses[?(@.type=="ExternalIP")].address}{"\n"}{end}'
Kubernetes has a database of nodes in the cluster which you can query with kubectl get nodes. This is a powerful database for automation and integration with existing tools. One such tool is the Fabric SSH utility, which is known as a fabfile.py. Introduced by CoreOS, this integrates Kubernetes nodes and Fabric together, enabling really powerful capabilities like SSH'ing into all machines running in a particular AWS failure domain.
fab -u core -R failure-domain.beta.kubernetes.io/zone=us-west-2a -- date
Learn more at the Fabric Kubernetes Nodes project.
Many other resources exist for working with the kubectl command line interface. Be sure to take a look at the cheat sheet in the kubernetes.io docs section as well!
Take a look at our recent webinar where we covered all of these features.