To quickly setup kubernetes and monitoring using kops, follow my example github repository.
Recently, i am helping our client move their applications to containerized microservices.
docker containers provide many benefits. However, docker alone does not allow us to orchestrate our containers in productions. To solve that problem, we need a container platform/orchestrator technology. We adopted
kubernetes as our containers orchestrator of choice. Kubernetes allow us to horizontally scale our microservices across nodes to provide highly availablility and scalability.
In this post, i would like to document how we are explored the various orchestrator options and how we are running
kubernetes in production.
KOPS ( Kubernetes Operation)
There are various ways to operate a kubernetes clusters and it can range from being really complex to simple hosted cloud solutions.
These are some of the various tools and platforms:
- Kops ( run on aws )
- Google container engine ( google cloud )
- Jujucharms, canonical distribution of kubernetes ( baremetal, metal cloud ( Machine as a service ), virtual machines )
- Kubeadm ( baremetal, virtual machines )
After evaluating the various options, we decided with
kops which will provision our cluster on
aws. We found
kops to be the most robust and it works seamlessly for us. Using
kops we get multi-az nodes deployment and auto scaling group at the cluster level. On top of it, our client is already on
kops project has a strong community behind it.
Learn more about
The first deployment to run in kubernetes is the
kubernetes dashboard. Running kubernetes dashboard allow us to visualize what kubernetes resources are running or getting scheduled, destroyed. It also shows quickly let us know if there is any errors in the cluster.
elasticsearch outside of kubernetes to store log data generated by fluentd and metricbeat daemonsets. Running ELK outside kubernetes is more robust, in case kubernetes clusters goes down, we can still see logs on what happened before kubernetes died.
It is important that before going into production we setup centralized logging of pods/containers running in kubernetes. Pods/containers are running in different kubernetes nodes, scaling up and down, it is hard to understand what is going on without centralized logging. What we want is a single place to look logs from stdout and stderr streams of running containers to quickly detect and diagnose issues.
fluentd as daemonsets on every kubernetes nodes. In kubernetes, all containers write their stdout/stderr streams to the
/var/log/containers/ folders. Fluentd can be configured to monitor this folder and then send logs into elasticsearch.
Monitoring is done using metricbeat and kube-state-metric.
kube-state-metric is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
Metricbeat grab data from
Kubernetes dashboard in kibana:
Metricbeat dashboard in kibana:
Centralized logging and monitoring is important to setup before moving to production with kubernetes. I have created an example kubernetes github repo that contains the kubernetes yaml files to deploy everything we discussed.