Simply put, kubernetes is a tool for managing computing resources. It does this very efficently by abstracting your hardware into one (or more, if you like) big computing resource and therefore highly efficient use of your hardware with very little overhead unlike Virtual Machines for example (doesn’t mean there is no use case for them anymore though as there are also pros and cons as it is with every technology).
However, kubernetes mostly manages containers like docker, rkt or podman which allow you to build cost efficient and high available (microservice) architectures.<\/p>\n\n\n\n
In this post I\u2019ll cover how to set up a kubernetes cluster (master and worker) and the most basic commands.<\/p>\n\n\n\n
Let’s install the hosts (kubernetes-master1, kubernetes-node1 ):<\/p>\n\n\n\n
apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common<\/code><\/pre>\n\n\n\nWe need some basic packages, so that we can work with external repositories.
(If you are behind a proxy, see instructions for working behind a proxy in the tips section at the end of this post).<\/p>\n\n\n\n
curl -fsSL https:\/\/download.docker.com\/linux\/debian\/gpg | sudo apt-key add -\nadd-apt-repository \"deb [arch=amd64] https:\/\/download.docker.com\/linux\/debian $(lsb_release -cs) stable\"\napt-get update\napt-get install docker-ce docker-ce-cli containerd.io<\/code><\/pre>\n\n\n\nNow docker is installed on our master and our node(s)<\/p>\n\n\n\n
vi \/etc\/default\/grub : GRUB_CMDLINE_LINUX=\"cgroup_enable=memory swapaccount=1\"<\/code><\/pre>\n\n\n\nEdit your grub config and enable the memory controlgroup (node only).<\/p>\n\n\n\n
vi \/etc\/sysctl.conf\nnet.ipv4.conf.all.forwarding=1\nnet.ipv4.ip_nonlocal_bind=1\nnet.bridge.bridge-nf-call-iptables=1\n\nnet.ipv6.conf.all.disable_ipv6=1\nnet.ipv6.conf.default.disable_ipv6=1\nnet.ipv6.conf.lo.disable_ipv6=1<\/code><\/pre>\n\n\n\nOn your node(s) only, edit your sysctl to enable the use of the docker firewall, forwarding and the binding of non-local ips, so that we may expose our services via kubernetes later. Also I found it convenient to disable ipv6 (didn’t get it to work with ipv6 enabled).<\/p>\n\n\n\n
vi \/etc\/hosts: \n192.168.1.10 kubernetes-master1.your.domain kubernetes-master1\n192.168.1.11 kubernetes-node1.your.domain kubernetes-node1<\/code><\/pre>\n\n\n\nI also add master and nodes to \/etc\/hosts here, but you may skip this if you trust in the reliability of your local dns (or use something like dnsmasq).<\/p>\n\n\n\n
vi \/etc\/docker\/daemon.json\n{\n \"exec-opts\": [\"native.cgroupdriver=systemd\"],\n \"log-driver\": \"json-file\",\n \"log-opts\": {\n \"max-size\": \"100m\"\n },\n \"storage-driver\": \"overlay2\",\n \"dns\": [\"192.168.1.33\",\"192.168.1.34\"]\n}<\/code><\/pre>\n\n\n\n This Part is important (at least at the moment, when working with debian).
We change the cgroup driver to systemd, so docker cgroups are handled by systemd.
Also we set the storage driver to “overlay2” (docker needs an overlay fs to handle local filesystem access, here we use overlayfs2 because the old AUFS isn’t supported anymore by recent linux kernels.
You may also set things like dns here, but that’s not really necessary under normal circumstances (here it is though…).<\/p>\n\n\n\n
mkdir -p \/etc\/systemd\/system\/docker.service.d<\/code><\/pre>\n\n\n\nBecause debian (at least atm) leaves it up to the user to handle the configuration of docker with systemd, we have to create this directory (also see: https:\/\/kubernetes.io\/docs\/setup\/cri\/).<\/p>\n\n\n\n
systemctl daemon-reload\nsystemctl restart docker<\/code><\/pre>\n\n\n\ndocker should be running now on your master and nodes.<\/p>\n\n\n\n
Troubleshooting Tips:<\/h6>\n\n\n\nrm -f \/var\/lib\/docker\/network\/files\/local-kv.db<\/code><\/pre>\n\n\n\n If docker won’t start because of network\/virtual ip problems, delete the above file as it was created during the installation but may contain wrong information now, because of our configurations. It will get recreated on restart with the correct information.<\/p>\n\n\n\n
ip link add name docker0 type bridge && ip addr add dev docker0 192.168.5.1\/24<\/code><\/pre>\n\n\n\nIf that doesn’t help create a virtual ip address, delete the above file and the restart (virtual ip will get re-created on reboot because systemd handles docker and docker has the correct info then), you may not run into this but I encountered this problem several times.<\/p>\n\n\n\n
installing and configuring kubernetes<\/h5>\n\n\n\n
Like with the docker installation, everything here must be done on the master and the nodes (I’ll mark the few exceptions explicitly).<\/p>\n\n\n\n
curl -s https:\/\/packages.cloud.google.com\/apt\/doc\/apt-key.gpg | sudo apt-key add -\nadd-apt-repository \"deb http:\/\/apt.kubernetes.io\/ kubernetes-xenial main\"\napt-get update<\/code><\/pre>\n\n\n\nConfigure the kubernetes sources\/repositories, at the moment these are the xenial sources (they work for debian too of course).<\/p>\n\n\n\n
deb-src [arch=amd64] https:\/\/download.docker.com\/linux\/debian stretch stable\ndeb http:\/\/apt.kubernetes.io\/ kubernetes-xenial main<\/code><\/pre>\n\n\n\nYour sources.list (or whatever you use) should now contain the above entries.<\/p>\n\n\n\n
apt-get install ebtables socat conntrack<\/code><\/pre>\n\n\n\nInstall some dependencies first.<\/p>\n\n\n\n
apt-get install -t kubernetes-xenial -y kubelet kubeadm kubectl<\/code><\/pre>\n\n\n\nNow install kubernetes.<\/p>\n\n\n\n
vi \/etc\/systemd\/system\/kubelet.service.d\/10-kubeadm.conf\n\nEnvironment=\"KUBELET_CGROUP_ARGS=--cgroup-driver=systemd\"\nExecStart=\/usr\/bin\/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CGROUP_ARGS<\/code><\/pre>\n\n\n\nHere we go again, cgroups driver has to be configured and this configuration has to be added to ExecStart Parameters.<\/p>\n\n\n\n
systemctl daemon-reload\nsystemctl kubelet restart<\/code><\/pre>\n\n\n\n Issue the above command on the master<\/em><\/strong>.<\/p>\n\n\n\n<\/p>\n\n\n\n
kubeadm init --pod-network-cidr=10.244.0.0\/16 --apiserver-advertise-address=192.168.1.90 --kubernetes-version \"1.14.0\"<\/code><\/pre>\n\n\n\nNow execute the above command on your master<\/em><\/strong> to initialize the control server with the API IP to advertise (the IP of your master node) and a range for your pod network (the virtual network your services will use to communicate, make sure<\/strong> that the pod network is the same as written above, <\/strong>it won’t work otherwise in flannel).
One may give a lot of parameters here (https:\/\/kubernetes.io\/docs\/reference\/setup-tools\/kubeadm\/kubeadm-init\/) but for now this should be sufficient.
This generates a token (and prints the entire command) one can use to join nodes<\/em><\/strong> to the cluster, after<\/strong> deploying a pod network, let’s use flanneld because it’s simple::<\/p>\n\n\n\nkubectl apply -f https:\/\/raw.githubusercontent.com\/coreos\/flannel\/master\/Documentation\/kube-flannel.yml<\/code><\/pre>\n\n\n\nNow it’s time to join our nodes with the previously generated token:<\/p>\n\n\n\n
kubeadm join 192.168.1.90:6443 --token j54q85.kxpvl5hwlssvcd3 --discovery-token-ca-cert-hash sha256:dde35c22122fa8fbc0c16ddd448dc98ce1e22341e50a1463ae0dda99974cfd9f <\/code><\/pre>\n\n\n\nKubernetes should now be up and running (for now this is kubelet, which is the kubernetes node agent that runs on every cluster node, be it master or worker, read more about it, here: https:\/\/kubernetes.io\/docs\/reference\/command-line-tools-reference\/kubelet\/).<\/p>\n\n\n\n
\n\n\n\nYour kubernetes cluster should be up and running!<\/strong>
(see the below section for some Troubleshooting Tips (if the join didn’t work, you are behind a proxy pod network didn’t start, etc.)
Also let’s deploy something<\/strong>: the kubernetes dashboard<\/em> (which is extremely useful as a graphical user interface and for a general overview about your cluster). <\/p>\n\n\n\n
\n\n\n\nTips and Troubleshooting<\/h5>\n\n\n\n
Check if your cluster is working as expected (do it to see if your join worked):<\/p>\n\n\n\n
kubectl get nodes\nroot@kubmastertest:~# kubectl get nodes\nNAME STATUS ROLES AGE VERSION\nkubmastertest Ready master 17h v1.14.0\nkubnodetest01 NotReady <none> 4m23s v1.14.0<\/code><\/pre>\n\n\n\nObviously everything should be in state “Ready”. Here it is not and I fixed it by manually editing flannel config in the end, on the node<\/strong>:<\/p>\n\n\n\nmkdir -p \/etc\/cni\/net.d\nvi \/etc\/cni\/net.d\/10-flannel.conflist\n{\n \"name\": \"cbr0\",\n \"plugins\": [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\": true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n }\n }\n ]\n}<\/code><\/pre>\n\n\n\n
\n\n\n\n if you should “forget” the join command, just look it up on your master with: <\/p>\n\n\n\n
kubeadm token create --print-join-command<\/code><\/pre>\n\n\n\n
\n\n\n\nIf you are behind a proxy, configure the following to let docker and kubernetes communicate (though you might encounter several issues as these things are not build to be run behind a proxy…):<\/p>\n\n\n\n
vi \/etc\/environment\nhttp_proxy=\"http:\/\/192.168.255.1:3128\"\nhttps_proxy=\"http:\/\/192.168.255.1:3128\"\nno_proxy=\"127.0.0.1, localhost, alltheserversyouwanttobeexcluded <\/code><\/pre>\n\n\n\nEdit \/etc\/environment, which is OS default for using proxies.
One has to exclude servers by single ips, because the no_proxy directive ignores netmasks.<\/p>\n\n\n\n
vi \/etc\/apt\/apt.conf.d\/99HttpProxy\nhttp_proxy=\"http:\/\/192.168.255.1:3128\"\nhttps_proxy=\"http:\/\/192.168.255.1:3128\"\nAcquire::http::Proxy::my.debianrepo.de DIRECT;\nAcquire::http::Proxy::corporate.debian.repo.de DIRECT;<\/code><\/pre>\n\n\n\nExcluding repositories is done via “DIRECT”, so one can use internal repos and the external ones, mentioned above.<\/p>\n\n\n\n
vi \/etc\/systemd\/system\/docker.service.d\/https-proxy.conf\nvi \/etc\/systemd\/system\/docker.service.d\/http-proxy.conf\n[Service]\nEnvironment=\"HTTPS_PROXY=http:\/\/192.168.255.1:3128\"\nEnvironment=\"NO_PROXY=127.0.0.1, localhost, alltheserversyouwanttoexclude\"\n[Service]\nEnvironment=\"HTTP_PROXY=http:\/\/192.168.255.1:3128\"\nEnvironment=\"NO_PROXY=127.0.0.1, localhost, alltheserversyouwanttoexclude\"<\/code><\/pre>\n\n\n\nProxy settings for docker.<\/p>\n\n\n\n
\n\n\n\nkubeadm complains about missing kubeconfig or the like:<\/p>\n\n\n\n
mkdir -p $HOME\/.kube\nsudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config\nsudo chown $(id -u):$(id -g) $HOME\/.kube\/config\n\nuseradd -U -G users,sudo -m -s \/bin\/bash kubeadm\nvi \/etc\/sudoers\nkubeadm ALL=(ALL) NOPASSWD:ALL<\/code><\/pre>\n\n\n\nTry adding a kubeadm user withthe kubeadm config or just do it directly as root.<\/p>\n\n\n\n
\n\n\n\n
\n\n\n\n Kubernetes dashboard deployment<\/h5>\n\n\n\n
We use RBAC (Role Based Authentication) with our kubernetes cluster, this allows us to manage access and rights on a granular level. First we need to create a user:<\/p>\n\n\n\n
vi dashboard-adminuser.yaml\n\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: admin-user\n namespace: kube-system<\/code><\/pre>\n\n\n\nIn case, that the role “admin-user” already exists, just add a cluster role binding:<\/p>\n\n\n\n
vi dashboard-adminuser.yaml\n\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRoleBinding\nmetadata:\n name: admin-user\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: cluster-admin\nsubjects:\n- kind: ServiceAccount\n name: admin-user\n namespace: kube-system<\/code><\/pre>\n\n\n\nkubectl apply -f dashboard-adminuser.yaml<\/code><\/pre>\n\n\n\nAnyway, create it with the above command.
Now find out the auth token for the user\/role, we will need it to log into the dashboard (unfortunatley ldap is not supported as of yet).:<\/p>\n\n\n\n
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')<\/code><\/pre>\n\n\n\n Now deploy the dashboard:<\/p>\n\n\n\n
kubectl apply -f https:\/\/raw.githubusercontent.com\/kubernetes\/dashboard\/v2.0.0-rc5\/aio\/deploy\/recommended.yaml<\/code><\/pre>\n\n\n\nhttp:\/\/192.168.1.90:8001\/api\/v1\/namespaces\/kube-system\/services\/https:kubernetes-dashboard:\/proxy\/<\/a><\/p>\n\n\n\nThe dashboard should be accessible now on your api server address.
With newer Dashboards you’ll need a client certificate (if you use the method described above and access it through the api), you may extract the information from your .kube\/config.<\/p>\n\n\n\n
grep 'client-certificate-data' $HOME\/.kube\/config | awk '{print $2}' | base64 -d >> admin.crt\n\ngrep 'client-key-data' $HOME\/.kube\/config | awk '{print $2}' | base64 -d >> admin.key\n\nopenssl pkcs12 -export -in admin.crt -inkey admin.key -out kub_client.p12<\/code><\/pre>\n\n\n\nExtract the Key and the Cert and put them together as pkcs12. You may now import them into firefox and access the dashboard through the api (don’t forget to enter the token).<\/p>\n\n\n\n
(Read everything about the dashboard here<\/strong>: https:\/\/github.com\/kubernetes\/dashboard)
You should have a fully working and manageable kubernetes cluster by now!<\/strong><\/p>\n\n\n\n
\n\n\n\nsome closing words and Resources<\/h5>\n\n\n\n
This post covers the manual setup of a kubernetes cluster, which is good for understanding the basics, for a more production ready setup you may want to take a look at common automation tools (puppet<\/a> and ansible):<\/p>\n\n\n\n- https:\/\/forge.puppet.com\/puppetlabs\/kubernetes<\/li>
- https:\/\/kubespray.io\/#\/<\/li><\/ul>\n\n\n\n
Also, because the topic is huge, you might want to consider reading a book about it:<\/p>\n\n\n\n
- https:\/\/www.hanser-fachbuch.de\/buch\/Kubernetes+in+Action\/9783446455108<\/li><\/ul>\n\n\n\n
It’s good, I’m reading (parts of) it at the moment, it uses minikube for examples a lot though.<\/p>\n","protected":false},"excerpt":{"rendered":"
What is kubernetes and why should I need it? Simply put, kubernetes is a tool for managing computing resources. It does this very efficently by abstracting your hardware into one (or more, if you like) big computing resource and therefore highly efficient use of your hardware with very little overhead unlike Virtual Machines for example … <\/p>\n