Setting Up a Kubernetes Cluster on Ubuntu 16.04 via kubeadm

Post Reply
Shane
Posts: 4
Joined: Sat Apr 28, 2018 8:52 am

Setting Up a Kubernetes Cluster on Ubuntu 16.04 via kubeadm

Post by Shane » Thu May 24, 2018 4:03 pm

Anybody here use Kubernetes?

I have just redone the software stack on my homelab cluster from scratch. Here's a photo of the 5 node cluster:

5-node-nano-itx-cluster-2.jpg
5-node-nano-itx-cluster-2.jpg (94.62 KiB) Viewed 272 times

I am still using Ubuntu 16.04 since the Docker versions that are currently available for 18.04 are not yet supported by Kubernetes. These are the lab notes that I compiled while installing Kubernetes v1.10.3 via kubeadm. I chose Calico for pod networking.

All Nodes
In this section we’ll prepare the master and worker nodes for Kubernetes. We’ll start from a newly minted Ubuntu 16.04 on each node.

Install Docker 17.03:

Code: Select all

apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')
Install kubeadm, kubelet and kubectl:

Code: Select all

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
Turn swap off:

Code: Select all

swapoff -a
Master Node
Make sure that the cgroup driver used by kubelet is the same as the one used by Docker. To check whether the Docker cgroup driver matches the kubelet config:

Code: Select all

docker info | grep -i cgroup
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
If the Docker cgroup driver and the kubelet config don’t match, update the latter. The flag we need to change is –cgroup-driver. If it’s already set, we can update the configuration like so:

Code: Select all

sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Otherwise, open the systemd file and add the flag to an existing environment line. Then restart the kubelet:

Code: Select all

systemctl daemon-reload
systemctl restart kubelet
Initialise Kubernetes Master
Initialise the master node by running kubeadm init. We need to specify the pod network CIDR for network policy to work correctly when we install Calico in a later step.

Code: Select all

kubeadm init --pod-network-cidr=10.0.0.0/16
To be able to use kubectl as non-root user on master:

Code: Select all

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Install Calico for networking:

Code: Select all

kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
Once Calico has been installed, confirm that it is working by checking that the kube-dns pod is running before joining the worker nodes.

Code: Select all

shane@master1:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE
kube-system   calico-node-2zrrz                 2/2       Running   0          8m
kube-system   etcd-master1                      1/1       Running   0          11m
kube-system   kube-apiserver-master1            1/1       Running   0          11m
kube-system   kube-controller-manager-master1   1/1       Running   0          11m
kube-system   kube-dns-86f4d74b45-bpsgs         3/3       Running   0          12m
kube-system   kube-proxy-pkfjx                  1/1       Running   0          12m
kube-system   kube-scheduler-master1            1/1       Running   0 
Worker Nodes
Next we’ll join the worker nodes to our new Kubernetes cluster. Run the command that was output by kubeadm init on each of the nodes:

Code: Select all

kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
We should see nodes joining the cluster shortly:

Code: Select all

shane@master1:~$ kubectl get nodes
NAME      STATUS     ROLES     AGE       VERSION
master1   Ready      master    22m       v1.10.3
minion1   Ready      <none>    6m        v1.10.3
minion2   Ready      <none>    4m        v1.10.3
minion3   NotReady   <none>    4m        v1.10.3
minion4   NotReady   <none>    4m        v1.10.3
Configure Access from Workstation
To control the cluster remotely from our workstation, we grab the contents of /etc/kubernetes/admin.conf from the master node and merge it into our local ~/.kube/config configuration file.

Shane
Posts: 4
Joined: Sat Apr 28, 2018 8:52 am

Re: Setting Up a Kubernetes Cluster on Ubuntu 16.04 via kubeadm

Post by Shane » Fri May 25, 2018 5:09 am

If you are not familiar with Kubernetes, look up Kelsey Hightower videos. He has some great presentations about the technology. Since this is the sysadmin sub forum, this one is particularly relevant:

https://www.youtube.com/watch?v=HlAXp0-M6SY

appinv
Posts: 27
Joined: Mon Apr 30, 2018 6:11 pm

Re: Setting Up a Kubernetes Cluster on Ubuntu 16.04 via kubeadm

Post by appinv » Sun May 27, 2018 4:36 pm

a proper in-forum tuto, well done !

User avatar
PC_THE_GREAT
Core team
Posts: 61
Joined: Mon Apr 16, 2018 5:28 pm

Re: Setting Up a Kubernetes Cluster on Ubuntu 16.04 via kubeadm

Post by PC_THE_GREAT » Wed May 30, 2018 1:06 pm

great article Shane, thanks for sharing!

Shane
Posts: 4
Joined: Sat Apr 28, 2018 8:52 am

Re: Setting Up a Kubernetes Cluster on Ubuntu 16.04 via kubeadm

Post by Shane » Mon Jun 04, 2018 3:07 pm

Thanks guys 👍

I'm documenting the rest of my homelab setup in this repo. Other parts of the stack:
  • Ceph for storage
  • Istio service mesh
  • Concourse CI

Post Reply