Kubernetes: From zero to hero

Outline

Intro: containers
Why tho?
Kubernetes resourses
Kubernetes architecture
Role Based Access Control
Network Policy
Revisiting Docker
PodSecurity Policy

Intro: containers

https://cloud.google.com/containers

Chroot jail


copy /bin/bash /jail/bash
mount -o bind /home /jail/home

Namespaces


$ unshare --fork --pid --mount-proc bash
$ ps auxf
USER PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root   1  0.0  0.0   8964  3924 pts/4    S    00:25   0:00 bash
root   8  0.0  0.0  10612  3304 pts/4    R+   00:25   0:00 ps auxf

🤔

Namespaces

Cgroup
IPC
Network
Mount
PID
User
UTS

http://ifeanyi.co/posts/linux-namespaces-part-1/

Docker 🐋


$ docker run -ti --rm ubuntu bash
root@03f784316baa:/# ps auxf
USER PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root   1  0.2  0.0   4112  3436 pts/0    Ss   21:40   0:00 bash
root   8  0.0  0.0   5884  2948 pts/0    R+   21:40   0:00 ps auxf

😎

❗

Any Docker container is just a process

Dockerfile


FROM ubuntu
RUN apt update && apt install nginx
CMD ["nginx", "-g", "daemon off;"]


$ docker build -t my-nginx .
$ docker run -p 80:80 my-nginx

How does it work?


$ ps axf -o user,pid,command
root     19616 /usr/bin/containerd
root     21721  \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/35a2b88eb0659e452be831115395352359e35669efa766931d350296aa154c6d -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root     21745      \_ nginx: master process nginx -g daemon off;
nobody   21788          \_ nginx: worker process
nobody   21789          \_ nginx: worker process
root     23571 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

But how is it secure?🤔

Chromium already used all these Linux feautures!
Seccomp - block unwanted syscalls
Apparmor - restrict Linux capabilities
SELinux - access control for files (== everything)

More layers == more attack surface

Kernel exploit
arbitary binary runC overwrite
docker cp RCE + escape
Mounted /var/run/docker.sock
dockerd pwn, it runs as root!
Disabling security mechanisms
- --privileged
- --cap-add=SYS_ADMIN
- --security-opt seccomp=unconfined

Recap

Containers are cool
Containers are versatile
Containers can be secure
Containers can be insecure

You need to run and scale 1000 containers

They all should be talking to each other via network

They should use network storage to run database on it

They have a load balancer

They should dynamically adjust nodes base on load

Kubernetes!☸

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.


$ kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
kube-a   Ready    master   13d   v1.18.6   10.166.0.8    <none>        Ubuntu 16.04.7 LTS   4.15.0-1080-gcp   docker://18.9.7
kube-b   Ready    <none>   13d   v1.18.6   10.166.0.7    <none>        Ubuntu 16.04.7 LTS   4.15.0-1080-gcp   docker://18.9.7

Pods


apiVersion: v1
kind: Pod
metadata:
    name: nginx
    namespace: default
spec:
    containers:
    - name: nginx
      image: nginx


$ kubectl apply -f nginx.yaml
pod/nginx created

https://kubernetes.io/docs/concepts/overview/components/

Kubernetes versions

Bare metal: bootstraped by kubeadm
Cloud managed: GKE, EKS, ...
OpenShift
...

Rule of thumb

No workloads or access on control-plane!

Example:

etcd on kubeadm < 1.16 accepts TLS certificates issued by apiserver CA (certs used for auth)

Exploitation: create a pod with hostNetwork: true

Every resource has a namespace it belongs to


apiVersion: v1
kind: Pod
metadata:
    name: nginx
    namespace: default
spec:
    containers:
    - name: nginx
        image: nginx

Namespaces only effects resourses in RBAC!

e.g. namespaces do not provide network isolation

ServiceAccount


apiVersion: v1
kind: ServiceAccount
metadata:
    name: remover

Allowes workloads using Kubernetes API

ServiceAccount token in pods


root@nginx:/# ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt  namespace  token
root@nginx:/var/run/secrets/kubernetes.io/serviceaccount# cat token
eyJhbGc...UNg


apiVersion: v1
kind: ServiceAccount
metadata:
    name: remover
automountServiceAccountToken: false
...

Role Based Access Control (RBAC)

ServiceAccount <---> RoleBinding <---> Role


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: remover
rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: remover
subjects:
- kind: ServiceAccount
  name: remover
roleRef:
    kind: Role
    name: remover
    apiGroup: rbac.authorization.k8s.io

Sum up

Find a token for ServiceAccount
Enumerate permissions


$ kubectl auth can-i --list
Resources   Non-Resource URLs   Resource Names   Verbs
*.*         []                  []               [*]
            [*]                 []               [*]
            [/api/*]            []               [get]
            [/api]              []               [get]
            [/apis/*]           []               [get]
            [/apis]             []               [get]

Try to escalate! Repeat...

Redhat 2021 Kubernetes security report

Revisiting Docker

hostPath - mounts directory from node
privileged
HostNetwork
allowPrivilegeEscalation

Hvatit eto terpet'!

PodSecurityPolicies

Enables us to set restrictive policy on security context


apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restrictive
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'runtime/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'
spec:
  privileged: false
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  requiredDropCapabilities:
  - ALL
  hostNetwork: false
  hostPID: false
  hostIPC: false
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - 'configMap'
  - 'downwardAPI'
  - 'emptyDir'
  - 'persistentVolumeClaim'
  - 'secret'
  - 'projected'
  allowedCapabilities:
  - 'NET_RAW'


kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: restrictive-psp
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames:
  - "restrictive"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: restrictive-psp
subjects:
- kind: ServiceAccount
  name: remover
roleRef:
    kind: Role
    name: restrictive-psp
    apiGroup: rbac.authorization.k8s.io

Network Policies


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
    name: default-deny-egress
    namespace: app
spec:
    podSelector:
    matchLabels: {}
    policyTypes:
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
    name: allow-dns-access
    namespace: app
spec:
    podSelector:
    matchLabels: {}
    policyTypes:
    - Egress
    egress:
    - to:
      - namespaceSelector:
          matchLabels:
              name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

editor.cilium.io

Network plugin support required.

You won't get a warning if it's not working!

Cloud things


$ curl "http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/token"
{"access_token":"ya29.c.Ko...QtkM","expires_in":2837,"token_type":"Bearer"}
$ curl "https://cloudresourcemanager.googleapis.com/v1/projects?alt=json" -H "Authorization: Bearer ya29..."
...
$ curl -H "Authorization: Bearer ya29..." "http://kube-apiserver:6443/api/v1"
{...}

CoreDNS

*name*.*namespace*.svc.cluster.local

Kubernetes dashboard

Combine together

curl http://kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local:443/api/v1/ -k

Cool links

Future stack

Not today

Networking (CNI)
Supply chain attacks
Custom container runtimes (CRI)
- cri-o
- firecracker
- gvisor
Open Policy Agent
Service mesh
- Istio
- Linkerd

Alt + click to zoom

Kubernetes: From zero to hero

Outline

Intro: containers

Chroot jail

Namespaces

Namespaces

Docker 🐋

❗

Dockerfile

How does it work?

Links

But how is it secure?🤔

More layers == more attack surface

Recap

Kubernetes!☸

Pods

Kubernetes versions

Rule of thumb

No workloads or access on control-plane!

Every resource has a namespace it belongs to

ServiceAccount

ServiceAccount token in pods

Role Based Access Control (RBAC)

Sum up

Revisiting Docker

PodSecurityPolicies

Network Policies

Cloud things

CoreDNS

Kubernetes dashboard

Combine together

Cool links

Future stack

Not today