K8S) 1주차 과제

zosys 2026. 1. 8. 00:41

2026. 1. 8. 00:41

1. Bootstrap Kubernetes the hard way

00. Kind K8s 설치

WSL2를 이용해서 진행(기설치 된kind를 통해서 실습을 진행한다.)

  (⎈|N/A:N/A) zosys@4:~$ kind version
  kind v0.30.0 go1.24.6 linux/amd64
  (⎈|N/A:N/A) zosys@4:~$ helm version
  version.BuildInfo{Version:"v3.19.0", GitCommit:"3d8990f0836691f0229297773f3524598f46bda6", GitTreeState:"clean", GoVersion:"go1.24.7"}

1주차 실습을 위한 kind k8s 배포(WSL2)

  (⎈|N/A:N/A) zosys@4:~$ kind create cluster --name myk8s --image kindest/node:v1.32.8 --config - <<EOF
  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  nodes:
  - role: control-plane
    extraPortMappings:
    - containerPort: 30000
      hostPort: 30000
    - containerPort: 30001
      hostPort: 30001
  - role: worker
  EOF
  Creating cluster "myk8s" ...
   ✓ Ensuring node image (kindest/node:v1.32.8) 🖼
   ✓ Preparing nodes 📦 📦
   ✓ Writing configuration 📜
   ✓ Starting control-plane 🕹️
   ✓ Installing CNI 🔌
   ✓ Installing StorageClass 💾
   ✓ Joining worker nodes 🚜
  Set kubectl context to "kind-myk8s"
  You can now use your cluster with:

  (⎈|kind-myk8s:N/A) zosys@4:~$ kind get nodes --name myk8s
  myk8s-control-plane
  myk8s-worker

  (⎈|kind-myk8s:default) zosys@4:~$ kubectl get node -o wide
  NAME                  STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                     CONTAINER-RUNTIME
  myk8s-control-plane   Ready    control-plane   2m30s   v1.32.8   172.18.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.6.87.2-microsoft-standard-WSL2   containerd://2.1.3
  myk8s-worker          Ready    <none>          2m14s   v1.32.8   172.18.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.6.87.2-microsoft-standard-WSL2   containerd://2.1.3
  (⎈|kind-myk8s:default) zosys@4:~$ docker exec -it myk8s-control-plane ss -tnlp
  State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port Process
  LISTEN  0       4096         127.0.0.1:45859        0.0.0.0:*     users:(("containerd",pid=106,fd=11))
  LISTEN  0       4096        172.18.0.3:2379         0.0.0.0:*     users:(("etcd",pid=645,fd=9))
  LISTEN  0       4096        172.18.0.3:2380         0.0.0.0:*     users:(("etcd",pid=645,fd=7))
  LISTEN  0       4096        127.0.0.11:46007        0.0.0.0:*
  LISTEN  0       4096         127.0.0.1:10248        0.0.0.0:*     users:(("kubelet",pid=706,fd=20))
  LISTEN  0       4096         127.0.0.1:10249        0.0.0.0:*     users:(("kube-proxy",pid=899,fd=17))
  LISTEN  0       4096         127.0.0.1:10259        0.0.0.0:*     users:(("kube-scheduler",pid=514,fd=3))
  LISTEN  0       4096         127.0.0.1:10257        0.0.0.0:*     users:(("kube-controller",pid=554,fd=3))
  LISTEN  0       4096         127.0.0.1:2379         0.0.0.0:*     users:(("etcd",pid=645,fd=8))
  LISTEN  0       4096         127.0.0.1:2381         0.0.0.0:*     users:(("etcd",pid=645,fd=16))
  LISTEN  0       4096                 *:6443               *:*     users:(("kube-apiserver",pid=566,fd=3))
  LISTEN  0       4096                 *:10256              *:*     users:(("kube-proxy",pid=899,fd=16))
  LISTEN  0       4096                 *:10250              *:*     users:(("kubelet",pid=706,fd=22))

01. Pre requisites

실습용 Vagrant배포의 경우 리소스 문제로 다른 PC에서 진행한다.


  PS C:\Users\bom\Desktop\스터디\onpremisk8s> dir

      디렉터리: C:\Users\bom\Desktop\스터디\onpremisk8s

  Mode                 LastWriteTime         Length Name
  ----                 -------------         ------ ----
  -a----      2026-01-05  오후 11:09           1172 init_cfg.sh
  -a----      2026-01-05  오후 11:08           3234 Vagrantfile

  PS C:\Users\bom\Desktop\스터디\onpremisk8s> vagrant.exe up
  Bringing machine 'jumpbox' up with 'virtualbox' provider...
  Bringing machine 'server' up with 'virtualbox' provider...
  Bringing machine 'node-0' up with 'virtualbox' provider...
  Bringing machine 'node-1' up with 'virtualbox' provider...
  ==> jumpbox: Box 'bento/debian-12' could not be found. Attempting to find and install...
      jumpbox: Box Provider: virtualbox
      jumpbox: Box Version: 202510.26.0
  ==> jumpbox: Loading metadata for box 'bento/debian-12'
      jumpbox: URL: https://vagrantcloud.com/api/v2/vagrant/bento/debian-12
  ==> jumpbox: Adding box 'bento/debian-12' (v202510.26.0) for provider: virtualbox (amd64)
      jumpbox: Downloading: https://vagrantcloud.com/bento/boxes/debian-12/versions/202510.26.0/providers/virtualbox/amd64/vagrant.box

  ############################중략############################################    

  #배포 가상머신 확인   
     PS C:\Users\bom\Desktop\스터디\onpremisk8s> vagrant status
  Current machine states:

  jumpbox                   running (virtualbox)
  server                    running (virtualbox)
  node-0                    running (virtualbox)
  node-1                    running (virtualbox)

jumpbox 가상머신 접속 : vagrant ssh jumpbox

  PS C:\Users\bom\Desktop\스터디\onpremisk8s> vagrant ssh jumpbox
  Linux jumpbox 6.1.0-40-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.153-1 (2025-09-20) x86_64

  This system is built by the Bento project by Chef Software
  More information can be found at https://github.com/chef/bento

  Use of this system is acceptance of the OS vendor EULA and License Agreements.

  The programs included with the Debian GNU/Linux system are free software;
  the exact distribution terms for each program are described in the
  individual files in /usr/share/doc/*/copyright.

  Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
  permitted by applicable law.
  root@jumpbox:~# whoami
  root
  root@jumpbox:~# pwd
  /root
  root@jumpbox:~#

02 - Set up The jumpbox

vagrant ssh jumpbox

  root@jumpbox:~# whoami
  root
  root@jumpbox:~# pwd
  /root
  root@jumpbox:~# cat /home/vagrant/.bashrc | tail -n 1
  sudo su -
  root@jumpbox:~# git clone --depth 1 https://github.com/kelseyhightower/kubernetes-the-hard-way.git
  Cloning into 'kubernetes-the-hard-way'...
  remote: Enumerating objects: 41, done.
  remote: Counting objects: 100% (41/41), done.
  remote: Compressing objects: 100% (40/40), done.
  remote: Total 41 (delta 3), reused 14 (delta 1), pack-reused 0 (from 0)
  Receiving objects: 100% (41/41), 29.27 KiB | 14.63 MiB/s, done.
  Resolving deltas: 100% (3/3), done.
  root@jumpbox:~# cd kubernetes-the-hard-way/
  root@jumpbox:~/kubernetes-the-hard-way# tree
  .
  ├── ca.conf
  ├── configs
  │   ├── 10-bridge.conf
  │   ├── 99-loopback.conf
  │   ├── containerd-config.toml
  │   ├── encryption-config.yaml
  │   ├── kube-apiserver-to-kubelet.yaml
  │   ├── kubelet-config.yaml
  │   ├── kube-proxy-config.yaml
  │   └── kube-scheduler.yaml
  ├── CONTRIBUTING.md
  ├── COPYRIGHT.md
  ├── docs
  │   ├── 01-prerequisites.md
  │   ├── 02-jumpbox.md
  │   ├── 03-compute-resources.md
  │   ├── 04-certificate-authority.md
  │   ├── 05-kubernetes-configuration-files.md
  │   ├── 06-data-encryption-keys.md
  │   ├── 07-bootstrapping-etcd.md
  │   ├── 08-bootstrapping-kubernetes-controllers.md
  │   ├── 09-bootstrapping-kubernetes-workers.md
  │   ├── 10-configuring-kubectl.md
  │   ├── 11-pod-network-routes.md
  │   ├── 12-smoke-test.md
  │   └── 13-cleanup.md
  ├── downloads-amd64.txt
  ├── downloads-arm64.txt
  ├── LICENSE
  ├── README.md
  └── units
      ├── containerd.service
      ├── etcd.service
      ├── kube-apiserver.service
      ├── kube-controller-manager.service
      ├── kubelet.service
      ├── kube-proxy.service
      └── kube-scheduler.service

  4 directories, 35 files

  #CPU 아키텍쳐 확인

  root@jumpbox:~/kubernetes-the-hard-way# dpkg --print-architecture
  amd64
  root@jumpbox:~/kubernetes-the-hard-way# ls -l downloads-*
  -rw-r--r-- 1 root root 839 Jan  5 23:25 downloads-amd64.txt
  -rw-r--r-- 1 root root 839 Jan  5 23:25 downloads-arm64.txt

  root@jumpbox:~/kubernetes-the-hard-way# cat downloads-$(dpkg --print-architecture).txt
  https://dl.k8s.io/v1.32.3/bin/linux/amd64/kubectl
  https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-apiserver
  https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-controller-manager
  https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-scheduler
  https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-proxy
  https://dl.k8s.io/v1.32.3/bin/linux/amd64/kubelet
  https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.32.0/crictl-v1.32.0-linux-amd64.tar.gz
  https://github.com/opencontainers/runc/releases/download/v1.3.0-rc.1/runc.amd64
  https://github.com/containernetworking/plugins/releases/download/v1.6.2/cni-plugins-linux-amd64-v1.6.2.tgz
  https://github.com/containerd/containerd/releases/download/v2.1.0-beta.0/containerd-2.1.0-beta.0-linux-amd64.tar.gz
  https://github.com/etcd-io/etcd/releases/download/v3.6.0-rc.3/etcd-v3.6.0-rc.3-linux-amd64.tar.gz
  root@jumpbox:~/kubernetes-the-hard-way# wget -q --show-progress \
    --https-only \
    --timestamping \
    -P downloads \
    -i downloads-$(dpkg --print-architecture).txt
  kubectl                       100%[=================================================>]  54.67M  64.4MB/s    in 0.8s
  kube-apiserver                100%[=================================================>]  88.94M  28.8MB/s    in 3.1s
  kube-controller-manager       100%[=================================================>]  82.00M  17.7MB/s    in 4.6s
  kube-scheduler                100%[=================================================>]  62.79M  22.0MB/s    in 2.9s
  kube-proxy                    100%[=================================================>]  63.75M  25.1MB/s    in 2.5s
  kubelet                       100%[=================================================>]  73.82M  12.8MB/s    in 6.5s
  crictl-v1.32.0-linux-amd64.ta 100%[=================================================>]  18.21M  55.1MB/s    in 0.3s
  runc.amd64                    100%[=================================================>]  11.30M  66.3MB/s    in 0.2s
  cni-plugins-linux-amd64-v1.6. 100%[=================================================>]  50.35M  36.3MB/s    in 1.4s
  containerd-2.1.0-beta.0-linux 100%[=================================================>]  37.01M  56.7MB/s    in 0.7s
  etcd-v3.6.0-rc.3-linux-amd64. 100%[=================================================>]  22.48M  47.2MB/s    in 0.5s

  root@jumpbox:~/kubernetes-the-hard-way# ls -oh downloads
  total 566M
  -rw-r--r-- 1 root 51M Jan  7  2025 cni-plugins-linux-amd64-v1.6.2.tgz
  -rw-r--r-- 1 root 38M Mar 18  2025 containerd-2.1.0-beta.0-linux-amd64.tar.gz
  -rw-r--r-- 1 root 19M Dec  9  2024 crictl-v1.32.0-linux-amd64.tar.gz
  -rw-r--r-- 1 root 23M Mar 28  2025 etcd-v3.6.0-rc.3-linux-amd64.tar.gz
  -rw-r--r-- 1 root 89M Mar 12  2025 kube-apiserver
  -rw-r--r-- 1 root 83M Mar 12  2025 kube-controller-manager
  -rw-r--r-- 1 root 55M Mar 12  2025 kubectl
  -rw-r--r-- 1 root 74M Mar 12  2025 kubelet
  -rw-r--r-- 1 root 64M Mar 12  2025 kube-proxy
  -rw-r--r-- 1 root 63M Mar 12  2025 kube-scheduler
  -rw-r--r-- 1 root 12M Mar  4  2025 runc.amd64

  root@jumpbox:~/kubernetes-the-hard-way# mkdir -p downloads/{client,cni-plugins,controller,worker}
  root@jumpbox:~/kubernetes-the-hard-way# tree -d downloads
  downloads
  ├── client
  ├── cni-plugins
  ├── controller
  └── worker

  5 directories

  #압축풀기 
  root@jumpbox:~/kubernetes-the-hard-way# tar -xvf downloads/crictl-v1.32.0-linux-${ARCH}.tar.gz \
    -C downloads/worker/ && tree -ug downloads
  crictl
  [root     root    ]  downloads
  ├── [root     root    ]  client
  ├── [root     root    ]  cni-plugins
  ├── [root     root    ]  cni-plugins-linux-amd64-v1.6.2.tgz
  ├── [root     root    ]  containerd-2.1.0-beta.0-linux-amd64.tar.gz
  ├── [root     root    ]  controller
  ├── [root     root    ]  crictl-v1.32.0-linux-amd64.tar.gz
  ├── [root     root    ]  etcd-v3.6.0-rc.3-linux-amd64.tar.gz
  ├── [root     root    ]  kube-apiserver
  ├── [root     root    ]  kube-controller-manager
  ├── [root     root    ]  kubectl
  ├── [root     root    ]  kubelet
  ├── [root     root    ]  kube-proxy
  ├── [root     root    ]  kube-scheduler
  ├── [root     root    ]  runc.amd64
  └── [root     root    ]  worker
      └── [1001     127     ]  crictl

  5 directories, 12 files

  ################그외 중략 ########################

  #압축해제 후 확인진행
  root@jumpbox:~/kubernetes-the-hard-way# tree downloads/worker/
  downloads/worker/
  ├── containerd
  ├── containerd-shim-runc-v2
  ├── containerd-stress
  ├── crictl
  └── ctr

  1 directory, 5 files
  root@jumpbox:~/kubernetes-the-hard-way# tree downloads/cni-plugins
  downloads/cni-plugins
  ├── bandwidth
  ├── bridge
  ├── dhcp
  ├── dummy
  ├── firewall
  ├── host-device
  ├── host-local
  ├── ipvlan
  ├── LICENSE
  ├── loopback
  ├── macvlan
  ├── portmap
  ├── ptp
  ├── README.md
  ├── sbr
  ├── static
  ├── tap
  ├── tuning
  ├── vlan
  └── vrf

  1 directory, 20 files
  #파일 이동 및 확인
  root@jumpbox:~/kubernetes-the-hard-way# mv downloads/{etcdctl,kubectl} downloads/client/
  mv downloads/{etcd,kube-apiserver,kube-controller-manager,kube-scheduler} downloads/controller/
  mv downloads/{kubelet,kube-proxy} downloads/worker/
  mv downloads/runc.${ARCH} downloads/worker/runc

  root@jumpbox:~/kubernetes-the-hard-way# tree downloads/client/
  tree downloads/controller/
  tree downloads/worker/
  downloads/client/
  ├── etcdctl
  └── kubectl

  1 directory, 2 files
  downloads/controller/
  ├── etcd
  ├── kube-apiserver
  ├── kube-controller-manager
  └── kube-scheduler

  1 directory, 4 files
  downloads/worker/
  ├── containerd
  ├── containerd-shim-runc-v2
  ├── containerd-stress
  ├── crictl
  ├── ctr
  ├── kubelet
  ├── kube-proxy
  └── runc

  1 directory, 8 files
  #그외 작업 진행후 최종 kubtectl 확인
  root@jumpbox:~/kubernetes-the-hard-way# kubectl version --client
  Client Version: v1.32.3
  Kustomize Version: v5.5.0

03 - Provisioning Compute Resources

SSH 접속 환경 설정

  root@jumpbox:~/kubernetes-the-hard-way# cat <<EOF > machines.txt
  192.168.10.100 server.kubernetes.local server
  192.168.10.101 node-0.kubernetes.local node-0 10.200.0.0/24
  192.168.10.102 node-1.kubernetes.local node-1 10.200.1.0/24
  EOF
  root@jumpbox:~/kubernetes-the-hard-way# cat machines.txt
  192.168.10.100 server.kubernetes.local server
  192.168.10.101 node-0.kubernetes.local node-0 10.200.0.0/24
  192.168.10.102 node-1.kubernetes.local node-1 10.200.1.0/24

  root@jumpbox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
    echo "${IP} ${FQDN} ${HOST} ${SUBNET}"
  done < machines.txt
  192.168.10.100 server.kubernetes.local server
  192.168.10.101 node-0.kubernetes.local node-0 10.200.0.0/24
  192.168.10.102 node-1.kubernetes.local node-1 10.200.1.0/24

  root@jumpbox:~/kubernetes-the-hard-way# ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa
  Generating public/private rsa key pair.
  Your identification has been saved in /root/.ssh/id_rsa
  Your public key has been saved in /root/.ssh/id_rsa.pub
  The key fingerprint is:
  SHA256:------------------------------------ root@jumpbox
  The key's randomart image is:
  +---[RSA 3072]----+
  |      .+o.B=*++++|
  |     oooEo Xo=oB |
  |      ooo =.+o*..|
  |     . o..o . .=.|
  |      ..So .   .o|
  |      . . .      |
  |       + o       |
  |      + +        |
  |     . .         |
  +----[SHA256]-----+

  root@jumpbox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
    sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@${IP}
  done < machines.txt
  /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
  /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
  /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

  Number of key(s) added: 1

  Now try logging into the machine, with:   "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.10.100'"
  and check to make sure that only the key(s) you wanted were added.

  /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
  /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
  /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

  Number of key(s) added: 1

  Now try logging into the machine, with:   "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.10.101'"
  and check to make sure that only the key(s) you wanted were added.

  /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
  /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
  /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

  Number of key(s) added: 1

  Now try logging into the machine, with:   "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.10.102'"
  and check to make sure that only the key(s) you wanted were added.

  root@jumpbox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
    ssh -n root@${IP} cat /root/.ssh/authorized_keys
  done < machines.txt
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUoQb+fRLPJN03IsqZiCa0nxOuMA7Mo5VpVml+8XsCPa0JCoGxlHn5C8+xmPbp5qg4VhNVRVBIyt3/ipgURJbhgxw/Yo+3Tcq0C5BcmxzrDZGfK8mUAGXPWrHPDtECgZP4+kRyFcGMOwJCJvjTlbFXc/cDypp9RpDtbAXsBR/P+M9gYHtcAI2VRJMjHS0yTvFzf01WwoCYBWg5QG7NgIKVS2qMS75kAdnveBT+nu5E5TN2TmCi5vaD64LC1uuhg3NHDdUw14U0wAENNphQleERk0Y0jvcnFsf5XT6+KNYCfZjgkBPvBCcJRq2szo8Df740lGVoe8vWttAg79DkCB/QZV3UT+k+UN89gskAFtnWKv3MnSsAsjcevxSMsky3eEbZJ5lrN+NC32bTUkovb0paQUhDUf/gHsPQtwD/8FIolNWXopB2cZkwFZIkqs8JIV1N+C5fDczVsioJRgJmJQ53P/lliQXn82hlfe1/ZB+ZxO4mTHHHWC765b8dV0CVn2k= root@jumpbox
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUoQb+fRLPJN03IsqZiCa0nxOuMA7Mo5VpVml+8XsCPa0JCoGxlHn5C8+xmPbp5qg4VhNVRVBIyt3/ipgURJbhgxw/Yo+3Tcq0C5BcmxzrDZGfK8mUAGXPWrHPDtECgZP4+kRyFcGMOwJCJvjTlbFXc/cDypp9RpDtbAXsBR/P+M9gYHtcAI2VRJMjHS0yTvFzf01WwoCYBWg5QG7NgIKVS2qMS75kAdnveBT+nu5E5TN2TmCi5vaD64LC1uuhg3NHDdUw14U0wAENNphQleERk0Y0jvcnFsf5XT6+KNYCfZjgkBPvBCcJRq2szo8Df740lGVoe8vWttAg79DkCB/QZV3UT+k+UN89gskAFtnWKv3MnSsAsjcevxSMsky3eEbZJ5lrN+NC32bTUkovb0paQUhDUf/gHsPQtwD/8FIolNWXopB2cZkwFZIkqs8JIV1N+C5fDczVsioJRgJmJQ53P/lliQXn82hlfe1/ZB+ZxO4mTHHHWC765b8dV0CVn2k= root@jumpbox
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUoQb+fRLPJN03IsqZiCa0nxOuMA7Mo5VpVml+8XsCPa0JCoGxlHn5C8+xmPbp5qg4VhNVRVBIyt3/ipgURJbhgxw/Yo+3Tcq0C5BcmxzrDZGfK8mUAGXPWrHPDtECgZP4+kRyFcGMOwJCJvjTlbFXc/cDypp9RpDtbAXsBR/P+M9gYHtcAI2VRJMjHS0yTvFzf01WwoCYBWg5QG7NgIKVS2qMS75kAdnveBT+nu5E5TN2TmCi5vaD64LC1uuhg3NHDdUw14U0wAENNphQleERk0Y0jvcnFsf5XT6+KNYCfZjgkBPvBCcJRq2szo8Df740lGVoe8vWttAg79DkCB/QZV3UT+k+UN89gskAFtnWKv3MnSsAsjcevxSMsky3eEbZJ5lrN+NC32bTUkovb0paQUhDUf/gHsPQtwD/8FIolNWXopB2cZkwFZIkqs8JIV1N+C5fDczVsioJRgJmJQ53P/lliQXn82hlfe1/ZB+ZxO4mTHHHWC765b8dV0CVn2k= root@jumpbox
  root@jumpbox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
    ssh -n root@${IP} hostname
  done < machines.txt
  server
  node-0
  node-1

  #확인진행
  root@jumpbox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
    ssh -n root@${IP} hostname --fqdn
  done < machines.txt
  server.kubernetes.local
  node-0.kubernetes.local
  node-1.kubernetes.local

  root@jumpbox:~/kubernetes-the-hard-way# cat /etc/hosts
  while read IP FQDN HOST SUBNET; do
    sshpass -p 'qwe123' ssh -n -o StrictHostKeyChecking=no root@${HOST} hostname
  done < machines.txt
  127.0.0.1       localhost

  # The following lines are desirable for IPv6 capable hosts
  ::1     localhost ip6-localhost ip6-loopback
  ff02::1 ip6-allnodes
  ff02::2 ip6-allrouters
  192.168.10.10  jumpbox
  192.168.10.100 server.kubernetes.local server
  192.168.10.101 node-0.kubernetes.local node-0
  192.168.10.102 node-1.kubernetes.local node-1
  Warning: Permanently added 'server' (ED25519) to the list of known hosts.
  server
  Warning: Permanently added 'node-0' (ED25519) to the list of known hosts.
  node-0
  Warning: Permanently added 'node-1' (ED25519) to the list of known hosts.
  node-1

04 - Provisioning a CA and Generating TLS Certificates

상호 TLS 인증(MTLS)를 사용하여 통신을 진행한다. 즉 통신하는 컴포넌트의 인증서는 전부 생성이 필요하며, 일반적인 k8s의 경우 kubeadm이 생성해준다. 아래는 생성해야하는 목록이다.

|  | 개인키 | CSR | 인증서 | 참고 정보 |
| --- | --- | --- | --- | --- |
| Root CA | ca.key | X | ca.crt |  |
| admin | admin.key | admin.csr | admin.crt | CN = admin, O = system:masters |
| node-0 | node-0.key | node-0.csr | node-0.crt | CN = system:node:node-0, O = system:nodes |
| node-1 | node-1.key | node-1.csr | node-1.crt | CN = system:node:node-1, O = system:nodes |
| kube-proxy | kube-proxy.key | kube-proxy.csr | kube-proxy.crt | CN = system:kube-proxy, O = system:node-proxier |
| kube-scheduler | kube-scheduler.key | kube-scheduler | kube-scheduler.crt | CN = system:kube-scheduler, O = system:kube-scheduler |
| kube-controller-manager | kube-controller-manager.key | kube-controller-manager.csr | kube-controller-manager.crt | CN = system:kube-controller-manager, O = system:kube-controller-manager |
| kube-api-server | kube-api-server.key | kube-api-server.csr | kube-api-server.crt | CN = kubernetes, SAN: IP(127.0.0.1, **10.32.0.1**), DNS(kubernetes,..) |
| service-accounts | service-accounts.key | service-accounts.csr | service-accounts.crt | CN = service-accounts |
- kind환경의 인증서 확인

    ```bash
    (⎈|kind-myk8s:default) zosys@4:~$ docker exec -i myk8s-control-plane kubeadm certs check-expiration
    [check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
    [check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.

    CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
    admin.conf                 Jan 05, 2027 13:52 UTC   364d            ca                      no
    apiserver                  Jan 05, 2027 13:52 UTC   364d            ca                      no
    apiserver-etcd-client      Jan 05, 2027 13:52 UTC   364d            etcd-ca                 no
    apiserver-kubelet-client   Jan 05, 2027 13:52 UTC   364d            ca                      no
    controller-manager.conf    Jan 05, 2027 13:52 UTC   364d            ca                      no
    etcd-healthcheck-client    Jan 05, 2027 13:52 UTC   364d            etcd-ca                 no
    etcd-peer                  Jan 05, 2027 13:52 UTC   364d            etcd-ca                 no
    etcd-server                Jan 05, 2027 13:52 UTC   364d            etcd-ca                 no
    front-proxy-client         Jan 05, 2027 13:52 UTC   364d            front-proxy-ca          no
    scheduler.conf             Jan 05, 2027 13:52 UTC   364d            ca                      no
    super-admin.conf           Jan 05, 2027 13:52 UTC   364d            ca                      no

    CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
    ca                      Jan 03, 2036 13:52 UTC   9y              no
    etcd-ca                 Jan 03, 2036 13:52 UTC   9y              no
    front-proxy-ca          Jan 03, 2036 13:52 UTC   9y              no

    ```

CA 생성

  root@jumpbox:~/kubernetes-the-hard-way# openssl genrsa -out ca.key 4096
  root@jumpbox:~/kubernetes-the-hard-way# ls -l ca.key
  -rw------- 1 root root 3272 Jan  6 00:05 ca.key
  root@jumpbox:~/kubernetes-the-hard-way# openssl rsa -in ca.key -text -noout
  Private-Key: (4096 bit, 2 primes)
  modulus:
      00:bb:3b:8f:cf:85:b5:3e:48:2e:9a:ff:ce:c7:99:
      5b:3a:de:7b:28:47:1e:92:a5:a0:9b:b5:d1:fd:9e:
      10:af:49:b1:6b:12:2d:cb:78:cb:e0:a4:c5:9d:d4:
      b8:52:60:46:38:37:bb:f8:c7:a9:1e:d7:d5:55:82:
      30:8c:80:ae:66:8d:83:25:18:2a:21:d4:66:8c:db:
      3c:c2:4c:d0:e0:15:a4:b2:d0:1b:9d:ae:9d:9e:bd:
      2b:57:f1:b6:b8:f0:ad:9b:06:90:43:7e:8c:58:5d:

      #################중략#########################

  root@jumpbox:~/kubernetes-the-hard-way# openssl req -x509 -new -sha512 -noenc \
    -key ca.key -days 3653 \
    -config ca.conf \
    -out ca.crt
  root@jumpbox:~/kubernetes-the-hard-way# ls -l ca.crt
  -rw-r--r-- 1 root root 1899 Jan  6 00:06 ca.crt

  #인증서 전체내용 확인
  root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in ca.crt -text -noout | more
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              21:a5:b4:64:7c:a3:3d:a7:a4:3f:40:14:05:e9:25:b9:21:a9:7e:be
          Signature Algorithm: sha512WithRSAEncryption
          Issuer: C = US, ST = Washington, L = Seattle, CN = CA
          Validity
              Not Before: Jan  5 15:06:27 2026 GMT
              Not After : Jan  6 15:06:27 2036 GMT
          Subject: C = US, ST = Washington, L = Seattle, CN = CA
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (4096 bit)
  ############################중략 #################################

참고) kind환경의 ca.crt 확인

(⎈|kind-myk8s:default) zosys@4:~$ docker exec -i myk8s-control-plane cat /etc/kubernetes/pki/ca.crt | openssl x509 -text -noout
Certificate:
  Data:
      Version: 3 (0x2)
      Serial Number: 8519439439972151590 (0x763b210860fe3126)
      Signature Algorithm: sha256WithRSAEncryption
      Issuer: CN = kubernetes
      Validity
          Not Before: Jan  5 13:47:29 2026 GMT
          Not After : Jan  3 13:52:29 2036 GMT
      Subject: CN = kubernetes
      Subject Public Key Info:
          Public Key Algorithm: rsaEncryption
              Public-Key: (2048 bit)
              Modulus:
                  00:d4:a5:b1:57:06:bc:c7:ff:85:ba:d6:44:74:8c:
                  0a:a4:3e:3c:c7:e7:15:09:f9:31:f1:a0:8e:92:d0:
                  90:f8:19:52:98:8e:29:92:17:37:8d:9d:fe:10:fa:
                  d7:0f:1e:41:89:53:62:18:d9:b5:c4:b7:43:50:e6:
                  5c:fd:2c:87:46:a7:54:b0:ea:ea:6d:f6:d8:93:cc:
                  1f:2f:7a:18:28:6e:17:e7:e2:e8:3b:26:c8:ae:3d:

### 중략#####

admin인증서 생성

  root@jumpbox:~/kubernetes-the-hard-way# openssl genrsa -out admin.key 4096
  root@jumpbox:~/kubernetes-the-hard-way# openssl req -new -key admin.key -sha256 \
    -config ca.conf -section admin \
    -out admin.csr
  root@jumpbox:~/kubernetes-the-hard-way# ls -l admin.csr
  -rw-r--r-- 1 root root 1830 Jan  6 00:14 admin.csr

  root@jumpbox:~/kubernetes-the-hard-way# openssl req -in admin.csr -text -noout | more
  Certificate Request:
      Data:
          Version: 1 (0x0)
          Subject: CN = admin, O = system:masters
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (4096 bit)
                  Modulus:
                      00:93:3b:e0:7a:87:5d:80:69:b1:ad:6a:38:74:55:
                      c0:a5:d4:5f:f6:cc:9d:b7:3c:6a:dc:7c:f1:fd:cc:
                      c1:dd:00:27:5e:6c:7f:2d:77:81:02:7c:ce:55:36:
                      79:75:dc:83:bf:60:23:67:df:32:53:cc:19:15:0e:
                      04:23:7f:d7:79:da:43:79:a9:57:9d:8e:dd:96:d2:
    ### 중략####

  root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -req -days 3653 -in admin.csr \
    -copy_extensions copyall \
    -sha256 -CA ca.crt \
    -CAkey ca.key \
    -CAcreateserial \
    -out admin.crt
  Certificate request self-signature ok
  subject=CN = admin, O = system:masters
  root@jumpbox:~/kubernetes-the-hard-way# ls -l admin.crt
  openssl x509 -in admin.crt -text -noout | more
  -rw-r--r-- 1 root root 2021 Jan  6 00:16 admin.crt
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              36:ea:c4:ee:d2:b3:c2:0c:0d:4c:4a:26:a3:56:3b:e1:a2:22:fa:c7
          Signature Algorithm: sha256WithRSAEncryption
          Issuer: C = US, ST = Washington, L = Seattle, CN = CA
          Validity
              Not Before: Jan  5 15:16:49 2026 GMT
              Not After : Jan  6 15:16:49 2036 GMT
          Subject: CN = admin, O = system:masters
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (4096 bit)
                  Modulus:
                      00:93:3b:e0:7a:87:5d:80:69:b1:ad:6a:38:74:55:
                      c0:a5:d4:5f:f6:cc:9d:b7:3c:6a:dc:7c:f1:fd:cc:
                      c1:dd:00:27:5e:6c:7f:2d:77:81:02:7c:ce:55:36:
                      79:75:dc:83:bf:60:23:67:df:32:53:cc:19:15:0e:
                      04:23:7f:d7:79:da:43:79:a9:57:9d:8e:dd:96:d2:
                      2f:74:8a:83:41:80:01:21:f3:96:63:87:b6:27:65:
                      b0:e4:aa:5d:1d:31:a4:15:01:c5:98:66:ec:5a:9e:
                      27:78:2d:e8:2f:d1:e6:20:d6:15:d4:1e:43:2a:77:
      ### 중략####

참고 ) kind환경의 admin인증서 확인

  (⎈|kind-myk8s:default) zosys@4:~$ docker exec -i myk8s-control-plane cat /etc/kubernetes/super-admin.conf | grep client-certificate-data | cut -d ':' -f2 | tr -d ' ' | base64 -d | openssl x509 -text -noout
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number: 1007749187623256237 (0xdfc3e8bfef900ad)
          Signature Algorithm: sha256WithRSAEncryption
          Issuer: CN = kubernetes
          Validity
              Not Before: Jan  5 13:47:29 2026 GMT
              Not After : Jan  5 13:52:29 2027 GMT
          Subject: O = system:masters, CN = kubernetes-super-admin
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (2048 bit)
                  Modulus:
                      00:a4:12:67:9f:3d:22:5b:a0:f8:0c:05:5c:d0:11:
                      2c:cb:98:55:7e:d8:84:a9:cc:39:6d:89:c0:c2:12:
                      60:e1:32:ed:28:a4:33:2d:67:89:20:0e:f9:c1:d6:
                      bb:08:a7:9e:ec:f5:0a:de:9c:ca:ea:ed:82:da:50:
                      35:d7:92:2c:85:f0:df:2c:e3:d1:7f:ca:e0:52:32:

  # krew rbac-tool 플러그인 활용하기
  (⎈|kind-myk8s:default) zosys@4:~$ kubectl rbac-tool lookup system:masters
    SUBJECT        | SUBJECT TYPE | SCOPE       | NAMESPACE | ROLE          | BINDING
  -----------------+--------------+-------------+-----------+---------------+----------------
    system:masters | Group        | ClusterRole |           | cluster-admin | cluster-admin

  (⎈|kind-myk8s:default) zosys@4:~$ kubectl describe clusterroles cluster-admin
  Name:         cluster-admin
  Labels:       kubernetes.io/bootstrapping=rbac-defaults
  Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
  PolicyRule:
    Resources  Non-Resource URLs  Resource Names  Verbs
    ---------  -----------------  --------------  -----
    *.*        []                 []              [*]
               [*]                []              [*]

  (⎈|kind-myk8s:default) zosys@4:~$ kubectl describe clusterrolebindings cluster-admin
  Name:         cluster-admin
  Labels:       kubernetes.io/bootstrapping=rbac-defaults
  Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
  Role:
    Kind:  ClusterRole
    Name:  cluster-admin
  Subjects:
    Kind   Name            Namespace
    ----   ----            ---------
    Group  system:masters

나머지 인증서 생성

  #인증서 생성전 오타 변경
  root@jumpbox:~/kubernetes-the-hard-way# sed -i 's/system:system:kube-scheduler/system:kube-scheduler/' ca.conf
  root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf | grep system:kube-scheduler
  CN = system:kube-scheduler
  O  = system:kube-scheduler

  root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf | grep system:kube-scheduler
  CN = system:kube-scheduler
  O  = system:kube-scheduler
  root@jumpbox:~/kubernetes-the-hard-way# certs=(
    "node-0" "node-1"
    "kube-proxy" "kube-scheduler"
    "kube-controller-manager"
    "kube-api-server"
    "service-accounts"
  )
  root@jumpbox:~/kubernetes-the-hard-way# echo ${certs[*]}
  node-0 node-1 kube-proxy kube-scheduler kube-controller-manager kube-api-server service-accounts

  root@jumpbox:~/kubernetes-the-hard-way# for i in ${certs[*]}; do
    openssl genrsa -out "${i}.key" 4096

    openssl req -new -key "${i}.key" -sha256 \
      -config "ca.conf" -section ${i} \
      -out "${i}.csr"

    openssl x509 -req -days 3653 -in "${i}.csr" \
      -copy_extensions copyall \
      -sha256 -CA "ca.crt" \
      -CAkey "ca.key" \
      -CAcreateserial \
      -out "${i}.crt"
  done
  Certificate request self-signature ok
  subject=CN = system:node:node-0, O = system:nodes, C = US, ST = Washington, L = Seattle
  Certificate request self-signature ok
  subject=CN = system:node:node-1, O = system:nodes, C = US, ST = Washington, L = Seattle
  Certificate request self-signature ok
  subject=CN = system:kube-proxy, O = system:node-proxier, C = US, ST = Washington, L = Seattle
  Certificate request self-signature ok
  subject=CN = system:kube-scheduler, O = system:kube-scheduler, C = US, ST = Washington, L = Seattle
  Certificate request self-signature ok
  subject=CN = system:kube-controller-manager, O = system:kube-controller-manager, C = US, ST = Washington, L = Seattle
  Certificate request self-signature ok
  subject=CN = kubernetes, C = US, ST = Washington, L = Seattle
  Certificate request self-signature ok
  subject=CN = service-accounts

참고 ) kind k8s 인증서 보기

  (⎈|kind-myk8s:default) zosys@4:~$ docker exec -it myk8s-control-plane ls -al /etc/kubernetes
  total 60
  drwxr-xr-x 1 root root 4096 Jan  5 13:52 .
  drwxr-xr-x 1 root root 4096 Jan  5 13:52 ..
  -rw------- 1 root root 5643 Jan  5 13:52 admin.conf
  -rw------- 1 root root 5658 Jan  5 13:52 controller-manager.conf
  -rw------- 1 root root 2007 Jan  5 13:52 kubelet.conf
  drwxr-xr-x 1 root root 4096 Jan  5 13:52 manifests
  drwxr-xr-x 3 root root 4096 Jan  5 13:52 pki
  -rw------- 1 root root 5602 Jan  5 13:52 scheduler.conf
  -rw------- 1 root root 5663 Jan  5 13:52 super-admin.conf
  (⎈|kind-myk8s:default) zosys@4:~$ docker exec -it myk8s-control-plane ls -al /var/lib/kubelet/pki
  total 20
  drwxr-xr-x 2 root root 4096 Jan  5 13:52 .
  drwx------ 9 root root 4096 Jan  5 13:52 ..
  -rw------- 1 root root 2839 Jan  5 13:52 kubelet-client-2026-01-05-13-52-31.pem
  lrwxrwxrwx 1 root root   59 Jan  5 13:52 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-01-05-13-52-31.pem
  -rw-r--r-- 1 root root 2343 Jan  5 13:52 kubelet.crt
  -rw------- 1 root root 1679 Jan  5 13:52 kubelet.key

  ## API서버에 Alternative Name확인
  (⎈|kind-myk8s:default) zosys@4:~$ docker exec -i myk8s-control-plane cat /etc/kubernetes/pki/apiserver.crt | openssl x509 -text -noout
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number: 8699350173748144468 (0x78ba4ce452fda954)
          Signature Algorithm: sha256WithRSAEncryption
          Issuer: CN = kubernetes
          Validity
              Not Before: Jan  5 13:47:29 2026 GMT
              Not After : Jan  5 13:52:29 2027 GMT
          Subject: CN = kube-apiserver
          Subject Public Key Info:
              Public Key Algorithm: rsaEncryption
                  Public-Key: (2048 bit)
                  Modulus:
                      00:a5:26:b2:7b:33:e3:a8:c6:01:d5:ba:26:ba:e9:
                      2b:70:58:c1:0b:e3:35:a3:96:d1:de:c5:7e:80:44:
                      0b:61:af:12:1a:dd:e5:83:ed:88:bb:be:c7:c0:6f:
                      05:71:9b:4b:82:49:12:23:a0:46:44:91:ef:68:49:
                      12:45:26:2a:07:28:38:bd:33:c0:76:61:cb:51:af:
                      18:c9:4c:96:a6:db:98:e0:8c:82:50:2f:8a:3e:ed:
                      79:f3:d7:b6:89:45:9e:d2:fb:2c:0a:b2:1f:14:fa:
                      fa:f1:29:cb:5c:2b:d2:26:81:50:e7:0f:98:57:9c:
                      20:90:89:d3:d1:7b:d7:2f:c7:a6:a3:aa:b0:9b:f8:
                      78:c4:57:73:fb:82:a8:9d:1f:c6:c6:38:67:24:49:
                      4f:0f:cb:d7:61:f6:5d:0c:89:cf:b8:01:c6:af:af:
                      51:91:12:b8:57:e0:ab:13:30:c7:a5:1f:a8:24:49:
                      85:1e:e1:8c:d1:19:f8:68:2f:be:b3:eb:37:79:e5:
                      5f:b1:85:78:9e:05:a3:dd:b2:c2:92:03:1f:e1:a3:
                      39:f8:b5:9f:23:b2:b2:1a:c4:05:3a:3e:6c:17:3f:
                      86:94:47:b6:a3:36:87:3e:59:3c:40:06:25:11:a3:
                      26:8f:02:da:cd:c7:00:d0:ca:db:71:75:41:a6:f3:
                      5f:03
                  Exponent: 65537 (0x10001)
          X509v3 extensions:
              X509v3 Key Usage: critical
                  Digital Signature, Key Encipherment
              X509v3 Extended Key Usage:
                  TLS Web Server Authentication
              X509v3 Basic Constraints: critical
                  CA:FALSE
              X509v3 Authority Key Identifier:
                  F3:A5:5A:DF:0E:70:F9:F5:ED:2C:DC:76:A8:34:22:CF:A4:3A:64:F1
              X509v3 Subject Alternative Name:
                  DNS:**kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, DNS:myk8s-control-plane, IP Address:10.96.0.1, IP Address:172.18.0.3, IP Address:127.0.0.1**
      Signature Algorithm: sha256WithRSAEncryption
      Signature Value:
          5c:9e:b9:6f:e7:42:51:84:81:e0:f2:82:be:f8:d9:07:62:8b:
          51:23:0c:56:8f:f0:7a:43:e5:d7:93:b6:2a:0b:ba:98:55:9b:
          81:fd:2a:52:0a:e1:7d:7a:ec:bb:02:dd:d1:72:64:28:ba:d0:
          5a:50:4c:ca:f0:c4:3b:13:c7:9f:04:df:d5:5d:6f:9b:d7:bf:
          18:c5:b4:a3:7c:af:b5:bb:ae:ad:b3:c9:88:ca:6d:25:6c:86:
          5f:c8:d6:cb:ae:fa:2a:d1:ba:43:04:68:7f:78:78:75:9e:9a:
          54:cb:1d:00:f8:f8:91:9e:4b:2c:cb:bd:b7:15:51:1c:c5:80:

  (⎈|kind-myk8s:default) zosys@4:~$ kubectl describe pod -n kube-system kube-apiserver-myk8s-control-plane
  Name:                 kube-apiserver-myk8s-control-plane
  Namespace:            kube-system
  Priority:             2000001000
  #####################중략 후 인증서정보 확인############################ 
        --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
        --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
        --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
        --etcd-servers=https://127.0.0.1:2379
        --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
        --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
        --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
        --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
        --requestheader-allowed-names=front-proxy-client
        --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt

Distribute the Client and Server Certificates(인증서 배포)

  root@jumpbox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
    ssh root@${host} mkdir /var/lib/kubelet/

    scp ca.crt root@${host}:/var/lib/kubelet/

    scp ${host}.crt \
      root@${host}:/var/lib/kubelet/kubelet.crt

    scp ${host}.key \
      root@${host}:/var/lib/kubelet/kubelet.key
  done
  ca.crt                                                                                100% 1899     1.3MB/s   00:00
  node-0.crt                                                                            100% 2147     1.1MB/s   00:00
  node-0.key                                                                            100% 3272     1.9MB/s   00:00
  ca.crt                                                                                100% 1899     1.4MB/s   00:00
  node-1.crt                                                                            100% 2147     1.1MB/s   00:00
  node-1.key                                                                            100% 3268     2.0MB/s   00:00
  root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /var/lib/kubelet
  ssh node-1 ls -l /var/lib/kubelet
  total 12
  -rw-r--r-- 1 root root 1899 Jan  6 00:37 ca.crt
  -rw-r--r-- 1 root root 2147 Jan  6 00:37 kubelet.crt
  -rw------- 1 root root 3272 Jan  6 00:37 kubelet.key
  total 12
  -rw-r--r-- 1 root root 1899 Jan  6 00:37 ca.crt
  -rw-r--r-- 1 root root 2147 Jan  6 00:37 kubelet.crt
  -rw------- 1 root root 3268 Jan  6 00:37 kubelet.key

  root@jumpbox:~/kubernetes-the-hard-way# scp \
    ca.key ca.crt \
    kube-api-server.key kube-api-server.crt \
    service-accounts.key service-accounts.crt \
    root@server:~/
  ca.key                                                                                100% 3272     1.3MB/s   00:00
  ca.crt                                                                                100% 1899     1.4MB/s   00:00
  kube-api-server.key                                                                   100% 3272     2.4MB/s   00:00
  kube-api-server.crt                                                                   100% 2354     1.8MB/s   00:00
  service-accounts.key                                                                  100% 3268     2.2MB/s   00:00
  service-accounts.crt                                                                  100% 2004     1.4MB/s   00:00
  root@jumpbox:~/kubernetes-the-hard-way# ssh server ls -l /root
  total 24
  -rw-r--r-- 1 root root 1899 Jan  6 00:38 ca.crt
  -rw------- 1 root root 3272 Jan  6 00:38 ca.key
  -rw-r--r-- 1 root root 2354 Jan  6 00:38 kube-api-server.crt
  -rw------- 1 root root 3272 Jan  6 00:38 kube-api-server.key
  -rw-r--r-- 1 root root 2004 Jan  6 00:38 service-accounts.crt
  -rw------- 1 root root 3268 Jan  6 00:38 service-accounts.key

05 - Generating Kubernetes Configuration Files for Authentication

The kubelet Kubernetes Configuration File
kind환경의 kubelet설정 확인하기

  (⎈|kind-myk8s:default) zosys@4:~$ kubectl describe pod -n kube-system kube-apiserver-myk8s-control-plane
  Name:                 kube-apiserver-myk8s-control-plane
  Namespace:            kube-system
  Priority:             2000001000
  Priority Class Name:  system-node-critical
  Node:                 myk8s-control-plane/172.18.0.3
  Start Time:           Mon, 05 Jan 2026 22:52:38 +0900
  Labels:               component=kube-apiserver
                        tier=control-plane
  Annotations:          kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.18.0.3:6443
                        kubernetes.io/config.hash: 6dadff8c64cc62e7b3846733d2478bdd
                        kubernetes.io/config.mirror: 6dadff8c64cc62e7b3846733d2478bdd
                        kubernetes.io/config.seen: 2026-01-05T13:52:38.459844636Z
                        kubernetes.io/config.source: file
  Status:               Running
  SeccompProfile:       RuntimeDefault
  IP:                   172.18.0.3
  IPs:
    IP:           172.18.0.3
  Controlled By:  Node/myk8s-control-plane
  Containers:
    kube-apiserver:
      Container ID:  containerd://872e95b24c42f80855f00ffda199192af35f4b24ca9ee16587cfa03e13c692fe
      Image:         registry.k8s.io/kube-apiserver:v1.32.8
      Image ID:      sha256:0d4edaa48e2f940c934e0f7cfd5209fc85e65ab5e842b980f41263d1764661f1
      Port:          <none>
      Host Port:     <none>
      Command:
        kube-apiserver
        --advertise-address=172.18.0.3
        --allow-privileged=true
        --authorization-mode=Node,RBAC

  #kubeconfig 생성 및 확인

  root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=node-0.kubeconfig && ls -l node-0.kubeconfig && cat node-0.kubeconfig
  Cluster "kubernetes-the-hard-way" set.

  #############################중략########################################

  root@jumpbox:~/kubernetes-the-hard-way# ls -l *.kubeconfig
  -rw------- 1 root root 10161 Jan  7 23:02 node-0.kubeconfig
  -rw------- 1 root root 10157 Jan  7 23:03 node-1.kubeconfig

  #kind의 설정확인

  (⎈|kind-myk8s:default) zosys@4:~$ cat .kube/config
  apiVersion: v1
  clusters:
      - cluster:
          certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJZGpzaENHRCtNU1l3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1EVXhNelEzTWpsYUZ3MHpOakF4TURNeE16VXlNamxhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURVcGJGWEJyekgvNFc2MWtSMGpBcWtQanpINXhVSitUSHhvSTZTMEpENEdWS1lqaW1TRnplTm5mNFEKK3RjUEhrR0pVMklZMmJYRXQwTlE1bHo5TElkR3AxU3c2dXB0OXRpVHpCOHZlaGdvYmhmbjR1ZzdKc2l1UFNldwpYY2xnSkNjc0YvNDM5dGtrRVlETXVpSDFFVTZ6QXJ3YkJjSkVHWHk1Z3JrOHpQOG56N245U1UrSlNVOTR2S0RCCjFKb0hqS1doQnhtT3RiczJkSTByenFUcUVxRkRHWGo2b09HejdheGU4YUhBSDlVQ1IySUdFRnVyNHY3c1pTN2gKQ1AydllESEdCT1BXRS9vV20zSVJMZ0xVRnRzbVdSd0hqSTZWVmFyTlVqSXh2dzF0dFBlVDJSYUdRb0dJcExIUQpTdWFoT2NUV1VESkdnd2pCd3owSk80R2xGdnY1QWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJUenBWcmZEbkQ1OWUwczNIYW9OQ0xQcERwazhUQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQmlEMWRkZDkwUQpaS2UvU2pNbTQwYVVtZlU2YVhFNG9DMndJT1RwMlkxNjBobGMrUXF6aUZEdktGY3k2MjVzWEpRcDRmb2lBajFtCmxkWGdLcTZTVDM1aVFqdGU0OGhYNG95bzNXQ1lYYXYvWkJDVmZoZ2pBTmlKelAwUTAzc2lTdVk2RlNxTDBHTysKZktncEVGS3luclFKdmZ6ckVmU3gzTWJqRTdYOEk5QVZmUTUxLzhFNEVyb1JQUzBVRXZGbGdiUCtwWWQveTZsVgpUTm8rMjFCN0V0OThhU2Y5V3lrQnZpMTZZeUhGckJPOUkwRGdtNlFxSVl0QXd6S1N5dmRCTitXVk12UitHSFBmCkVwY3VONUcyQ2x6dUd5aTJNU0VCRzhnRDNnL29uSjQwL3p3MHdTMU1JT1lweU5FMnhtNExhYll3c3dqOWVBZkQKQ0hkbndaOFdLMU15Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
          server: https://127.0.0.1:34991
        name: kind-myk8s

  #(참고) kind system:node관련 정보
  (⎈|kind-myk8s:default) zosys@4:~$ kubectl describe clusterroles system:node
  Name:         system:node
  Labels:       kubernetes.io/bootstrapping=rbac-defaults
  Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
  PolicyRule:
    Resources                                       Non-Resource URLs  Resource Names  Verbs
    ---------                                       -----------------  --------------  -----
    leases.coordination.k8s.io                      []                 []              [create delete get patch update]
    csinodes.storage.k8s.io                         []                 []              [create delete get patch update]
    nodes                                           []                 []              [create get list watch patch update]
    certificatesigningrequests.certificates.k8s.io  []                 []              [create get list watch]
    events                                          []                 []              [create patch update]
    pods/eviction                                   []                 []              [create]
    serviceaccounts/token                           []                 []              [create]
    tokenreviews.authentication.k8s.io              []                 []              [create]
    localsubjectaccessreviews.authorization.k8s.io  []                 []              [create]
    subjectaccessreviews.authorization.k8s.io       []                 []              [create]
    pods                                            []                 []              [get list watch create delete]
    configmaps                                      []                 []              [get list watch]
    secrets                                         []                 []              [get list watch]
    services                                        []                 []              [get list watch]
    runtimeclasses.node.k8s.io                      []                 []              [get list watch]
    csidrivers.storage.k8s.io                       []                 []              [get list watch]
    persistentvolumeclaims/status                   []                 []              [get patch update]
    endpoints                                       []                 []              [get]
    persistentvolumeclaims                          []                 []              [get]
    persistentvolumes                               []                 []              [get]
    volumeattachments.storage.k8s.io                []                 []              [get]
    nodes/status                                    []                 []              [patch update]
    pods/status                                     []                 []              [patch update]

  #####################################중략###########################################

kube-proxy configfile생성

  root@jumpbox:~/kubernetes-the-hard-way# cat kube-proxy.kubeconfig
  apiVersion: v1
  clusters
  - cluster:
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL
      ######중략######

kube-controller-manager configfile 생성

  root@jumpbox:~/kubernetes-the-hard-way# cat kube-controller-manager.kubeconfig
  apiVersion: v1
  clusters:
  - cluster:
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0URS0
      ######중략######

kube-scheduler configfile생성

  root@jumpbox:~/kubernetes-the-hard-way# cat kube-scheduler.kubeconfig
  apiVersion: v1
  clusters:
  - cluster:
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZURENDQXpTZ0F3SUJBZ0lVSWFXMFpIeWpQYWVrUDBBVUJla2x1U0dwZnI0d0RRWUpLb1pJaHZjTkFRRU4KQlFBd1FURUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2xkaGMyaHBibWQwYjI0eEVEQU9CZ05WQkFjTQpCMU5sWVhSMGJHVXhDekFKQmdOVkJBTU1Ba05CTUI0WERUSTJNREV3TlRFMU1EWXlOMW9YRFRNMk1ERXdOakUxCk1EWXlOMW93UVRFTE1Ba0dBMVVFQmhNQ1ZWTXhFekFSQmdOVkJBZ01DbGRoYzJocGJtZDBiMjR4RURBT0JnTl

      ######중략#######

admin configfile생성

  root@jumpbox:~/kubernetes-the-hard-way# cat admin.kubeconfig
  apiVersion: v1
  clusters:
  - cluster:
      certificate-authority-data: LS0tLS1CRUdJTiBDRV

      ######중략#########

Distribute the Kubernetes Configuration Files

  root@jumpbox:~/kubernetes-the-hard-way# ls -l *.kubeconfig
  -rw------- 1 root root  9953 Jan  7 23:12 admin.kubeconfig
  -rw------- 1 root root 10305 Jan  7 23:09 kube-controller-manager.kubeconfig
  -rw------- 1 root root 10187 Jan  7 23:08 kube-proxy.kubeconfig
  -rw------- 1 root root 10215 Jan  7 23:10 kube-scheduler.kubeconfig
  -rw------- 1 root root 10161 Jan  7 23:02 node-0.kubeconfig
  -rw------- 1 root root 10157 Jan  7 23:03 node-1.kubeconfig

  # 확인
  root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /var/lib/*/kubeconfig
  ssh node-1 ls -l /var/lib/*/kubeconfig
  -rw------- 1 root root 10161 Jan  7 23:13 /var/lib/kubelet/kubeconfig
  -rw------- 1 root root 10187 Jan  7 23:13 /var/lib/kube-proxy/kubeconfig
  -rw------- 1 root root 10157 Jan  7 23:13 /var/lib/kubelet/kubeconfig
  -rw------- 1 root root 10187 Jan  7 23:13 /var/lib/kube-proxy/kubeconfig
  ssh server ls -l /root/*.kubeconfig

06 - Generating the Data Encryption Config and Key

root@jumpbox:~/kubernetes-the-hard-way# cat configs/encryption-config.yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}


  root@jumpbox:~/kubernetes-the-hard-way# envsubst < configs/encryption-config.yaml > encryption-config.yaml
root@jumpbox:~/kubernetes-the-hard-way# cat encryption-config.yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret:
      - identity: {}    

root@jumpbox:~/kubernetes-the-hard-way# scp encryption-config.yaml root@server:~/
encryption-config.yaml                                                                                  100%  227   183.1KB/s   00:00
root@jumpbox:~/kubernetes-the-hard-way# ssh server ls -l /root/encryption-config.yaml
-rw-r--r-- 1 root root 227 Jan  7 23:26 /root/encryption-config.yaml

07 - Bootstrapping the etcd Cluster

##etcd 기동
root@jumpbox:~/kubernetes-the-hard-way# cat units/etcd.service | grep controller
  --name controller \
  --initial-cluster controller=http://127.0.0.1:2380 \

  root@jumpbox:~/kubernetes-the-hard-way# ETCD_NAME=server
cat > units/etcd.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --initial-advertise-peer-urls http://127.0.0.1:2380 \\
  --listen-peer-urls http://127.0.0.1:2380 \\
  --listen-client-urls http://127.0.0.1:2379 \\
  --advertise-client-urls http://127.0.0.1:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster ${ETCD_NAME}=http://127.0.0.1:2380 \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
root@jumpbox:~/kubernetes-the-hard-way# cat units/etcd.service | grep server
  --name server \
  --initial-cluster server=http://127.0.0.1:2380 \

root@jumpbox:~/kubernetes-the-hard-way# cat units/etcd.service | grep server
  --name server \
  --initial-cluster server=http://127.0.0.1:2380 \
root@jumpbox:~/kubernetes-the-hard-way# scp \
  downloads/controller/etcd \
  downloads/client/etcdctl \
  units/etcd.service \
  root@server:~/
etcd                                                                                                    100%   24MB  47.3MB/s   00:00
etcdctl                                                                                                 100%   16MB  63.4MB/s   00:00
etcd.service 
root@jumpbox:~/kubernetes-the-hard-way# ssh root@server

root@server:~# mv etcd etcdctl /usr/local/bin/
root@server:~# mkdir -p /etc/etcd /var/lib/etcd
root@server:~# chmod 700 /var/lib/etcd
root@server:~# cp ca.crt kube-api-server.key kube-api-server.crt /etc/etcd/
root@server:~# systemctl status etcd --no-pager
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-01-07 23:29:51 KST; 4s ago
       Docs: https://github.com/etcd-io/etcd
   Main PID: 2607 (etcd)
      Tasks: 8 (limit: 2297)
     Memory: 10.4M
        CPU: 97ms            
 #######중략###########
 root@server:~# etcdctl member list
6702b0a34e2cfd39, started, server, http://127.0.0.1:2380, http://127.0.0.1:2379, false
root@server:~# etcdctl member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6702b0a34e2cfd39 | started | server | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+
root@server:~# etcdctl endpoint status -w table
+----------------+------------------+------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
|    ENDPOINT    |        ID        |  VERSION   | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+----------------+------------------+------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| 127.0.0.1:2379 | 6702b0a34e2cfd39 | 3.6.0-rc.3 |           3.6.0 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         2 |          4 |                  4 |        |                          |             false |
+----------------+------------------+------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

08 - Bootstrapping the Kubernetes Control Plane

설정파일 작성 및 server전달

  root@jumpbox:~/kubernetes-the-hard-way# cat ca.conf | grep '\[kube-api-server_alt_names' -A2
  [kube-api-server_alt_names]
  IP.0  = 127.0.0.1
  IP.1  = 10.32.0.1

  root@jumpbox:~/kubernetes-the-hard-way# cat units/kube-apiserver.service
  [Unit]
  Description=Kubernetes API Server
  Documentation=https://github.com/kubernetes/kubernetes

  [Service]
  ExecStart=/usr/local/bin/kube-apiserver \
    --allow-privileged=true \
    --apiserver-count=1 \
    --audit-log-maxage=30 \
    --audit-log-maxbackup=3 \
    --audit-log-maxsize=100 \
    --audit-log-path=/var/log/audit.log \
    --authorization-mode=Node,RBAC \

    ##########중략################

    root@jumpbox:~/kubernetes-the-hard-way# cat configs/kube-apiserver-to-kubelet.yaml ; echo
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:kube-apiserver-to-kubelet
  rules:
    - apiGroups:
        - ""

   #apiserver subject CN확인
   root@jumpbox:~/kubernetes-the-hard-way# openssl x509 -in kube-api-server.crt -text -noout
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              36:ea:c4:ee:d2:b3:c2:0c:0d:4c:4a:26:a3:56:3b:e1:a2:22:fa:cd
          Signature Algorithm: sha256WithRSAEncryption
          Issuer: C = US, ST = Washington, L = Seattle, CN = CA
          Validity
              Not Before: Jan  5 15:25:45 2026 GMT
              Not After : Jan  6 15:25:45 2036 GMT
          **Subject: CN = kubernetes, C = US, ST = Washington, L = Seattle**
  ####중략#####

  # kube-scheduler
  root@jumpbox:~/kubernetes-the-hard-way# cat units/kube-scheduler.service ; echo
  [Unit]
  Description=Kubernetes Scheduler
  Documentation=https://github.com/kubernetes/kubernetes

  [Service]
  ExecStart=/usr/local/bin/kube-scheduler \
    --config=/etc/kubernetes/config/kube-scheduler.yaml \
    --v=2
  Restart=on-failure
  RestartSec=5

  [Install]
  WantedBy=multi-user.target
  root@jumpbox:~/kubernetes-the-hard-way# cat configs/kube-scheduler.yaml ; echo
  apiVersion: kubescheduler.config.k8s.io/v1
  kind: KubeSchedulerConfiguration
  clientConnection:
    kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
  leaderElection:
    leaderElect: true

    root@jumpbox:~/kubernetes-the-hard-way# cat units/kube-controller-manager.service ; echo
  [Unit]
  Description=Kubernetes Controller Manager
  Documentation=https://github.com/kubernetes/kubernetes

  [Service]
  ExecStart=/usr/local/bin/kube-controller-manager \
    --bind-address=0.0.0.0 \
    --cluster-cidr=10.200.0.0/16 \
    --cluster-name=kubernetes \
    --cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \
    --cluster-signing-key-file=/var/lib/kubernetes/ca.key \
    --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
    --root-ca-file=/var/lib/kubernetes/ca.crt \
    --service-account-private-key-file=/var/lib/kubernetes/service-accounts.key \
    --service-cluster-ip-range=10.32.0.0/24 \
    --use-service-account-credentials=true \
    --v=2
  Restart=on-failure
  RestartSec=5

  [Install]
  WantedBy=multi-user.target
  #SCP전송후 확인
  root@jumpbox:~/kubernetes-the-hard-way# ssh server ls -l /root
  total 295436
  -rw------- 1 root root     9953 Jan  7 23:14 admin.kubeconfig
  -rw-r--r-- 1 root root     1899 Jan  6 00:38 ca.crt
  -rw------- 1 root root     3272 Jan  6 00:38 ca.key
  -rw-r--r-- 1 root root      227 Jan  7 23:26 encryption-config.yaml
  -rwxr-xr-x 1 root root 93261976 Jan  7 23:38 kube-apiserver
  -rw-r--r-- 1 root root     2354 Jan  6 00:38 kube-api-server.crt
  -rw------- 1 root root     3272 Jan  6 00:38 kube-api-server.key
  -rw-r--r-- 1 root root     1442 Jan  7 23:38 kube-apiserver.service
  -rw-r--r-- 1 root root      727 Jan  7 23:38 kube-apiserver-to-kubelet.yaml
  -rwxr-xr-x 1 root root 85987480 Jan  7 23:38 kube-controller-manager
  -rw------- 1 root root    10305 Jan  7 23:14 kube-controller-manager.kubeconfig
  -rw-r--r-- 1 root root      735 Jan  7 23:38 kube-controller-manager.service
  -rwxr-xr-x 1 root root 57323672 Jan  7 23:38 kubectl
  -rwxr-xr-x 1 root root 65843352 Jan  7 23:38 kube-scheduler
  -rw------- 1 root root    10215 Jan  7 23:14 kube-scheduler.kubeconfig
  -rw-r--r-- 1 root root      281 Jan  7 23:38 kube-scheduler.service
  -rw-r--r-- 1 root root      191 Jan  7 23:38 kube-scheduler.yaml
  -rw-r--r-- 1 root root     2004 Jan  6 00:38 service-accounts.crt
  -rw------- 1 root root     3268 Jan  6 00:38 service-accounts.key

Provision the Kubernetes Control Plane : kubectl 확인

  root@server:~# mkdir -p /etc/kubernetes/config
  root@server:~# mv kube-apiserver \
    kube-controller-manager \
    kube-scheduler kubectl \
    /usr/local/bin/
  root@server:~# ls -l /usr/local/bin/kube-*
  -rwxr-xr-x 1 root root 93261976 Jan  7 23:38 /usr/local/bin/kube-apiserver
  -rwxr-xr-x 1 root root 85987480 Jan  7 23:38 /usr/local/bin/kube-controller-manager
  -rwxr-xr-x 1 root root 65843352 Jan  7 23:38 /usr/local/bin/kube-scheduler

  # Configure the Kubernetes API Server 확인

  root@server:~# mkdir -p /var/lib/kubernetes/
  root@server:~# mv ca.crt ca.key \
    kube-api-server.key kube-api-server.crt \
    service-accounts.key service-accounts.crt \
    encryption-config.yaml \
    /var/lib/kubernetes/
  root@server:~# ls -l /var/lib/kubernetes/
  total 28
  -rw-r--r-- 1 root root 1899 Jan  6 00:38 ca.crt
  -rw------- 1 root root 3272 Jan  6 00:38 ca.key
  -rw-r--r-- 1 root root  227 Jan  7 23:26 encryption-config.yaml
  -rw-r--r-- 1 root root 2354 Jan  6 00:38 kube-api-server.crt
  -rw------- 1 root root 3272 Jan  6 00:38 kube-api-server.key
  -rw-r--r-- 1 root root 2004 Jan  6 00:38 service-accounts.crt
  -rw------- 1 root root 3268 Jan  6 00:38 service-accounts.key

  #파일 이동및 실행
  root@server:~# mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
  root@server:~# mv kube-controller-manager.service /etc/systemd/system/
  root@server:~# mv kube-scheduler.kubeconfig /var/lib/kubernetes/
  root@server:~# mv kube-scheduler.yaml /etc/kubernetes/config/
  root@server:~# mv kube-scheduler.service /etc/systemd/system/
  root@server:~# systemctl daemon-reload
  root@server:~# systemctl enable kube-apiserver kube-controller-manager kube-scheduler
  Created symlink /etc/systemd/system/multi-user.target.wants/kube-apiserver.service → /etc/systemd/system/kube-apiserver.service.
  Created symlink /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service → /etc/systemd/system/kube-controller-manager.service.
  Created symlink /etc/systemd/system/multi-user.target.wants/kube-scheduler.service → /etc/systemd/system/kube-scheduler.service.
  root@server:~# systemctl start  kube-apiserver kube-controller-manager kube-scheduler

  root@server:~# ss -tlp | grep kube
  LISTEN 0      4096               *:6443              *:*    users:(("kube-apiserver",pid=4929,fd=3))
  LISTEN 0      4096               *:10259             *:*    users:(("kube-scheduler",pid=2752,fd=3))
  LISTEN 0      4096               *:10257             *:*    users:(("kube-controller",pid=2751,fd=3))

  root@server:~# kubectl get clusterroles --kubeconfig admin.kubeconfig
  NAME                                                                   CREATED AT
  admin                                                                  2026-01-07T15:07:34Z
  cluster-admin                                                          2026-01-07T15:07:34Z
  edit                                                                   2026-01-07T15:07:34Z
  system:aggregate-to-admin                                              2026-01-07T15:07:34Z
  system:aggregate-to-edit                                               2026-01-07T15:07:34Z
  system:aggregate-to-view                                               2026-01-07T15:07:34Z
  system:auth-delegator                                                  2026-01-07T15:07:34Z
  system:basic-user                                                      2026-01-07T15:07:34Z
  system:certificates.k8s.io:certificatesigningrequests:nodeclient       2026-01-07T15:07:34Z
  system:certificates.k8s.io:certificatesigningrequests:selfnodeclient   2026-01-07T15:07:34Z
  system:certificates.k8s.io:kube-apiserver-client-approver              2026-01-07T15:07:34Z

RBAC for Kubelet Authorization

  root@server:~# cat kube-apiserver-to-kubelet.yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:kube-apiserver-to-kubelet
  rules:
    - apiGroups:
        - ""
      resources:
        - nodes/proxy
        - nodes/stats
        - nodes/log
        - nodes/spec
        - nodes/metrics
      verbs:
        - "*"

root@server:~# kubectl get clusterroles system:kube-apiserver-to-kubelet --kubeconfig admin.kubeconfig
NAME                               CREATED AT
system:kube-apiserver-to-kubelet   2026-01-07T15:10:03Z
root@server:~# kubectl get clusterrolebindings system:kube-apiserver --kubeconfig admin.kubeconfig

NAME                    ROLE                                           AGE
system:kube-apiserver   ClusterRole/system:kube-apiserver-to-kubelet   18s      
```

- jumpbox 서버에서 컨트롤플레인 확인

```bash
root@jumpbox:~/kubernetes-the-hard-way# curl -s -k --cacert ca.crt https://server.kubernetes.local:6443/version | jq
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.3",
  "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
  "gitTreeState": "clean",
  "buildDate": "2025-03-11T19:52:21Z",
  "goVersion": "go1.23.6",
  "compiler": "gc",
  "platform": "linux/amd64"
}
```

09 - Bootstrapping the Kubernetes Worker Nodes

워커노드세팅을 위한 사전준비

  root@jumpbox:~/kubernetes-the-hard-way# cat configs/10-bridge.conf | jq
  {
    "cniVersion": "1.0.0",
    "name": "bridge",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
      "type": "host-local",
      "ranges": [
        [
          {
            "subnet": "SUBNET"
          }
        ]
      ],
      "routes": [
        {
          "dst": "0.0.0.0/0"
        }
      ]
    }
  }

  root@jumpbox:~/kubernetes-the-hard-way# cat configs/kubelet-config.yaml | yq
  {
    "kind": "KubeletConfiguration",
    "apiVersion": "kubelet.config.k8s.io/v1beta1",
    "address": "0.0.0.0",
    "authentication": {
      "anonymous": {
        "enabled": false
      },
      "webhook": {
        "enabled": true
      },
  #######중략#############

  #파일전달
  root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /root
  ssh node-1 ls -l /root
  total 8
  -rw-r--r-- 1 root root 265 Jan  8 00:18 10-bridge.conf
  -rw-r--r-- 1 root root 610 Jan  8 00:18 kubelet-config.yaml
  total 8
  -rw-r--r-- 1 root root 265 Jan  8 00:18 10-bridge.conf
  -rw-r--r-- 1 root root 610 Jan  8 00:18 kubelet-config.yaml

  root@jumpbox:~/kubernetes-the-hard-way# cat configs/99-loopback.conf ; echo
  cat configs/containerd-config.toml ; echo
  cat configs/kube-proxy-config.yaml ; echo
  {
    "cniVersion": "1.1.0",
    "name": "lo",
    "type": "loopback"
  }
  version = 2

  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
      snapshotter = "overlayfs"
      default_runtime_name = "runc"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true
  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "/opt/cni/bin"
    conf_dir = "/etc/cni/net.d"
  kind: KubeProxyConfiguration
  apiVersion: kubeproxy.config.k8s.io/v1alpha1
  clientConnection:
    kubeconfig: "/var/lib/kube-proxy/kubeconfig"
  mode: "iptables"
  clusterCIDR: "10.200.0.0/16"

  #파일전송 및 확인
  root@jumpbox:~/kubernetes-the-hard-way# ssh node-0 ls -l /root
  ssh node-1 ls -l /root
  ssh node-0 ls -l /root/cni-plugins
  ssh node-1 ls -l /root/cni-plugins
  total 358584
  -rw-r--r-- 1 root root      265 Jan  8 00:18 10-bridge.conf
  -rw-r--r-- 1 root root       65 Jan  8 00:18 99-loopback.conf
  drwxr-xr-x 2 root root     4096 Jan  8 00:19 cni-plugins
  -rwxr-xr-x 1 root root 58584656 Jan  8 00:18 containerd
  -rw-r--r-- 1 root root      470 Jan  8 00:18 containerd-config.toml

  #########중략##########

node0 세팅

  root@jumpbox:~/kubernetes-the-hard-way# ssh root@node-0

  root@node-0:~# mkdir -p \
    /etc/cni/net.d \
    /opt/cni/bin \
    /var/lib/kubelet \
    /var/lib/kube-proxy \
    /var/lib/kubernetes \
    /var/run/kubernetes

  root@node-0:~# mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
  cat /etc/cni/net.d/10-bridge.conf
  {
    "cniVersion": "1.0.0",
    "name": "bridge",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
      "type": "host-local",
      "ranges": [
        [{"subnet": "10.200.0.0/24"}]
      ],
      "routes": [{"dst": "0.0.0.0/0"}]
    }
  }root@node-0:~#lsmod | grep netfilterr
  modprobe br-netfilter
  echo "br-netfilter" >> /etc/modules-load.d/modules.conf
  lsmod | grep netfilter
  br_netfilter           36864  0
  bridge                319488  1 br_netfilter

  root@node-0:~# mkdir -p /etc/containerd/
  mv containerd-config.toml /etc/containerd/config.toml
  mv containerd.service /etc/systemd/system/
  cat /etc/containerd/config.toml ; echo
  version = 2

  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
      snapshotter = "overlayfs"
      default_runtime_name = "runc"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true
  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "/opt/cni/bin"
    conf_dir = "/etc/cni/net.d"

  ###############중략############################

  #node0세팅확인
  root@jumpbox:~/kubernetes-the-hard-way# ssh server "kubectl get nodes -owide --kubeconfig admin.kubeconfig"
  NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
  node-0   Ready    <none>   19s   v1.32.3   192.168.10.101   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-40-amd64   containerd://2.1.0-beta.0

node1 세팅(node0과 동일하기에 결과값만 반환하고 세팅 생략)

  root@jumpbox:~/kubernetes-the-hard-way# ssh server "kubectl get nodes -owide --kubeconfig admin.kubeconfig"
  NAME     STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
  node-0   Ready    <none>   2m7s   v1.32.3   192.168.10.101   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-40-amd64   containerd://2.1.0-beta.0
  node-1   Ready    <none>   7s     v1.32.3   192.168.10.102   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-40-amd64   containerd://2.1.0-beta.0

10 - Configuring kubectl for Remote Access

jumpbox에 admin자격증명 세팅진행(kubectl사용)

  root@jumpbox:~/kubernetes-the-hard-way# curl -s --cacert ca.crt https://server.kubernetes.local:6443/version | jq
  {
    "major": "1",
    "minor": "32",
    "gitVersion": "v1.32.3",
    "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
    "gitTreeState": "clean",
    "buildDate": "2025-03-11T19:52:21Z",
    "goVersion": "go1.23.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  }

  root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443
  Cluster "kubernetes-the-hard-way" set.
  root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-credentials admin \
    --client-certificate=admin.crt \
    --client-key=admin.key
  User "admin" set.
  root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-credentials admin \
    --client-certificate=admin.crt \
    --client-key=admin.key
  User "admin" set.
  root@jumpbox:~/kubernetes-the-hard-way# kubectl config set-context kubernetes-the-hard-way \
    --cluster=kubernetes-the-hard-way \
    --user=admin
  Context "kubernetes-the-hard-way" created.
  root@jumpbox:~/kubernetes-the-hard-way# kubectl config use-context kubernetes-the-hard-way
  Switched to context "kubernetes-the-hard-way".

  root@jumpbox:~/kubernetes-the-hard-way# kubectl version
  Client Version: v1.32.3
  Kustomize Version: v5.5.0
  Server Version: v1.32.3

11 - Provisioning Pod Network Routes

node-0/1 pod간 통신을 위한 라우팅 수동설정진행

  root@jumpbox:~/kubernetes-the-hard-way# SERVER_IP=$(grep server machines.txt | cut -d " " -f 1)
  NODE_0_IP=$(grep node-0 machines.txt | cut -d " " -f 1)
  NODE_0_SUBNET=$(grep node-0 machines.txt | cut -d " " -f 4)
  NODE_1_IP=$(grep node-1 machines.txt | cut -d " " -f 1)
  NODE_1_SUBNET=$(grep node-1 machines.txt | cut -d " " -f 4)
  echo $SERVER_IP $NODE_0_IP $NODE_0_SUBNET $NODE_1_IP $NODE_1_SUBNET
  192.168.10.100 192.168.10.101 10.200.0.0/24 192.168.10.102 10.200.1.0/24

  root@jumpbox:~/kubernetes-the-hard-way# ssh server ip -c route
  default via 10.0.2.2 dev eth0
  10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
  192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.100

12 - Smoke Test

Data encryption

  root@jumpbox:~/kubernetes-the-hard-way# kubectl create secret generic kubernetes-the-hard-way --from-literal="mykey=mydata"
  secret/kubernetes-the-hard-way created

  #정상 적용 확인
  root@jumpbox:~/kubernetes-the-hard-way# kubectl get secret kubernetes-the-hard-way -o jsonpath='{.data.mykey}' ; echo
  bXlkYXRh

  root@jumpbox:~/kubernetes-the-hard-way# ssh root@server \
      'etcdctl get /registry/secrets/default/kubernetes-the-hard-way | hexdump -C'
  00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
  00000010  73 2f 64 65 66 61 75 6c  74 2f 6b 75 62 65 72 6e  |s/default/kubern|
  00000020  65 74 65 73 2d 74 68 65  2d 68 61 72 64 2d 77 61  |etes-the-hard-wa|
  00000030  79 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |y.k8s:enc:aescbc|
  00000040  3a 76 31 3a 6b 65 79 31  3a 96 7c 14 e2 52 45 50  |:v1:key1:.|..REP|
  00000050  f0 6a a5 66 a0 ac 73 62  d2 d4 63 e0 96 7a d5 55  |.j.f..sb..c..z.U|
  00000060  ff a5 3d 3f 47 fc 53 c7  f3 90 de c2 bb e4 ae 60  |..=?G.S........`|
  00000070  d0 0f 1e 34 a5 a0 59 11  d4 f2 8a 32 21 85 5f f5  |...4..Y....2!._.|
  00000080  1c 01 f5 58 96 dc e1 cc  e4 1a 47 c7 48 26 f6 53  |...X......G.H&.S|
  00000090  b0 12 b4 8e f5 eb 8a 01  c6 a7 7c 67 78 57 9c e0  |..........|gxW..|
  000000a0  a1 06 84 67 8c 57 c4 a0  23 1c 6a d0 a2 62 8b 78  |...g.W..#.j..b.x|
  000000b0  8c dc fe 68 60 f8 8d 38  14 90 46 97 bc ae 4d d5  |...h`..8..F...M.|
  000000c0  37 76 8f a9 fd 74 b8 85  f0 09 8d d7 0c 61 3e e3  |7v...t.......a>.|
  000000d0  04 1a a8 99 80 15 45 7d  a5 41 b7 75 54 a6 e0 dc  |......E}.A.uT...|
  000000e0  0e 57 ae e7 3b 8b bd 1b  43 25 39 2e 04 4b 90 be  |.W..;...C%9..K..|
  000000f0  ab 3d d2 0c e7 9c 97 cf  2d 3d 2f 91 b9 f3 05 f6  |.=......-=/.....|
  00000100  3f 47 93 3a a8 dd e3 54  55 15 42 8a 39 45 cb 2b  |?G.:...TU.B.9E.+|
  00000110  c3 cb 2d bf df 5f 2b c4  12 58 38 11 73 6a c6 f8  |..-.._+..X8.sj..|
  00000120  f7 97 1b bd d3 e3 95 e1  f5 ef d1 fb 5e 4b 1b ab  |............^K..|
  00000130  36 22 7c 3d d0 e8 80 b4  4d 85 20 05 9f d4 c2 10  |6"|=....M. .....|
  00000140  96 23 c0 88 e3 a1 22 f7  61 cd 70 00 86 18 5c 24  |.#....".a.p...\$|
  00000150  9a a4 14 e3 4b 39 39 57  ee 0a                    |....K99W..|
  0000015a

Deployments , Port Forwarding , Log, Exec, Service(NodePort)

  root@jumpbox:~/kubernetes-the-hard-way# kubectl get pod
  kubectl create deployment nginx --image=nginx:latest
  kubectl scale deployment nginx --replicas=2
  kubectl get pod -owide
  No resources found in default namespace.
  deployment.apps/nginx created
  deployment.apps/nginx scaled
  NAME                     READY   STATUS              RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
  nginx-54c98b4f84-dkn2w   0/1     ContainerCreating   0          0s    <none>   node-1   <none>           <none>
  nginx-54c98b4f84-kws4j   0/1     ContainerCreating   0          0s    <none>   node-0   <none>           <none>
  root@jumpbox:~/kubernetes-the-hard-way# kubectl get pod -owide
  NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
  nginx-54c98b4f84-dkn2w   1/1     Running   0          44s   10.200.1.2   node-1   <none>           <none>
  nginx-54c98b4f84-kws4j   1/1     Running   0          44s   10.200.0.2   node-0   <none>           <none>

  # server 노드에서 파드 IP로 호출 확인
  root@jumpbox:~/kubernetes-the-hard-way# ssh server curl -s 10.200.1.2 | grep title
  <title>Welcome to nginx!</title>
  root@jumpbox:~/kubernetes-the-hard-way# ssh server curl -s 10.200.0.2 | grep title
  <title>Welcome to nginx!</title>

  root@jumpbox:~/kubernetes-the-hard-way# echo $POD_NAME
  nginx-54c98b4f84-dkn2w
  root@jumpbox:~/kubernetes-the-hard-way# kubectl port-forward $POD_NAME 8080:80 &
  [1] 3130
  root@jumpbox:~/kubernetes-the-hard-way# Forwarding from 127.0.0.1:8080 -> 80
  Forwarding from [::1]:8080 -> 80

실습 환경 정리

  C:\Users\bom\Desktop\스터디\onpremisk8s>vagrant destroy -f
  ==> node-1: Forcing shutdown of VM...
  ==> node-1: Destroying VM and associated drives...
  ==> node-0: Forcing shutdown of VM...
  ==> node-0: Destroying VM and associated drives...
  ==> server: Forcing shutdown of VM...
  ==> server: Destroying VM and associated drives...
  ==> jumpbox: Forcing shutdown of VM...
  ==> jumpbox: Destroying VM and associated drives...

  C:\Users\bom\Desktop\스터디\onpremisk8s>rmdir /s /q .vagrant

'Study > K8S-Deploy' 카테고리의 다른 글

K8S)2주차 과제 (0)	2026.01.15

ZO