실습 환경 배포
| NAME | Description | CPU | RAM | NIC1 | NIC2 | Init Script |
|---|---|---|---|---|---|---|
| admin-lb | kubespary 실행, API LB | 2 | 1GB | 10.0.2.15 | 192.168.10.10 | admin-lb.sh |
| k8s-node1 | K8S ControlPlane | 4 | 2GB | 10.0.2.15 | 192.168.10.11 | init-cfg.sh |
| k8s-node2 | K8S ControlPlane | 4 | 2GB | 10.0.2.15 | 192.168.10.12 | init-cfg.sh |
| k8s-node3 | K8S ControlPlane | 4 | 2GB | 10.0.2.15 | 192.168.10.13 | init-cfg.sh |
| k8s-node4 | K8S Worker | 4 | 2GB | 10.0.2.15 | 192.168.10.14 | init-cfg.sh |
| k8s-node5 | K8S Worker | 4 | 2GB | 10.0.2.15 | 192.168.10.15 | init-cfg.sh |
1. 실습환경배포
howoo@ttokkang-ui-MacBookAir ~/Desktop/work/Gasida_series/k8s-ha-kubespary curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-ha-kubespary/Vagrantfile
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1894 100 1894 0 0 4894 0 --:--:-- --:--:-- --:--:-- 4906
howoo@ttokkang-ui-MacBookAir ~/Desktop/work/Gasida_series/k8s-ha-kubespary curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-ha-kubespary/admin-lb.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6930 100 6930 0 0 10127 0 --:--:-- --:--:-- --:--:-- 10116
howoo@ttokkang-ui-MacBookAir ~/Desktop/work/Gasida_series/k8s-ha-kubespary curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-ha-kubespary/init_cfg.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1709 100 1709 0 0 5381 0 --:--:-- --:--:-- --:--:-- 5374
howoo@ttokkang-ui-MacBookAir ~/Desktop/work/Gasida_series/k8s-ha-kubespary vagrant up
Bringing machine 'k8s-node1' up with 'virtualbox' provider...
Bringing machine 'k8s-node2' up with 'virtualbox' provider...
Bringing machine 'k8s-node3' up with 'virtualbox' provider...
Bringing machine 'k8s-node4' up with 'virtualbox' provider...
Bringing machine 'k8s-node5' up with 'virtualbox' provider...
Bringing machine 'admin-lb' up with 'virtualbox' provider...
==> k8s-node1: Cloning VM...
==> k8s-node1: Matching MAC address for NAT networking...
==> k8s-node1: Checking if box 'bento/rockylinux-10.0' version '202510.26.0' is up to date...
==> k8s-node1: Setting the name of the VM: k8s-node1
==> k8s-node1: Clearing any previously set network interfaces...
==> k8s-node1: Preparing network interfaces based on configuration...
k8s-node1: Adapter 1: nat
k8s-node1: Adapter 2: hostonly
==> k8s-node1: Forwarding ports...
k8s-node1: 22 (guest) => 60001 (host) (adapter 1)
==> k8s-node1: Running 'pre-boot' VM customizations...
==> k8s-node1: Booting VM...
==> k8s-node1: Waiting for machine to boot. This may take a few minutes...
k8s-node1: SSH address: 127.0.0.1:60001
k8s-node1: SSH username: vagrant
k8s-node1: SSH auth method: private key
.
.
.
admin-lb: curl: (6) Could not resolve host: raw.githubusercontent.com
admin-lb: [TASK 16] ETC
admin-lb: >>>> Initial Config End <<<<
howoo@ttokkang-ui-MacBookAir ~/Desktop/work/Gasida_series/k8s-ha-kubespary vagrant status
Current machine states:
k8s-node1 running (virtualbox)
k8s-node2 running (virtualbox)
k8s-node3 running (virtualbox)
k8s-node4 running (virtualbox)
k8s-node5 running (virtualbox)
admin-lb running (virtualbox)
howoo@ttokkang-ui-MacBookAir ~/ vagrant ssh admin-lb
root@admin-lb:~# sh admin-lb.sh 5
>>>> Initial Config Start <<<<
[TASK 1] Change Timezone and Enable NTP
[TASK 2] Disable firewalld and selinux
[TASK 3] Setting Local DNS Using Hosts file
admin-lb.sh: line 20: ((: i<=: syntax error: operand expected (error token is "<=")
[TASK 4] Delete default routing - enp0s9 NIC
[TASK 5] Install kubectl
.
.
.
+ echo 'sudo su -'
+ echo '>>>> Initial Config End <<<<'
>>>> Initial Config End <<<<
## Ansible로 k8s_node server에 init_cfg.sh 배포
root@admin-lb:~# cat /etc/ansible/hosts
[k8s_nodes]
k8s-node1 ansible_host=192.168.10.11
k8s-node2 ansible_host=192.168.10.12
k8s-node3 ansible_host=192.168.10.13
k8s-node4 ansible_host=192.168.10.14
k8s-node5 ansible_host=192.168.10.15
[k8s_nodes:vars]
ansible_user=root
ssh-copy-id root@192.168.10.11
ssh-copy-id root@192.168.10.12
ssh-copy-id root@192.168.10.13
ssh-copy-id root@192.168.10.14
ssh-copy-id root@192.168.10.15
ansible k8s_nodes -m copy -a "src=/root/init_cfg.sh dest=/root/init_cfg.sh mode=0755"
ansible k8s_nodes -m shell -a "/root/init_cfg.sh"
TASK [Copy init script] ***************************************************************************************************************************************
ok: [k8s-node1]
ok: [k8s-node3]
ok: [k8s-node5]
ok: [k8s-node4]
ok: [k8s-node2]
TASK [Run init script] ****************************************************************************************************************************************
changed: [k8s-node4]
changed: [k8s-node2]
changed: [k8s-node5]
changed: [k8s-node1]
changed: [k8s-node3]
PLAY RECAP ****************************************************************************************************************************************************
k8s-node1 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
k8s-node2 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
k8s-node3 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
k8s-node4 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
k8s-node5 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
# HAProxy 상태 확인
systemctl status haproxy.service --no-pager
journalctl -u haproxy.service --no-pager
**ss -tnlp | grep haproxy**
*LISTEN 0 3000 0.0.0.0:6443 0.0.0.0:* users:(("haproxy",pid=4915,fd=7)) # k8s api loadbalancer
LISTEN 0 3000 0.0.0.0:9000 0.0.0.0:* users:(("haproxy",pid=4915,fd=8)) # haproxy stat dashbaord
LISTEN 0 3000 0.0.0.0:8405 0.0.0.0:* users:(("haproxy",pid=4915,fd=9)) # metrics exporter*
# 통계 페이지 접속
open http://192.168.10.10:9000/haproxy_stats
# (참고) 프로테우스 메트릭 엔드포인트 접속
curl http://192.168.10.10:8405/metrics
.
.
.
haproxy_resolver_truncated{resolver="default",nameserver="0.192.249.101"} 0
haproxy_resolver_truncated{resolver="default",nameserver="8.8.8.8"} 0
# HELP haproxy_resolver_outdated Outdated
# TYPE haproxy_resolver_outdated gauge
haproxy_resolver_outdated{resolver="default",nameserver="0.192.249.101"} 0
haproxy_resolver_outdated{resolver="default",nameserver="8.8.8.8"} 0
- kubespray를 통한 k8s 배포
.
.
.
]
},
"kube_node": {
"hosts": [
"k8s-node4"
]
}
}
root@admin-lb:~/kubespray# ansible-inventory -i /root/kubespray/inventory/mycluster/inventory.ini --graph
@all:
|--@ungrouped:
|--@etcd:
| |--@kube_control_plane:
| | |--k8s-node1
| | |--k8s-node2
| | |--k8s-node3
|--@kube_node:
| |--k8s-node4
root@admin-lb:~/kubespray# sed -i 's|kube_owner: kube|kube_owner: root|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
root@admin-lb:~/kubespray# sed -i 's|kube_network_plugin: calico|kube_network_plugin: flannel|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
root@admin-lb:~/kubespray# sed -i 's|kube_proxy_mode: ipvs|kube_proxy_mode: iptables|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
root@admin-lb:~/kubespray# sed -i 's|enable_nodelocaldns: true|enable_nodelocaldns: false|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
root@admin-lb:~/kubespray# grep -iE 'kube_owner|kube_network_plugin:|kube_proxy_mode|enable_nodelocaldns:' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
kube_owner: root
kube_network_plugin: flannel
kube_proxy_mode: iptables
enable_nodelocaldns: false
root@admin-lb:~/kubespray# echo "enable_dns_autoscaler: false" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
root@admin-lb:~/kubespray# echo "flannel_interface: enp0s9" >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
root@admin-lb:~/kubespray# grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
flannel_interface: enp0s9
root@admin-lb:~/kubespray# sed -i 's|metrics_server_enabled: false|metrics_server_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
root@admin-lb:~/kubespray# grep -iE 'metrics_server_enabled:' inventory/mycluster/group_vars/k8s_cluster/addons.yml
metrics_server_enabled: true
root@admin-lb:~/kubespray# echo "metrics_server_requests_cpu: 25m" >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
root@admin-lb:~/kubespray# echo "metrics_server_requests_memory: 16Mi" >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
root@admin-lb:~/kubespray# cat roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40
kubelet_checksums:
arm64:
1.33.7: sha256:3035c44e0d429946d6b4b66c593d371cf5bbbfc85df39d7e2a03c422e4fe404a
1.33.6: sha256:7d8b7c63309cfe2da2331a1ae13cce070b9ba01e487099e7881a4281667c131d
root@admin-lb:~/kubespray# ANSIBLE_FORCE_COLOR=true ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.32.9" | tee kubespray_install.log
.
.
.
Wednesday 04 February 2026 10:49:23 +0900 (0:00:00.043) 0:08:56.430 ****
===============================================================================
system_packages : Manage packages ------------------------------------- 106.28s
kubernetes/kubeadm : Join to cluster if needed ------------------------- 22.68s
download : Download_file | Download item ------------------------------- 16.14s
download : Download_file | Download item ------------------------------- 15.23s
download : Download_file | Download item ------------------------------- 14.49s
download : Download_file | Download item ------------------------------- 14.25s
container-engine/containerd : Download_file | Download item ------------ 13.63s
download : Download_container | Download image if required ------------- 13.49s
download : Download_container | Download image if required ------------- 13.32s
download : Download_container | Download image if required ------------- 12.83s
download : Download_container | Download image if required ------------- 12.62s
container-engine/crictl : Download_file | Download item ---------------- 11.10s
download : Download_container | Download image if required ------------- 10.18s
container-engine/runc : Download_file | Download item ------------------- 9.95s
download : Download_container | Download image if required -------------- 9.56s
download : Download_container | Download image if required -------------- 9.27s
container-engine/nerdctl : Download_file | Download item ---------------- 9.26s
kubernetes/control-plane : Joining control plane node to the cluster. --- 8.97s
download : Download_file | Download item -------------------------------- 8.25s
download : Download_file | Download item -------------------------------- 7.91s
root@admin-lb:~/kubespray# tree /tmp/
/tmp/
├── k8s-node1
├── k8s-node2
├── k8s-node3
├── k8s-node4
├── k8s-node5
├── k9s_linux_arm64.tar.gz
├── k9s_linux_arm64.tar.gz.1
├── LICENSE
├── README.md
├── systemd-private-0fed2a33a5284513bc7e6d84016170ad-chronyd.service-1oIvH2
│ └── tmp
├── systemd-private-0fed2a33a5284513bc7e6d84016170ad-dbus-broker.service-5V53JE
│ └── tmp
├── systemd-private-0fed2a33a5284513bc7e6d84016170ad-irqbalance.service-Bvj3Gc
│ └── tmp
├── systemd-private-0fed2a33a5284513bc7e6d84016170ad-polkit.service-SBFN4q
│ └── tmp
├── systemd-private-0fed2a33a5284513bc7e6d84016170ad-systemd-logind.service-eYQyW3
│ └── tmp
└── vagrant-shell
11 directories, 10 files
root@admin-lb:~/kubespray# ssh k8s-node1 tree /tmp/releases
/tmp/releases
├── cni-plugins-linux-arm64-1.8.0.tgz
├── containerd-2.1.5-linux-arm64.tar.gz
├── containerd-rootless-setuptool.sh
├── containerd-rootless.sh
├── crictl
├── crictl-1.32.0-linux-arm64.tar.gz
├── etcd-3.5.25-linux-arm64.tar.gz
├── etcd-v3.5.25-linux-arm64
│ ├── Documentation
│ │ ├── dev-guide
│ │ │ └── apispec
│ │ │ └── swagger
│ │ │ ├── rpc.swagger.json
│ │ │ ├── v3election.swagger.json
│ │ │ └── v3lock.swagger.json
│ │ └── README.md
│ ├── etcd
│ ├── etcdctl
│ ├── etcdutl
│ ├── README-etcdctl.md
│ ├── README-etcdutl.md
│ ├── README.md
│ └── READMEv2-etcdctl.md
├── images
├── kubeadm-1.32.9-arm64
├── kubectl-1.32.9-arm64
├── kubelet-1.32.9-arm64
├── nerdctl
├── nerdctl-2.1.6-linux-arm64.tar.gz
└── runc-1.3.4.arm64
7 directories, 24 files
## ETCD backup 확인
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i tree /var/backups; echo; done
>> k8s-node1 <<
/var/backups
└── etcd-2026-02-04_10:55:41
├── member
│ ├── snap
│ │ └── db
│ └── wal
│ └── 0000000000000000-0000000000000000.wal
└── snapshot.db
5 directories, 3 files
>> k8s-node2 <<
/var/backups
└── etcd-2026-02-04_10:55:40
├── member
│ ├── snap
│ │ └── db
│ └── wal
│ └── 0000000000000000-0000000000000000.wal
└── snapshot.db
5 directories, 3 files
>> k8s-node3 <<
/var/backups
└── etcd-2026-02-04_10:55:41
├── member
│ ├── snap
│ │ └── db
│ └── wal
│ └── 0000000000000000-0000000000000000.wal
└── snapshot.db
5 directories, 3 files
## k8s 호출 확인
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; curl -sk https://192.168.10.1$i:6443/version | grep Version; echo; done
>> k8s-node1 <<
"gitVersion": "v1.32.9",
"goVersion": "go1.23.12",
>> k8s-node2 <<
"gitVersion": "v1.32.9",
"goVersion": "go1.23.12",
>> k8s-node3 <<
"gitVersion": "v1.32.9",
"goVersion": "go1.23.12",
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; curl -sk https://k8s-node$i:6443/version | grep Version; echo; done
>> k8s-node1 <<
"gitVersion": "v1.32.9",
"goVersion": "go1.23.12",
>> k8s-node2 <<
"gitVersion": "v1.32.9",
"goVersion": "go1.23.12",
>> k8s-node3 <<
"gitVersion": "v1.32.9",
"goVersion": "go1.23.12",
## k8s 자격증명 확인
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i kubectl cluster-info -v=6; echo; done
>> k8s-node1 <<
I0204 11:00:47.227034 29567 loader.go:402] Config loaded from file: /root/.kube/config
I0204 11:00:47.227368 29567 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0204 11:00:47.227378 29567 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0204 11:00:47.227380 29567 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0204 11:00:47.227381 29567 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0204 11:00:47.235211 29567 round_trippers.go:560] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue 200 OK in 5 milliseconds
Kubernetes control plane is running at https://127.0.0.1:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
>> k8s-node2 <<
I0204 11:00:47.479391 28901 loader.go:402] Config loaded from file: /root/.kube/config
I0204 11:00:47.479691 28901 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0204 11:00:47.479700 28901 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0204 11:00:47.479703 28901 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0204 11:00:47.479704 28901 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0204 11:00:47.483018 28901 round_trippers.go:560] GET https://127.0.0.1:6443/api?timeout=32s 200 OK in 3 milliseconds
I0204 11:00:47.484053 28901 round_trippers.go:560] GET https://127.0.0.1:6443/apis?timeout=32s 200 OK in 0 milliseconds
I0204 11:00:47.490292 28901 round_trippers.go:560] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue 200 OK in 3 milliseconds
Kubernetes control plane is running at https://127.0.0.1:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
>> k8s-node3 <<
I0204 11:00:47.714411 28937 loader.go:402] Config loaded from file: /root/.kube/config
I0204 11:00:47.714828 28937 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0204 11:00:47.714839 28937 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0204 11:00:47.714841 28937 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0204 11:00:47.714843 28937 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0204 11:00:47.721851 28937 round_trippers.go:560] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue 200 OK in 5 milliseconds
Kubernetes control plane is running at https://127.0.0.1:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@admin-lb:~/kubespray# mkdir /root/.kube
root@admin-lb:~/kubespray# scp k8s-node1:/root/.kube/config /root/.kube/
config 100% 5661 6.7MB/s 00:00
root@admin-lb:~/kubespray# cat /root/.kube/config | grep server
server: https://127.0.0.1:6443
root@admin-lb:~/kubespray# ssh k8s-node1 etcdctl.sh member list -w table
+------------------+---------+-------+----------------------------+----------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------+----------------------------+----------------------------+------------+
| 8b0ca30665374b0 | started | etcd3 | https://192.168.10.13:2380 | https://192.168.10.13:2379 | false |
| 2106626b12a4099f | started | etcd2 | https://192.168.10.12:2380 | https://192.168.10.12:2379 | false |
| c6702130d82d740f | started | etcd1 | https://192.168.10.11:2380 | https://192.168.10.11:2379 | false |
+------------------+---------+-------+----------------------------+----------------------------+------------+
root@admin-lb:~/kubespray# for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i etcdctl.sh endpoint status -w table; echo; done
>> k8s-node1 <<
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | c6702130d82d740f | 3.5.25 | 6.3 MB | true | false | 4 | 3103 | 3103 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
>> k8s-node2 <<
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 2106626b12a4099f | 3.5.25 | 6.3 MB | false | false | 4 | 3103 | 3103 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
>> k8s-node3 <<
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 8b0ca30665374b0 | 3.5.25 | 6.3 MB | false | false | 4 | 3103 | 3103 | |
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
- 환경 설정 확인
root@admin-lb:~/kubespray# kubectl get svc -n kube-system coredns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 65m
root@admin-lb:~/kubespray# kubectl get cm -n kube-system kubelet-config -o yaml | grep clusterDNS -A2
clusterDNS:
- 10.233.0.3
clusterDomain: cluster.local
root@admin-lb:~/kubespray# kubectl get svc,ep -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 66m
service/metrics-server ClusterIP 10.233.53.50 <none> 443/TCP 66m
NAME ENDPOINTS AGE
endpoints/coredns 10.233.65.2:53,10.233.66.2:53,10.233.65.2:53 + 3 more... 66m
endpoints/metrics-server 10.233.66.3:10250 66m
root@admin-lb:~/kubespray# ssh k8s-node4 curl -sk https://127.0.0.1:6443/version | grep Version
"gitVersion": "v1.32.9",
"goVersion": "go1.23.12",
root@admin-lb:~/kubespray# ssh k8s-node4 ss -tnlp | grep nginx
LISTEN 0 511 127.0.0.1:6443 0.0.0.0:* users:(("nginx",pid=17935,fd=5),("nginx",pid=17934,fd=5),("nginx",pid=17904,fd=5))
LISTEN 0 511 0.0.0.0:8081 0.0.0.0:* users:(("nginx",pid=17935,fd=6),("nginx",pid=17934,fd=6),("nginx",pid=17904,fd=6))- Control plane node component → k8s api enpoint 분석
root@admin-lb:~/kubespray# kubectl describe pod -n kube-system kube-apiserver-k8s-node1 | grep -E 'address|secure-port'
Annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.10.11:6443
--advertise-address=192.168.10.11
--bind-address=::
--kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP
--secure-port=6443
root@admin-lb:~/kubespray# kubectl get cm -n kube-system kube-proxy -o yaml | grep server
server: https://127.0.0.1:6443
- External LB → HA control plane node ( 3대 ) + Worker Client-Side LoadBalancing )
root@admin-lb:~/kubespray# helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 \
--set service.main.type=NodePort,service.main.ports.http.nodePort=30000 \
--set env.TZ="Asia/Seoul" --namespace kube-system \
--set image.repository="abihf/kube-ops-view" --set image.tag="latest"
NAME: kube-ops-view
LAST DEPLOYED: Wed Feb 4 12:13:20 2026
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace kube-system -o jsonpath="{.spec.ports[0].nodePort}" services kube-ops-view)
export NODE_IP=$(kubectl get nodes --namespace kube-system -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
root@admin-lb:~/kubespray# kubectl get deploy,pod,svc,ep -n kube-system -l app.kubernetes.io/instance=kube-ops-view
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kube-ops-view 1/1 1 1 35s
NAME READY STATUS RESTARTS AGE
pod/kube-ops-view-8484bdc5df-cfb22 1/1 Running 0 35s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-ops-view NodePort 10.233.59.124 <none> 8080:30000/TCP 35s
NAME ENDPOINTS AGE
endpoints/kube-ops-view 10.233.66.4:8080 35s
root@admin-lb:~/kubespray# kubectl get deploy,svc,ep webpod -owide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/webpod 2/2 2 2 77m webpod traefik/whoami app=webpod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/webpod NodePort 10.233.51.23 <none> 80:30003/TCP 77m app=webpod
NAME ENDPOINTS AGE
endpoints/webpod 10.233.66.5:80,10.233.66.6:80 77m
root@admin-lb:~/kubespray# ssh k8s-node1 curl -s webpod -I
HTTP/1.1 200 OK
Date: Wed, 04 Feb 2026 04:49:40 GMT
Content-Length: 202
Content-Type: text/plain; charset=utf-8
root@admin-lb:~/kubespray# ssh k8s-node1 curl -s webpod.default -I
HTTP/1.1 200 OK
Date: Wed, 04 Feb 2026 04:49:59 GMT
Content-Length: 210
Content-Type: text/plain; charset=utf-8- ( 장애 재현 ) 만약 컨트롤 플레인 1번 노드 장애 발생시 영향도
a. Terminal * 4EA로 모니터링
root@k8s-node1:~# poweroff
root@k8s-node1:~# Connection to k8s-node1 closed by remote host.
Connection to k8s-node1 closed.
root@admin-lb:~/kubespray# while true; do kubectl get node ; echo ; curl -sk https://192.168.10.12:6443/version | grep gitVersion ; sleep 1; echo ; done
Unable to connect to the server: dial tcp 192.168.10.11:6443: connect: no route to host
.
.
.
nable to connect to the server: dial tcp 192.168.10.11:6443: connect: no route to host
"gitVersion": "v1.32.9",
root@admin-lb:~/kubespray# ^C
root@admin-lb:~/kubespray# sed -i 's/192.168.10.11/192.168.10.12/g' /root/.kube/config
root@admin-lb:~/kubespray# while true; do kubectl get node ; echo ; curl -sk https://192.168.10.12:6443/version | grep gitVersion ; sleep 1; echo ; done
NAME STATUS ROLES AGE VERSION
k8s-node1 NotReady control-plane 3h6m v1.32.9
k8s-node2 Ready control-plane 3h6m v1.32.9
k8s-node3 Ready control-plane 3h6m v1.32.9
k8s-node4 Ready <none> 3h6m v1.32.9b. VirtualBox에서 k8s-node1 머신 시작
- External LB → HA 컨트롤플레인 노드 ( 3대 ) : k8s apiserver 호출 설정
root@admin-lb:~/kubespray# curl -sk https://192.168.10.10:6443/version | grep gitVersion
"gitVersion": "v1.32.9",
root@admin-lb:~/kubespray# sed -i 's/192.168.10.12/192.168.10.10/g' /root/.kube/config
root@admin-lb:~/kubespray# kubectl get node
E0204 14:08:19.726144 21183 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.10.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.12, 192.168.10.13, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10"
root@admin-lb:~/kubespray# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "control-plane" --list-tasks
Using /root/kubespray/ansible.cfg as config file
[WARNING]: Could not match supplied host pattern, ignoring: bastion
[WARNING]: Could not match supplied host pattern, ignoring: k8s_cluster
[WARNING]: Could not match supplied host pattern, ignoring: calico_rr
[WARNING]: Could not match supplied host pattern, ignoring:
_kubespray_needs_etcd
.
.
.
kubernetes/preinstall : Create kubernetes directories TAGS: [apps, bootstrap_os, control-plane, kube-apiserver, kube-controller-manager, kubelet, network, node, resolvconf]
kubernetes/preinstall : Create other directories of root owner TAGS: [apps, bootstrap_os, control-plane, kube-apiserver, kube-controller-manager, kubelet, network, node, resolvconf]
**ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "control-plane" --limit kube_control_plane -e kube_version="1.32.9"
.
.
.
Gather necessary facts (network) ---------------------------------------- 0.23s
kubernetes/control-plane : Kubeadm | aggregate all SANs ----------------- 0.22s
etcd : Gen_certs | Get etcd certificate serials ------------------------- 0.19s
root@admin-lb:~/kubespray# kubectl get node -v=6
I0204 14:11:19.264116 23169 loader.go:402] Config loaded from file: /root/.kube/config
I0204 14:11:19.264614 23169 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0204 14:11:19.264640 23169 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0204 14:11:19.264684 23169 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0204 14:11:19.264700 23169 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0204 14:11:19.272759 23169 round_trippers.go:560] GET https://192.168.10.10:6443/api/v1/nodes?limit=500 200 OK in 5 milliseconds
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready control-plane 3h15m v1.32.9
k8s-node2 Ready control-plane 3h15m v1.32.9
k8s-node3 Ready control-plane 3h15m v1.32.9
k8s-node4 Ready <none> 3h14m v1.32.9**- 노드 관리
a. 노드 추가
root@admin-lb:~/kubespray# cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3
[etcd:children]
kube_control_plane
[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15
EOF
root@admin-lb:~/kubespray# ansible-inventory -i /root/kubespray/inventory/mycluster/inventory.ini --graph
@all:
|--@ungrouped:
|--@etcd:
| |--@kube_control_plane:
| | |--k8s-node1
| | |--k8s-node2
| | |--k8s-node3
|--@kube_node:
| |--k8s-node4
| |--k8s-node5
root@admin-lb:~/kubespray# ansible -i inventory/mycluster/inventory.ini k8s-node5 -m ping
k8s-node5 | SUCCESS => {
"changed": false,
"ping": "pong"
}
ansible-playbook -i inventory/mycluster/inventory.ini -v scale.yml --list-tasks
.
.
.
kubernetes/preinstall : Flush handlers TAGS: [resolvconf]
kubernetes/preinstall : Check if we are running inside a Azure VM TAGS: [bootstrap_os, resolvconf]
Run calico checks TAGS: [resolvconf]
ANSIBLE_FORCE_COLOR=true **ansible-playbook -i inventory/mycluster/inventory.ini -v scale.yml --limit=k8s-node5 -e kube_version="1.32.9" | tee kubespray_add_worker_node.log
.
.
.
download : Extract_file | Unpacking archive ----------------------------- 1.50s
container-engine/nerdctl : Extract_file | Unpacking archive ------------- 1.38s
network_plugin/cni : CNI | Copy cni plugins ----------------------------- 1.21s
root@admin-lb:~/kubespray# kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node1 Ready control-plane 3h43m v1.32.9 192.168.10.11 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node2 Ready control-plane 3h42m v1.32.9 192.168.10.12 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node3 Ready control-plane 3h42m v1.32.9 192.168.10.13 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node4 Ready <none> 3h42m v1.32.9 192.168.10.14 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node5 Ready <none> 46s v1.32.9 192.168.10.15 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
root@admin-lb:~/kubespray# kubectl get pod -n kube-system -owide | grep k8s-node5
kube-flannel-ds-arm64-rsd9r 1/1 Running 2 (63s ago) 68s 192.168.10.15 k8s-node5 <none> <none>
kube-proxy-q4clq 1/1 Running 0 68s 192.168.10.15 k8s-node5 <none> <none>
nginx-proxy-k8s-node5 1/1 Running 0 67s 192.168.10.15 k8s-node5 <none> <none>
root@admin-lb:~/kubespray# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
webpod-697b545f57-bhhvl 1/1 Running 0 130m 10.233.66.6 k8s-node4 <none> <none>
webpod-697b545f57-hdq76 1/1 Running 0 130m 10.233.66.5 k8s-node4 <none> <none>
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 1
deployment.apps/webpod scaled
root@admin-lb:~/kubespray# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
webpod-697b545f57-hdq76 1/1 Running 0 130m 10.233.66.5 k8s-node4 <none> <none>
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 2
deployment.apps/webpod scaled
root@admin-lb:~/kubespray# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
webpod-697b545f57-2fdql 0/1 ContainerCreating 0 2s <none> k8s-node5 <none> <none>
webpod-697b545f57-hdq76 1/1 Running 0 130m 10.233.66.5 k8s-node4 <none> <none>**b. 노드 제거
root@admin-lb:~/kubespray# cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: webpod
namespace: default
spec:
maxUnavailable: 0
selector:
matchLabels:
app: webpod
EOF
poddisruptionbudget.policy/webpod created
root@admin-lb:~/kubespray# kubectl get pdb
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
webpod N/A 0 0 4s
ansible-playbook -i inventory/mycluster/inventory.ini -v remove-node.yml --list-tags
**ansible-playbook -i inventory/mycluster/inventory.ini -v remove-node.yml -e node=k8s-node5
cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini**
*[**kube_control_plane**]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3
[**etcd**:children]
kube_control_plane
[**kube_node**]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
**k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15***
EOF
****ANSIBLE_FORCE_COLOR=true **ansible-playbook -i inventory/mycluster/inventory.ini -v scale.yml --limit=k8s-node5 -e kube_version="1.32.9" | tee kubespray_add_worker_node.log
root@admin-lb:~/kubespray# kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node1 Ready control-plane 4h51m v1.32.9 192.168.10.11 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node2 Ready control-plane 4h51m v1.32.9 192.168.10.12 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node3 Ready control-plane 4h51m v1.32.9 192.168.10.13 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node4 Ready <none> 4h51m v1.32.9 192.168.10.14 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node5 Ready,SchedulingDisabled <none> 69m v1.32.9 192.168.10.15 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 1
deployment.apps/webpod scaled
root@admin-lb:~/kubespray# kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node1 Ready control-plane 4h52m v1.32.9 192.168.10.11 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node2 Ready control-plane 4h52m v1.32.9 192.168.10.12 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node3 Ready control-plane 4h52m v1.32.9 192.168.10.13 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node4 Ready <none> 4h52m v1.32.9 192.168.10.14 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node5 Ready,SchedulingDisabled <none> 70m v1.32.9 192.168.10.15 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
root@admin-lb:~/kubespray# kubectl scale deployment webpod --replicas 2
deployment.apps/webpod scaled**- 모니터링 설정
root@admin-lb:~/kubespray# kubectl create ns nfs-provisioner
namespace/nfs-provisioner created
root@admin-lb:~/kubespray# helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
"nfs-subdir-external-provisioner" has been added to your repositories
root@admin-lb:~/kubespray# helm install nfs-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner -n nfs-provisioner \
--set nfs.server=192.168.10.10 \
--set nfs.path=/srv/nfs/share \
--set storageClass.defaultClass=true
NAME: nfs-provisioner
LAST DEPLOYED: Wed Feb 4 17:31:58 2026
NAMESPACE: nfs-provisioner
STATUS: deployed
REVISION: 1
TEST SUITE: None
root@admin-lb:~/kubespray# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-client (default) cluster.local/nfs-provisioner-nfs-subdir-external-provisioner Delete Immediate true 43s
root@admin-lb:~/kubespray# kubectl get pod -n nfs-provisioner -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nfs-provisioner-nfs-subdir-external-provisioner-b549b9dff-l25n2 1/1 Running 0 79s 10.233.66.8 k8s-node4 <none> <none>
root@admin-lb:~/kubespray# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
root@admin-lb:~/kubespray# cat <<EOT > monitor-values.yaml
prometheus:
prometheusSpec:
scrapeInterval: "20s"
evaluationInterval: "20s"
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
additionalScrapeConfigs:
- job_name: 'haproxy-metrics'
static_configs:
- targets:
- '192.168.10.10:8405'
externalLabels:
cluster: "myk8s-cluster"
service:
type: NodePort
nodePort: 30001
grafana:
defaultDashboardsTimezone: Asia/Seoul
adminPassword: prom-operator
service:
type: NodePort
nodePort: 30002
alertmanager:
enabled: false
defaultRules:
create: false
kubeProxy:
enabled: false
prometheus-windows-exporter:
prometheus:
monitor:
enabled: false
EOT
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version **80.13.3** \
-f **monitor-values.yaml** --create-namespace --namespace **monitoring**
root@admin-lb:~/kubespray# kubectl exec -it sts/prometheus-kube-prometheus-stack-prometheus -n monitoring -c prometheus -- prometheus --version
prometheus, version 3.9.1 (branch: HEAD, revision: 9ec59baffb547e24f1468a53eb82901e58feabd8)
build user: root@61c3a9212c9e
build date: 20260107-16:11:39
go version: go1.25.5
platform: linux/arm64
tags: netgo,builtinassets
root@admin-lb:~/kubespray# sed -i -e 's/${DS_PROMETHEUS}/prometheus/g' 12693_rev12.json
root@admin-lb:~/kubespray# sed -i -e 's/${DS__VICTORIAMETRICS-PROD-ALL}/prometheus/g' 15661_rev2.json
root@admin-lb:~/kubespray# sed -i -e 's/${DS_PROMETHEUS}/prometheus/g' k8s-system-api-server.json
root@admin-lb:~/kubespray# kubectl create configmap my-dashboard --from-file=12693_rev12.json --from-file=15661_rev2.json --from-file=k8s-system-api-server.json -n monitoring
configmap/my-dashboard created
root@admin-lb:~/kubespray# kubectl label configmap my-dashboard grafana_dashboard="1" -n monitoring
configmap/my-dashboard labeled
root@admin-lb:~/kubespray# kubectl exec -it -n monitoring deploy/kube-prometheus-stack-grafana -- ls -l /tmp/dashboards
error: cannot exec into a container in a completed pod; current phase is Succeeded
root@admin-lb:~/kubespray# cat << EOF >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
etcd_metrics: true
etcd_listen_metrics_urls: "http://0.0.0.0:2381"
EOF
ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "etcd" --list-tasks
**ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "etcd" --limit etcd -e kube_version="1.32.9"**
root@admin-lb:~/kubespray# ssh k8s-node1 etcdctl.sh member list -w table
+------------------+---------+-------+----------------------------+----------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------+----------------------------+----------------------------+------------+
| 8b0ca30665374b0 | started | etcd3 | https://192.168.10.13:2380 | https://192.168.10.13:2379 | false |
| 2106626b12a4099f | started | etcd2 | https://192.168.10.12:2380 | https://192.168.10.12:2379 | false |
| c6702130d82d740f | started | etcd1 | https://192.168.10.11:2380 | https://192.168.10.11:2379 | false |
+------------------+---------+-------+----------------------------+----------------------------+------------+
root@admin-lb:~/kubespray# helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 80.13.3 \
--reuse-values -f monitor-add-values.yaml --namespace monitoring
root@admin-lb:~/kubespray# helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 80.13.3 \
--reuse-values -f monitor-add-values.yaml --namespace monitoring
Release "kube-prometheus-stack" has been upgraded. Happy Helming!
NAME: kube-prometheus-stack
LAST DEPLOYED: Wed Feb 4 17:45:41 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 2
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace monitoring get pods -l "release=kube-prometheus-stack"
Get Grafana 'admin' user password by running:
kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo
Access Grafana local instance:
export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kube-prometheus-stack" -oname)
kubectl --namespace monitoring port-forward $POD_NAME 3000
Get your grafana admin user password by running:
kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.- 업그레이드
# 관련 변수 검색
grep -Rni "flannel" inventory/mycluster/ playbooks/ roles/ --include="*.yml" -A2 -B1
.
.
.
roles/validate_inventory/tasks/main.yml-169- that:
roles/validate_inventory/tasks/main.yml:170: - kube_network_plugin in ['calico', 'flannel', 'cloud', 'cilium', 'cni', 'kube-ovn', 'kube-router', 'macvlan', 'custom_cni', 'none']
roles/validate_inventory/tasks/main.yml-171- - dns_mode in ['coredns', 'coredns_dual', 'manual', 'none']
roles/validate_inventory/tasks/main.yml-172- - kube_proxy_mode in ['iptables', 'ipvs', 'nftables']
root@admin-lb:~/kubespray# kubectl get ds -n kube-system -owide
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
kube-flannel 0 0 0 0 0 <none> 22h kube-flannel docker.io/flannel/flannel:v0.27.3 app=flannel
kube-flannel-ds-arm 0 0 0 0 0 <none> 22h kube-flannel docker.io/flannel/flannel:v0.27.3 app=flannel
kube-flannel-ds-arm64 5 5 5 5 5 <none> 22h kube-flannel docker.io/flannel/flannel:v0.27.3 app=flannel
kube-flannel-ds-ppc64le 0 0 0 0 0 <none> 22h kube-flannel docker.io/flannel/flannel:v0.27.3 app=flannel
kube-flannel-ds-s390x 0 0 0 0 0 <none> 22h kube-flannel docker.io/flannel/flannel:v0.27.3 app=flannel
kube-proxy 5 5 5 5 5 kubernetes.io/os=linux 22h kube-proxy registry.k8s.io/kube-proxy:v1.32.9 k8s-app=kube-proxy
root@admin-lb:~/kubespray# cat << EOF >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
flannel_version: 0.27.4
EOF
root@admin-lb:~/kubespray# grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
flannel_interface: enp0s9
flannel_version: 0.27.4
ansible-playbook -i inventory/mycluster/inventory.ini -v upgrade-cluster.yml --tags "flannel" --list-tasks
**ansible-playbook -i inventory/mycluster/inventory.ini -v upgrade-cluster.yml --tags "flannel" --limit k8s-node3 -e kube_version="1.32.9"**
**ansible-playbook -i inventory/mycluster/inventory.ini -v upgrade-cluster.yml --tags "flannel" -e kube_version="1.32.9"
root@admin-lb:~/kubespray# kubectl get ds -n kube-system -owide
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
kube-flannel 0 0 0 0 0 <none> 28h kube-flannel docker.io/flannel/flannel:v0.27.4 app=flannel
kube-flannel-ds-arm 0 0 0 0 0 <none> 28h kube-flannel docker.io/flannel/flannel:v0.27.4 app=flannel
kube-flannel-ds-arm64 5 5 4 4 4 <none> 28h kube-flannel docker.io/flannel/flannel:v0.27.4 app=flannel
kube-flannel-ds-ppc64le 0 0 0 0 0 <none> 28h kube-flannel docker.io/flannel/flannel:v0.27.4 app=flannel
kube-flannel-ds-s390x 0 0 0 0 0 <none> 28h kube-flannel docker.io/flannel/flannel:v0.27.4 app=flannel
kube-proxy 5 5 5 5 5 kubernetes.io/os=linux 28h kube-proxy registry.k8s.io/kube-proxy:v1.32.9 k8s-app=kube-proxy
root@admin-lb:~/kubespray# kubectl get pod -n kube-system -l app=flannel -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel-ds-arm64-5v4wx 1/1 Running 0 71s 192.168.10.11 k8s-node1 <none> <none>
kube-flannel-ds-arm64-62btm 1/1 Running 0 94s 192.168.10.12 k8s-node2 <none> <none>
kube-flannel-ds-arm64-6xvwt 1/1 Running 0 61s 192.168.10.13 k8s-node3 <none> <none>
kube-flannel-ds-arm64-mqpgb 1/1 Running 0 106s 192.168.10.15 k8s-node5 <none> <none>
kube-flannel-ds-arm64-xlkdc 1/1 Running 0 83s 192.168.10.14 k8s-node4 <none> <none>**
# limit 사용 전 모든 노드의 facts 캐시 최신화
ansible-playbook playbooks/facts.yml -b -i inventory/sample/hosts.ini
# 컨트롤 플레인
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=1.20.7 --limit "kube_control_plane:etcd"
# 워커 노드
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=1.20.7 --limit "node4:node6:node7:node12"
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=1.20.7 --limit "node5*"
# 워컨 노드 1번에 1대의 노드만 업그레이드
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=1.20.7 **-e "serial=1"**10-1. k8s upgrade : 1.32.9 → 1.32.10
ANSIBLE_FORCE_COLOR=true **ansible-playbook -i inventory/mycluster/inventory.ini -v upgrade-cluster.yml -e kube_version="1.32.10" --limit "kube_control_plane:etcd"** **| tee kubespray_upgrade.log
.
.
.
download : Download_file | Download item -------------------------------- 7.85s
network_plugin/cni : CNI | Copy cni plugins ----------------------------- 7.61s
etcd : Gen_certs | Write etcd member/admin and kube_control_plane client certs to other etcd nodes --- 6.91s
network_plugin/cni : CNI | Copy cni plugins ----------------------------- 6.61s
system_packages : Manage packages --------------------------------------- 6.53s
container-engine/containerd : Containerd | Unpack containerd archive ---- 5.73s
download : Prep_kubeadm_images | Copy kubeadm binary from download dir to system path --- 4.90s
root@admin-lb:~/kubespray# kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node1 Ready control-plane 29h v1.32.10 192.168.10.11 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node2 Ready control-plane 29h v1.32.10 192.168.10.12 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node3 Ready control-plane 29h v1.32.10 192.168.10.13 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node4 Ready <none> 29h v1.32.9 192.168.10.14 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5
k8s-node5 Ready,SchedulingDisabled <none> 25h v1.32.9 192.168.10.15 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.39.1.el10_0.aarch64 containerd://2.1.5**
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=1.20.7 **-e "serial=1"
# wk upgrade 2분 + 2분 소요 : (최초 한번) kube-proxy ds all node 재기동??? 1.32.9 → 1.32.10
ansible-playbook -i inventory/mycluster/inventory.ini -v upgrade-cluster.yml -e kube_version="1.32.10" --limit "k8s-node5"**
# 확인 후 나머지 노드 실행
**ansible-playbook -i inventory/mycluster/inventory.ini -v upgrade-cluster.yml -e kube_version="1.32.10" --limit "k8s-node4"
TASK [upgrade/pre-upgrade : Cordon node] ***************************************
changed: [k8s-node4 -> k8s-node1(192.168.10.11)] => {"changed": true, "cmd": ["/usr/local/bin/kubectl", "--kubeconfig", "/etc/kubernetes/admin.conf", "cordon", "k8s-node4"], "delta": "0:00:00.073049", "end": "2026-02-05 16:14:18.040359", "msg": "", "rc": 0, "start": "2026-02-05 16:14:17.967310", "stderr": "", "stderr_lines": [], "stdout": "node/k8s-node4 cordoned", "stdout_lines": ["node/k8s-node4 cordoned"]}
Thursday 05 February 2026 15:55:32 +0900 (0:00:00.245) 0:01:00.392 *****
.
.
.**'Study > K8S-Deploy' 카테고리의 다른 글
| K8S ) 7주차 과제 (0) | 2026.02.18 |
|---|---|
| K8S ) 6주차 과제 (1) | 2026.02.13 |
| K8S) 4주차 과제 (0) | 2026.01.31 |
| K8S)3주차 과제 (0) | 2026.01.24 |
| K8S)2주차 과제 (0) | 2026.01.15 |