Study/K8S-Deploy
K8S ) 6주차 과제
zosys
2026. 2. 13. 23:51
Kubespray offline 설치
폐쇄망환경에서 정상적인 서비스 제공을 위한 주요 구성 요소
- NTP 서버 : Kubernetes는 인증서(TLS), etcd, 로그, 토큰 만료 시간 등에서 시간 동기화가 매우 중요하기 때문에 모든 노드 간 동일한 시간 유지가 필수
- DNS 서버 : Kubernetes 노드 간 통신과 내부 서비스 접근은 FQDN 기반으로 동작하는 경우가 많기 때문에 안정적인 내부 이름 해석 환경이 필요
- Network Gateway (IGW/NATGW) : 완전 폐쇄망이 아닌 경우 외부 리소스 접근 또는 DMZ 구간과의 통신을 위해 네트워크 경계 및 트래픽 제어 구성이 필요
- Local (Mirror) YUM/DNF Repository : Kubespray 설치 과정에서 필요한 OS 패키지(container runtime, iptables, socat 등)를 외부 인터넷 없이 설치하기 위해 내부 패키지 저장소가 필요
- Private Container Image Registry : Kubernetes 구성요소(kube-apiserver, etcd, coredns, CNI 등)와 애플리케이션 이미지를 폐쇄망 환경에서 Pull 하기 위해 내부 컨테이너 이미지 저장소가 필요
- Helm Artifact Repository : Kubernetes 배포 이후 Helm 기반 애플리케이션(ingress, monitoring, logging 등)을 외부 Helm 저장소 없이 배포하기 위해 내부 Helm 차트 저장소가 필요
- Private PyPI Mirror : Kubespray(Ansible 기반) 및 Python 애플리케이션의 의존 패키지를 폐쇄망 환경에서 설치하기 위해 내부 Python 패키지 저장소가 필요
- Private Go Module Proxy : Go 기반 애플리케이션 및 Kubernetes 확장 개발 시 외부 Go 모듈 의존성을 내부에서 관리하기 위해 사설 Go 모듈 프록시가 필요
graph TB
subgraph InternalServices["내부망 내 필요한 서비스"]
NTP["⏱️ NTP Server"]
DNS["🌐 DNS Server"]
GW["🛜 Network Gateway<br/>(IGW, NATGW)"]
YUM["📦 Local (Mirror) YUM/DNF Repository"]
REG["🐳 Private Container (Image) Registry"]
HELM["🧭 Helm Artifact Repository"]
PYPI["🐍 Private PyPI Mirror"]
end폐쇄망 실습환경구성
NAT GATEWAY
설정 확인
실습내에 enp0s8 ⇒ 실인터페이스 : enp0s3 enp0s9⇒실인터페이스enp0s8
root@admin:~# cat /etc/NetworkManager/system-connections/enp0s3.nmconnection [connection] id=enp0s3 uuid=74d64498-c95a-435b-9340-81b2e4a7d1f2 type=ethernet interface-name=enp0s3 [ethernet] [ipv4] method=auto [ipv6] addr-gen-mode=eui64 method=ignore [proxy] root@admin:~# cat /etc/NetworkManager/system-connections/enp0s8.nmconnection [connection] id=enp0s8 uuid=60dff4a4-af47-4907-84b1-dc4b1e9a129a type=ethernet autoconnect-priority=-100 autoconnect-retries=1 interface-name=enp0s8 [ethernet] mac-address=08:00:27:6d:47:bb [ipv4] method=manual addresses=192.168.10.10/24 gateway=192.168.10.1 [user] org.freedesktop.NetworkManager.origin=vagrant네트워크 기본 설정 : enp0s3 연결 down, enp0s8 디폴트 라우팅
root@k8s-node1:~# nmcli connection down enp0s3 Connection 'enp0s3' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2) root@k8s-node1:~# nmcli connection modify enp0s3 connection.autoconnect no root@k8s-node1:~# nmcli connection modify enp0s8 +ipv4.routes "0.0.0.0/0 192.168.10.10 200" root@k8s-node1:~# nmcli connection up enp0s8 Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5) root@k8s-node1:~# ip route default via 192.168.10.1 dev enp0s8 proto static metric 100 default via 192.168.10.10 dev enp0s8 proto static metric 200 192.168.10.0/24 dev enp0s8 proto kernel scope link src 192.168.10.11 metric 100 root@k8s-node1:~# ping -w 1 -W 1 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. ^C --- 8.8.8.8 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms root@k8s-node1:~# curl www.google.com curl: (6) Could not resolve host: www.google.com root@k8s-node1:~# cat /etc/resolv.conf # Generated by NetworkManager root@k8s-node1:~# cat << EOF > /etc/resolv.conf nameserver 168.126.63.1 nameserver 8.8.8.8 EOF root@k8s-node1:~# curl www.google.com
NTP 설정
[admin]NTP 서버설정
root@admin:~# systemctl status chronyd.service --no-pager ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled) Active: active (running) since Fri 2026-02-13 21:33:01 KST; 14min ago Invocation: d599d3cbe51f4af0b3582d164c7e7afd Docs: man:chronyd(8) man:chrony.conf(5) Main PID: 753 (chronyd) Tasks: 1 (limit: 12339) Memory: 4.8M (peak: 5.5M) CPU: 110ms CGroup: /system.slice/chronyd.service └─753 /usr/sbin/chronyd -F 2 Feb 13 21:33:23 admin chronyd[753]: Source 2001:678:8::123 offline Feb 13 21:33:23 admin chronyd[753]: Source 240b:400d:3:3300:aeda:71da:9779:d8f1 offline Feb 13 21:33:23 admin chronyd[753]: Source 240b:400d:3:3300:aeda:71da:9779:d4f1 offline Feb 13 21:33:23 admin chronyd[753]: Source 2401:c080:1c00:24a1:5400:5ff:fe04:720 offline Feb 13 21:33:26 admin chronyd[753]: Source 2001:678:8::123 online Feb 13 21:33:26 admin chronyd[753]: Source 240b:400d:3:3300:aeda:71da:9779:d8f1 online Feb 13 21:33:26 admin chronyd[753]: Source 240b:400d:3:3300:aeda:71da:9779:d4f1 online Feb 13 21:33:26 admin chronyd[753]: Source 2401:c080:1c00:24a1:5400:5ff:fe04:720 online Feb 13 21:34:17 admin chronyd[753]: Selected source 175.195.167.194 (2.rocky.pool.ntp.org) Feb 13 21:34:54 admin chronyd[753]: Selected source 221.151.118.78 (2.rocky.pool.ntp.org) root@admin:~# grep "^[^#]" /etc/chrony.conf pool 2.rocky.pool.ntp.org iburst sourcedir /run/chrony-dhcp driftfile /var/lib/chrony/drift makestep 1.0 3 rtcsync ntsdumpdir /var/lib/chrony logdir /var/log/chrony root@admin:~# chronyc sources -v .-- Source mode '^' = server, '=' = peer, '#' = local clock. / .- Source state '*' = current best, '+' = combined, '-' = not combined, | / 'x' = may be in error, '~' = too variable, '?' = unusable. || .- xxxx [ yyyy ] +/- zzzz || Reachability register (octal) -. | xxxx = adjusted offset, || Log2(Polling interval) --. | | yyyy = measured offset, || \ | | zzzz = estimated error. || | | \ MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^- 175.210.18.47 2 6 377 7 -562us[ -562us] +/- 17ms ^- mail.innotab.com 3 6 377 8 -1275us[-1275us] +/- 38ms ^* time.ravnus.com 2 6 377 33 -73us[+1359ns] +/- 2687us ^+ ec2-3-39-176-65.ap-north> 2 6 377 7 +157us[ +157us] +/- 5593us root@admin:~# dig +short 2.rocky.pool.ntp.org 175.195.167.194 158.247.202.103 221.151.118.78 root@admin:~# cp /etc/chrony.conf /etc/chrony.bak root@admin:~# cat << EOF > /etc/chrony.conf # 외부 한국 공용 NTP 서버 설정 server pool.ntp.org iburst server kr.pool.ntp.org iburst # 내부망(192.168.10.0/24)에서 이 서버에 접속하여 시간 동기화 허용 allow 192.168.10.0/24 # 외부망이 끊겼을 때도 로컬 시계를 기준으로 내부망에 시간 제공 (선택 사항) local stratum 10 # 로그 logdir /var/log/chrony EOF root@admin:~# systemctl restart chronyd.service root@admin:~# systemctl status chronyd.service --no-pager ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled) Active: active (running) since Fri 2026-02-13 21:47:46 KST; 3s ago Invocation: ed143eef1e0145ba87bd062f11301cad Docs: man:chronyd(8) man:chrony.conf(5) Process: 5758 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 5760 (chronyd) Tasks: 1 (limit: 12339) Memory: 1.1M (peak: 2.9M) CPU: 34ms CGroup: /system.slice/chronyd.service └─5760 /usr/sbin/chronyd -F 2 Feb 13 21:47:46 admin systemd[1]: Starting chronyd.service - NTP client/server... Feb 13 21:47:46 admin chronyd[5760]: chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCF… +DEBUG) Feb 13 21:47:46 admin chronyd[5760]: Initial frequency -541.138 ppm Feb 13 21:47:46 admin chronyd[5760]: Loaded seccomp filter (level 2) Feb 13 21:47:46 admin systemd[1]: Started chronyd.service - NTP client/server. Hint: Some lines were ellipsized, use -l to show in full. root@admin:~# timedatectl status Local time: Fri 2026-02-13 21:47:52 KST Universal time: Fri 2026-02-13 12:47:52 UTC RTC time: Fri 2026-02-13 12:47:51 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@admin:~# chronyc sources -v .-- Source mode '^' = server, '=' = peer, '#' = local clock. / .- Source state '*' = current best, '+' = combined, '-' = not combined, | / 'x' = may be in error, '~' = too variable, '?' = unusable. || .- xxxx [ yyyy ] +/- zzzz || Reachability register (octal) -. | xxxx = adjusted offset, || Log2(Polling interval) --. | | yyyy = measured offset, || \ | | zzzz = estimated error. || | | \ MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^- 121.134.215.104 2 6 17 3 -94us[ -94us] +/- 2838us ^* 211.108.117.211 2 6 17 4 -11us[+1053us] +/- 2114us[k8snode] ntp client 설정
root@k8s-node1:~# timedatectl status Local time: Fri 2026-02-13 21:50:15 KST Universal time: Fri 2026-02-13 12:50:15 UTC RTC time: Fri 2026-02-13 12:50:15 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@k8s-node1:~# chronyc sources -v .-- Source mode '^' = server, '=' = peer, '#' = local clock. / .- Source state '*' = current best, '+' = combined, '-' = not combined, | / 'x' = may be in error, '~' = too variable, '?' = unusable. || .- xxxx [ yyyy ] +/- zzzz || Reachability register (octal) -. | xxxx = adjusted offset, || Log2(Polling interval) --. | | yyyy = measured offset, || \ | | zzzz = estimated error. || | | \ MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^- 121.174.142.82 3 7 360 444 +988us[ +988us] +/- 43ms ^- 175.195.167.194 3 7 34 509 -309us[ -360us] +/- 35ms ^- 121.174.142.81 3 7 340 484 -132us[ -132us] +/- 43ms ^* 211.108.117.211 2 6 300 504 -264us[ -316us] +/- 2203us root@k8s-node1:~# root@k8s-node1:~# root@k8s-node1:~# cp /etc/chrony.conf /etc/chrony.bak root@k8s-node1:~# cat << EOF > /etc/chrony.conf server 192.168.10.10 iburst logdir /var/log/chrony EOF root@k8s-node1:~# systemctl restart chronyd.service root@k8s-node1:~# systemctl status chronyd.service --no-pager ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled) Active: active (running) since Fri 2026-02-13 21:52:50 KST; 1s ago Invocation: 41174c2716074722b33c8f5c506d64c5 Docs: man:chronyd(8) man:chrony.conf(5) Process: 5662 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 5665 (chronyd) Tasks: 1 (limit: 12339) Memory: 892K (peak: 2.9M) CPU: 30ms CGroup: /system.slice/chronyd.service └─5665 /usr/sbin/chronyd -F 2 Feb 13 21:52:50 k8s-node1 systemd[1]: Starting chronyd.service - NTP client/server... Feb 13 21:52:50 k8s-node1 chronyd[5665]: chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP … +DEBUG) Feb 13 21:52:50 k8s-node1 chronyd[5665]: Initial frequency -540.965 ppm Feb 13 21:52:50 k8s-node1 chronyd[5665]: Loaded seccomp filter (level 2) Feb 13 21:52:50 k8s-node1 systemd[1]: Started chronyd.service - NTP client/server. Hint: Some lines were ellipsized, use -l to show in full. root@k8s-node1:~# timedatectl status Local time: Fri 2026-02-13 21:52:55 KST Universal time: Fri 2026-02-13 12:52:55 UTC RTC time: Fri 2026-02-13 12:52:54 Time zone: Asia/Seoul (KST, +0900) System clock synchronized: yes NTP service: active RTC in local TZ: no root@k8s-node1:~# chronyc sources -v .-- Source mode '^' = server, '=' = peer, '#' = local clock. / .- Source state '*' = current best, '+' = combined, '-' = not combined, | / 'x' = may be in error, '~' = too variable, '?' = unusable. || .- xxxx [ yyyy ] +/- zzzz || Reachability register (octal) -. | xxxx = adjusted offset, || Log2(Polling interval) --. | | yyyy = measured offset, || \ | | zzzz = estimated error. || | | \ MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* admin 3 6 17 1 -64us[-2630ns] +/- 5878us ##어드민 상태확인 root@admin:~# chronyc clients Hostname NTP Drop Int IntL Last Cmd Drop Int Last =============================================================================== k8s-node2 6 0 5 - 4 0 0 - - k8s-node1 4 0 1 - 4 0 0 - -
DNS설정
[admin] DNS 서버(bind) 설정
root@admin:~# dnf install -y bind bind-utils Last metadata expiration check: 0:22:21 ago on Fri 13 Feb 2026 09:33:32 PM KST. Package bind-utils-32:9.18.33-4.el10_0.x86_64 is already installed. Dependencies resolved. ======================================================================================================================== Package Architecture Version Repository Size ======================================================================================================================== Installing: bind x86_64 32:9.18.33-10.el10_1.2 appstream 333 k Upgrading: bind-libs x86_64 32:9.18.33-10.el10_1.2 appstream 1.3 M bind-license noarch 32:9.18.33-10.el10_1.2 appstream 13 k bind-utils x86_64 32:9.18.33-10.el10_1.2 appstream 225 k crypto-policies noarch 20250905-2.gitc7eb7b2.el10_1.1 baseos 94 k crypto-policies-scripts noarch 20250905-2.gitc7eb7b2.el10_1.1 baseos 134 k openssl x86_64 1:3.5.1-7.el10_1 baseos 1.2 M openssl-libs x86_64 1:3.5.1-7.el10_1 baseos 2.2 M Installing dependencies: openssl-fips-provider x86_64 1:3.5.1-7.el10_1 baseos 812 k Installing weak dependencies: bind-dnssec-utils x86_64 32:9.18.33-10.el10_1.2 appstream 151 k Transaction Summary ======================================================================================================================== Install 3 Packages root@admin:~# named-checkconf /etc/named.conf root@admin:~# systemctl enable --now named Created symlink '/etc/systemd/system/multi-user.target.wants/named.service' → '/usr/lib/systemd/system/named.service'. root@admin:~# root@admin:~# echo "nameserver 192.168.10.10" > /etc/resolv.conf root@admin:~# dig +short google.com @192.168.10.10 172.217.213.139 172.217.213.113 172.217.213.138 172.217.213.100 172.217.213.102 172.217.213.101[k8snode] DNS 클라이언트 설정 : NetworkManager DNS 관리 종료
root@k8s-node1:~# cat /etc/NetworkManager/conf.d/99-dns-none.conf cat: /etc/NetworkManager/conf.d/99-dns-none.conf: No such file or directory root@k8s-node1:~# cat << EOF > /etc/NetworkManager/conf.d/99-dns-none.conf [main] dns=none EOF root@k8s-node1:~# systemctl restart NetworkManager root@k8s-node1:~# echo "nameserver 192.168.10.10" > /etc/resolv.conf root@k8s-node1:~# dig +short google.com @192.168.10.10 172.217.213.138 172.217.213.113 172.217.213.101 172.217.213.102 172.217.213.100 172.217.213.139
Kubespray Offline 설치소개
Kubespray : Ansible 기반 Kubernetes 배포 도구로, 오프라인(Air-Gap) 환경에서 Kubernetes를 설치할 수 있도록 사전 다운로드·미러링·이미지 관리 편의 기능을 지원
1️⃣ 오프라인 배포 준비 기능
- Kubernetes 설치 시 필요한 바이너리 파일 목록 및 컨테이너 이미지 목록 자동 생성
- 필요한 컨테이너 이미지 사전 다운로드
- 다운로드한 이미지를 내부 프라이빗 레지스트리에 업로드(등록)
- 필수 파일(바이너리 등) 다운로드 후 Nginx 컨테이너를 통해 내부 HTTP 파일 서버 제공
- 외부 인터넷이 수행하는 다운로드 역할을 내부 환경에서 대체하도록 지원
2️⃣ Kubespray Offline 동작 흐름
- 다운로드 대상 파일 및 이미지 목록 생성
- 외부망에서 모든 바이너리·이미지 사전 다운로드
- 내부 Registry 및 Web 서버 구성
- 내부 노드들은 외부가 아닌 내부 미러/레지스트리에서만 다운로드
- Kubernetes 클러스터 배포 수행
기본환경준비
download.sh 수행 및 용량확인
root@admin:~/kubespray-offline# ./download-all.sh ###중략########## root@admin:~/kubespray-offline# du -sh ~/.venv 491M /root/.venv root@admin:~/kubespray-offline# tree ~/.venv | more /root/.venv └── 3.12 ├── bin │ ├── activate │ ├── activate.csh │ ├── activate.fish │ ├── Activate.ps1 │ ├── ansible │ ├── ansible-community │ ├── ansible-config │ ├── ansible-connection │ ├── ansible-console │ ├── ansible-doc │ ├── ansible-galaxy │ ├── ansible-inventory │ ├── ansible-playbook │ ├── ansible-pull root@admin:~/kubespray-offline# tree /root/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/ /root/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/ ├── docker-daemon.json ├── generate_list.sh ├── generate_list.yml ├── manage-offline-container-images.sh ├── manage-offline-files.sh ├── nginx.conf ├── README.md ├── registries.conf ├── temp │ ├── files.list │ ├── files.list.template │ ├── images.list │ └── images.list.template └── upload2artifactory.py 2 directories, 13 files root@admin:~/kubespray-offline# du -sh /root/kubespray-offline/outputs/ 3.7G /root/kubespray-offline/outputs/[1] outputs 디렉터리 이동 후 setup-container.sh 실행 : 추가로 install-containerd.sh 실행됨
root@admin:~/kubespray-offline# cd outputs/ root@admin:~/kubespray-offline/outputs# ./setup-container.sh ==> Install runc ==> Install nerdctl nerdctl containerd-rootless-setuptool.sh containerd-rootless.sh ==> Install containerd bin/containerd-stress bin/containerd bin/ctr bin/containerd-shim-runc-v2 ==> Start containerd Created symlink '/etc/systemd/system/multi-user.target.wants/containerd.service' → '/etc/systemd/system/containerd.service'. ==> Install CNI plugins ./ ./README.md ./static ./host-device ./ipvlan ./dhcp ./LICENSE ./portmap ./tap ./host-local ./vlan ./loopback ./sbr ./firewall ./bandwidth ./bridge ./vrf ./macvlan ./tuning ./dummy ./ptp ==> Load registry, nginx images unpacking docker.io/library/registry:2.8.1 (sha256:1e6c7d1be0dd576c7e50f786e3333382e209907eba72f5414a91025af241e16d)... Loaded image: registry:2.8.1 unpacking docker.io/library/registry:3.0.0 (sha256:09d6d68c85b98bac6699850ad2c071714e01a3bb7f67f68e636b98e4123275d2)... Loaded image: registry:3.0.0 unpacking docker.io/library/nginx:1.28.0-alpine (sha256:dc8e6d3967a06c0c9bb10d16cfc5770686de05da4c34d4224ef2aec61142e8f1)... Loaded image: nginx:1.28.0-alpine unpacking docker.io/library/nginx:1.29.4 (sha256:93c49ce72e039396ca3c51a43e2703bfc165500d1fe0faa697fb60c9e60fe99f)... Loaded image: nginx:1.29.4 root@admin:~/kubespray-offline/outputs# cat /etc/containerd/config.toml version = 2 root = "/var/lib/containerd" state = "/run/containerd" oom_score = 0 [grpc] address = "/run/containerd/containerd.sock" uid = 0 gid = 0 [debug] address = "/run/containerd/debug.sock" uid = 0 gid = 0 level = "info" [metrics] address = "" grpc_histogram = false [cgroup] path = "" [plugins] [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" snapshotter = "overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] systemdCgroup = true root@admin:~/kubespray-offline/outputs# cat /etc/systemd/system/containerd.service # Copyright The containerd Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=infinity # Comment TasksMax if your systemd version does not supports it. # Only systemd 226 and above support this version. TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target root@admin:~/kubespray-offline/outputs# systemctl status containerd.service --no-pager ● containerd.service - containerd container runtime Loaded: loaded (/etc/systemd/system/containerd.service; enabled; preset: disabled) Active: active (running) since Fri 2026-02-13 22:30:24 KST; 1min 12s ago Invocation: 642c661b243c4e7198228361ee5c4310 Docs: https://containerd.io Process: 19509 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 19511 (containerd) Tasks: 10 Memory: 634.9M (peak: 636.7M) CPU: 3.348s CGroup: /system.slice/containerd.service └─19511 /usr/local/bin/containerd Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.267143354+09:00" level=info msg="Start strea… server" Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.267151440+09:00" level=info msg="Registered …ith NRI" Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.267158303+09:00" level=info msg="runtime int…g up..." Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.267163092+09:00" level=info msg="starting plugins..." Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.267173832+09:00" level=info msg="Synchronizi…e state" Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.267988368+09:00" level=info msg=serving... a…bug.sock Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.268123675+09:00" level=info msg=serving... a…ck.ttrpc Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.268172639+09:00" level=info msg=serving... a…erd.sock Feb 13 22:30:24 admin containerd[19511]: time="2026-02-13T22:30:24.268615622+09:00" level=info msg="containerd …091145s" Feb 13 22:30:24 admin systemd[1]: Started containerd.service - containerd container runtime. Hint: Some lines were ellipsized, use -l to show in full. root@admin:~/kubespray-offline/outputs# nerdctl images REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE nginx 1.29.4 93c49ce72e03 About a minute ago linux/amd64 171MB 164.3MB nginx 1.28.0-alpine dc8e6d3967a0 About a minute ago linux/amd64 51.18MB 49.69MB registry 3.0.0 09d6d68c85b9 About a minute ago linux/amd64 58.44MB 58.26MB registry 2.8.1 1e6c7d1be0dd About a minute ago linux/amd64 26.65MB 26.49MB[2] start-nginx.sh 실행 : 웹 서버로 files, images, pypi, rpms 제공
root@admin:~/kubespray-offline/outputs# cp nginx-default.conf nginx-default.bak root@admin:~/kubespray-offline/outputs# cat << EOF > nginx-default.conf server { listen 80; listen [::]:80; server_name localhost; location / { root /usr/share/nginx/html; # index index.html index.htm; autoindex on; # 디렉터리 목록 표시 autoindex_exact_size off; # 파일 크기 KB/MB/GB 단위로 보기 좋게 autoindex_localtime on; # 서버 로컬 타임으로 표시 } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } # Force sendfile to off sendfile off; } EOF root@admin:~/kubespray-offline/outputs# ./start-nginx.sh ===> Stop nginx nginx nginx ===> Start nginx f6c4f06c2d2cb8e2c70dca6eb4c4d1e1a2afe3449df46064c99e53695a8da371 root@admin:~/kubespray-offline/outputs# nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f6c4f06c2d2c docker.io/library/nginx:1.29.4 "/docker-entrypoint.…" 5 seconds ago Up nginx root@admin:~/kubespray-offline/outputs# ss -tnlp | grep nginx LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=19951,fd=6),("nginx",pid=19950,fd=6),("nginx",pid=19949,fd=6),("nginx",pid=19948,fd=6),("nginx",pid=19912,fd=6)) LISTEN 0 511 [::]:80 [::]:* users:(("nginx",pid=19951,fd=7),("nginx",pid=19950,fd=7),("nginx",pid=19949,fd=7),("nginx",pid=19948,fd=7),("nginx",pid=19912,fd=7))
[3] setup-offline.sh 실행 : offline repo 설정, pypi mirror 전역 설정
root@admin:~/kubespray-offline/outputs# cat /etc/redhat-release Rocky Linux release 10.0 (Red Quartz) root@admin:~/kubespray-offline/outputs# ./setup-offline.sh /bin/rm: cannot remove '/etc/yum.repos.d/offline.repo': No such file or directory ===> Disable all yumrepositories ===> Setup local yum repository [offline-repo] name=Offline repo baseurl=http://localhost/rpms/local/ enabled=1 gpgcheck=0 ===> Setup PyPI mirror root@admin:~/kubespray-offline/outputs# tree /etc/yum.repos.d/ /etc/yum.repos.d/ ├── offline.repo ├── rocky-addons.repo.original ├── rocky-devel.repo.original ├── rocky-extras.repo.original └── rocky.repo.original 1 directory, 5 files root@admin:~/kubespray-offline/outputs# cat ~/.config/pip/pip.conf [global] index = http://localhost/pypi/ index-url = http://localhost/pypi/ trusted-host = localhost[4] setup-py.sh 실행 : offline repo 로 부터 python${PY} 설치 시도 → offline repo 동작 여부 확인
root@admin:~/kubespray-offline/outputs# ./setup-py.sh ===> Install python, venv, etc Offline repo 9.6 MB/s | 85 kB 00:00 Package python3-3.12.12-3.el10_1.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! root@admin:~/kubespray-offline/outputs# source pyver.sh root@admin:~/kubespray-offline/outputs# echo -e "python_version $python${PY}" python_version 3.12 root@admin:~/kubespray-offline/outputs# dnf info python3 Last metadata expiration check: 0:00:11 ago on Fri 13 Feb 2026 10:34:50 PM KST. Installed Packages Name : python3 Version : 3.12.12 Release : 3.el10_1 Architecture : x86_64 Size : 31 k Source : python3.12-3.12.12-3.el10_1.src.rpm Repository : @System From repo : baseos Summary : Python 3.12 interpreter URL : https://www.python.org/ License : Python-2.0.1[5] start-registry.sh 실행 : (컨테이너) 이미지 저장소 컨테이너로 기동
root@admin:~/kubespray-offline/outputs# ./start-registry.sh ===> Start registry 62e114b329464d54f5c06611b8b2ef55add063a821ef9fc0bfa00d5ca123ff34 root@admin:~/kubespray-offline/outputs# source config.sh root@admin:~/kubespray-offline/outputs# echo -e "registry_port: $REGISTRY_PORT" registry_port: 35000 root@admin:~/kubespray-offline/outputs# nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 62e114b32946 docker.io/library/registry:3.0.0 "/entrypoint.sh /etc…" 9 seconds ago Up registry f6c4f06c2d2c docker.io/library/nginx:1.29.4 "/docker-entrypoint.…" 4 minutes ago Up nginx root@admin:~/kubespray-offline/outputs# ss -tnlp | grep registry LISTEN 0 4096 *:5001 *:* users:(("registry",pid=20064,fd=7)) LISTEN 0 4096 *:35000 *:* users:(("registry",pid=20064,fd=3)) root@admin:~/kubespray-offline/outputs# curl 192.168.10.10:5001/metrics # HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 0.000606569 go_gc_duration_seconds{quantile="0.25"} 0.000606569 go_gc_duration_seconds{quantile="0.5"} 0.000680398 go_gc_duration_seconds{quantile="0.75"} 0.000680398 go_gc_duration_seconds{quantile="1"} 0.000680398 go_gc_duration_seconds_sum 0.001286967 go_gc_duration_seconds_count 2[6] load-push-images.sh 실행 : (컨테이너) 이미지 저장소에 이미지 push
root@admin:~/kubespray-offline/outputs# echo -e "cpu arch: $IMAGE_ARCH" cpu arch: amd64 root@admin:~/kubespray-offline/outputs# echo -e "Additional container registry hosts: $ADDITIONAL_CONTAINER_REGISTRY_LIST" Additional container registry hosts: myregistry.io root@admin:~/kubespray-offline/outputs# ls -l images/*.tar.gz -rw-r--r--. 1 root root 11194751 Feb 13 22:21 images/docker.io_amazon_aws-alb-ingress-controller-v1.1.9.tar.gz -rw-r--r--. 1 root root 175407403 Feb 13 22:22 images/docker.io_amazon_aws-ebs-csi-driver-v0.5.0.tar.gz -rw-r--r--. 1 root root 101536170 Feb 13 22:18 images/docker.io_cloudnativelabs_kube-router-v2.1.1.tar.gz -rw-r--r--. 1 root root 4735281 Feb 13 22:16 images/docker.io_flannel_flannel-cni-plugin-v1.7.1-flannel1.tar.gz load_images() { for image in $BASEDIR/images/*.tar.gz; do echo "===> Loading $image" sudo $NERDCTL load --all-platforms -i $image || exit 1 done } root@admin:~/kubespray-offline/outputs# nerdctl images | more REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE localhost:35000/kube-proxy v1.34.3 fbe99026b627 34 seconds ago linux/amd64 75.24MB 73.14MB localhost:35000/kube-scheduler v1.34.3 f9e384f4d1e8 35 seconds ago linux/amd64 55.6MB 53.85MB localhost:35000/kube-controller-manager v1.34.3 685d1c802d6c 40 seconds ago linux/amd64 77.75MB 76MB localhost:35000/kube-apiserver v1.34.3 7dd47dd94b4d 46 seconds ago linux/amd64 90.79MB 89.04MB localhost:35000/metallb/controller v0.13.9 b9859bda36a2 51 seconds ago linux/amd64 64.36MB 64.35MB localhost:35000/metallb/speaker v0.13.9 36f24e20f6aa 57 seconds ago linux/amd64 114MB 114MB localhost:35000/kubernetesui/metrics-scraper v1.0.8 04131c31ea1c 58 seconds ago linux/amd64 43.82MB 43.82MB localhost:35000/kubernetesui/dashboard v2.7.0 root@admin:~/kubespray-offline/outputs# nerdctl images | grep -i kube-proxy localhost:35000/kube-proxy v1.34.3 fbe99026b627 52 seconds ago linux/amd64 75.24MB 73.14MB registry.k8s.io/kube-proxy v1.34.3 fbe99026b627 5 minutes ago linux/amd64 75.24MB 73.14MB root@admin:~/kubespray-offline/outputs# nerdctl images | grep localhost | wc -l 55 root@admin:~/kubespray-offline/outputs# nerdctl images | grep -v localhost | wc -l 56 root@admin:~/kubespray-offline/outputs# curl -s http://localhost:35000/v2/_catalog | jq { "repositories": [ "amazon/aws-alb-ingress-controller", "amazon/aws-ebs-csi-driver", "calico/apiserver", "calico/cni", "calico/kube-controllers", "calico/node", "calico/typha", "cilium/certgen",
[7] extract-kubespary.sh 실행 : kubespary 저장소 압축 해제
root@admin:~/kubespray-offline/outputs# ls -lh files/kubespray-* -rw-r--r--. 1 root root 2.5M Feb 13 22:06 files/kubespray-2.30.0.tar.gz root@admin:~/kubespray-offline/outputs# tree patches/ patches/ └── 2.18.0 ├── 0001-nerdctl-insecure-registry-config-8339.patch ├── 0002-Update-config.toml.j2-8340.patch └── 0003-generate-list-8537.patch 2 directories, 3 files root@admin:~/kubespray-offline/outputs# ./extract-kubespray.sh kubespray-2.30.0/ kubespray-2.30.0/.ansible-lint kubespray-2.30.0/.ansible-lint-ignore kubespray-2.30.0/.editorconfig kubespray-2.30.0/.gitattributes kubespray-2.30.0/.github/ kubespray-2.30.0/.github/ISSUE_TEMPLATE/ kubespray-2.30.0/.github/ISSUE_TEMPLATE/bug-report.yaml kubespray-2.30.0/.github/ISSUE_TEMPLATE/config.yml kubespray-2.30.0/.github/ISSUE_TEMPLATE/enhancement.yaml kubespray-2.30.0/.github/ISSUE_TEMPLATE/failing-test.yaml kubespray-2.30.0/.github/PULL_REQUEST_TEMPLATE.md kubespray-2.30.0/.github/dependabot.yml kubespray-2.30.0/.github/workflows/ ########## 중략 ################# root@admin:~/kubespray-offline/outputs# tree kubespray-2.30.0/ -L 1 kubespray-2.30.0/ ├── ansible.cfg ├── CHANGELOG.md ├── cluster.yml ├── CNAME ├── code-of-conduct.md ├── _config.yml ├── contrib ├── CONTRIBUTING.md ├── Dockerfile ├── docs ├── extra_playbooks ├── galaxy.yml ├── index.html ├── inventory ├── library ├── LICENSE ├── logo ├── meta ├── OWNERS ├── OWNERS_ALIASES ├── pipeline.Dockerfile ├── playbooks ├── plugins ├── README.md ├── recover-control-plane.yml ├── RELEASE.md ├── remove-node.yml ├── remove_node.yml ├── requirements.txt ├── reset.yml ├── roles ├── scale.yml ├── scripts ├── SECURITY_CONTACTS ├── test-infra ├── tests ├── upgrade-cluster.yml ├── upgrade_cluster.yml └── Vagrantfile 14 directories, 26 files
kubespray 설치
kubespray설치
root@admin:~/kubespray-offline/outputs# python --version Python 3.12.12 root@admin:~/kubespray-offline/outputs# python3.12 -m venv ~/.venv/3.12 root@admin:~/kubespray-offline/outputs# source ~/.venv/3.12/bin/activate ((3.12) ) root@admin:~/kubespray-offline/outputs# which ansible /root/.venv/3.12/bin/ansible ((3.12) ) root@admin:~/kubespray-offline/outputs# tree ~/.venv/3.12/ -L 4 /root/.venv/3.12/ ├── bin │ ├── activate │ ├── activate.csh │ ├── activate.fish ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node1 dnf repolist repo id repo name appstream Rocky Linux 10 - AppStream baseos Rocky Linux 10 - BaseOS extras Rocky Linux 10 - Extras offline-repo Offline repo for kubespray ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 dnf repolist repo id repo name offline-repo Offline repo for kubespray ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node1 dnf repolist repo id repo name offline-repo Offline repo for kubespray ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 cat /etc/resolv.conf nameserver 192.168.10.10 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 cat /etc/NetworkManager/conf.d/dns.conf [global-dns-domain-*] servers = 10.233.0.3,192.168.10.10 [global-dns] searches = default.svc.cluster.local,svc.cluster.local options = ndots:2,timeout:2,attempts:2 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# kubectl get deploy,sts,ds -n kube-system -owide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/coredns 2/2 2 2 3m24s coredns 192.168.10.10:35000/coredns/coredns:v1.12.1 k8s-app=kube-dns deployment.apps/metrics-server 1/1 1 1 3m3s metrics-server 192.168.10.10:35000/metrics-server/metrics-server:v0.8.0 app.kubernetes.io/name=metrics-server,version=0.8.0 NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR daemonset.apps/kube-flannel 2 2 2 2 2 <none> 3m41s kube-flannel 192.168.10.10:35000/flannel/flannel:v0.27.3 app=flannel daemonset.apps/kube-flannel-ds-arm 0 0 0 0 0 <none> 3m41s kube-flannel 192.168.10.10:35000/flannel/flannel:v0.27.3 app=flannel daemonset.apps/kube-flannel-ds-arm64 0 0 0 0 0 <none> 3m41s kube-flannel 192.168.10.10:35000/flannel/flannel:v0.27.3 app=flannel daemonset.apps/kube-flannel-ds-ppc64le 0 0 0 0 0 <none> 3m41s kube-flannel 192.168.10.10:35000/flannel/flannel:v0.27.3 app=flannel daemonset.apps/kube-flannel-ds-s390x 0 0 0 0 0 <none> 3m41s kube-flannel 192.168.10.10:35000/flannel/flannel:v0.27.3 app=flannel daemonset.apps/kube-proxy 2 2 2 2 2 kubernetes.io/os=linux 4m27s kube-proxy 192.168.10.10:35000/kube-proxy:v1.34.3 k8s-app=kube-proxy[k8s-node] 이미지 저장소 관련 정보 확인
root@k8s-node1:~# crictl images IMAGE TAG IMAGE ID SIZE 192.168.10.10:35000/coredns/coredns v1.12.1 52546a367cc9e 76.1MB 192.168.10.10:35000/flannel/flannel-cni-plugin v1.7.1-flannel1 48b5d33f9a21f 11MB 192.168.10.10:35000/flannel/flannel v0.27.3 3475d115f79b6 92.2MB 192.168.10.10:35000/kube-apiserver v1.34.3 aa27095f56193 89MB 192.168.10.10:35000/kube-controller-manager v1.34.3 5826b25d990d7 76MB 192.168.10.10:35000/kube-proxy v1.34.3 36eef8e07bdd6 73.1MB 192.168.10.10:35000/kube-scheduler v1.34.3 aec12dadf56dd 53.8MB 192.168.10.10:35000/metrics-server/metrics-server v0.8.0 b9e1e3849e070 83.7MB 192.168.10.10:35000/pause 3.10.1 cd073f4c5f6a8 739kB root@k8s-node1:~# cat /etc/containerd/certs.d/192.168.10.10\:35000/hosts.toml server = "https://192.168.10.10:35000" [host."http://192.168.10.10:35000"] capabilities = ["pull","resolve"] skip_verify = true override_path = falsekubespary-offline 에
kube_version변경 적용하여 관련 파일 다운로드((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# cd /root/kubespray-offline ((3.12) ) root@admin:~/kubespray-offline# tree cache/kubespray-2.30.0/contrib/offline/temp/ cache/kubespray-2.30.0/contrib/offline/temp/ ├── files.list ├── files.list.template ├── images.list └── images.list.template 1 directory, 4 files ((3.12) ) root@admin:~/kubespray-offline# mv cache/kubespray-2.30.0/contrib/offline/temp/files.list cache/kubespray-2.30.0/contrib/offline/temp/files-2.list ((3.12) ) root@admin:~/kubespray-offline# mv cache/kubespray-2.30.0/contrib/offline/temp/images.list cache/kubespray-2.30.0/contrib/offline/temp/images-2.list ((3.12) ) root@admin:~/kubespray-offline# cat download-kubespray-files.sh #!/bin/bash umask 022 source ./config.sh source scripts/common.sh source scripts/images.sh KUBESPRAY_DIR=./cache/kubespray-${KUBESPRAY_VERSION} if [ ! -e $KUBESPRAY_DIR ]; then echo "No kubespray dir at $KUBESPRAY_DIR" exit 1 fi FILES_DIR=outputs/files # Decide relative directory of file from URL # # kubernetes/vx.x.x : kubeadm/kubectl/kubelet # kubernetes/etcd : etcd # kubernetes/cni : CNI plugins # kubernetes/cri-tools : crictl # kubernetes/calico/vx.x.x : calico # kubernetes/calico : calicoctl # runc/vx.x.x : runc # cilium-cli/vx.x.x : cilium-cli # gvisor/{ver}/{arch} : gvisor (sunrc, containerd-shim) # scopeo/vx.x.x : scopeo # yq/vx.x.x : yq # decide_relative_dir() { local url=$1 local rdir rdir=$url rdir=$(echo $rdir | sed "s@.*/\(v[0-9.]*\)/.*/kube\(adm\|ctl\|let\)@kubernetes/\1@g") rdir=$(echo $rdir | sed "s@.*/etcd-.*.tar.gz@kubernetes/etcd@") rdir=$(echo $rdir | sed "s@.*/cni-plugins.*.tgz@kubernetes/cni@") rdir=$(echo $rdir | sed "s@.*/crictl-.*.tar.gz@kubernetes/cri-tools@") rdir=$(echo $rdir | sed "s@.*/\(v.*\)/calicoctl-.*@kubernetes/calico/\1@") rdir=$(echo $rdir | sed "s@.*/\(v.*\)/runc.${IMAGE_ARCH}@runc/\1@") rdir=$(echo $rdir | sed "s@.*/\(v.*\)/cilium-linux-.*@cilium-cli/\1@") rdir=$(echo $rdir | sed "s@.*/\([^/]*\)/\([^/]*\)/runsc@gvisor/\1/\2@") rdir=$(echo $rdir | sed "s@.*/\([^/]*\)/\([^/]*\)/containerd-shim-runsc-v1@gvisor/\1/\2@") rdir=$(echo $rdir | sed "s@.*/\(v[^/]*\)/skopeo-linux-.*@skopeo/\1@") rdir=$(echo $rdir | sed "s@.*/\(v[^/]*\)/yq_linux_*@yq/\1@") if [ "$url" != "$rdir" ]; then echo $rdir return fi rdir=$(echo $rdir | sed "s@.*/calico/.*@kubernetes/calico@") if [ "$url" != "$rdir" ]; then echo $rdir else echo "" fi } get_url() { url=$1 filename="${url##*/}" rdir=$(decide_relative_dir $url) if [ -n "$rdir" ]; then if [ ! -d $FILES_DIR/$rdir ]; then mkdir -p $FILES_DIR/$rdir fi else rdir="." fi if [ ! -e $FILES_DIR/$rdir/$filename ]; then echo "==> Download $url" for i in {1..3}; do curl --location --show-error --fail --output $FILES_DIR/$rdir/$filename $url && return echo "curl failed. Attempt=$i" done echo "Download failed, exit : $url" exit 1 else echo "==> Skip $url" fi } # execute offline generate_list.sh generate_list() { #if [ $KUBESPRAY_VERSION == "2.18.0" ]; then # export containerd_version=${containerd_version:-1.5.8} # export host_os=linux # export image_arch=amd64 #fi LANG=C /bin/bash ${KUBESPRAY_DIR}/contrib/offline/generate_list.sh || exit 1 #if [ $KUBESPRAY_VERSION == "2.18.0" ]; then # # check roles/download/default/main.yml to decide version # snapshot_controller_tag=${snapshot_controller_tag:-v4.2.1} # sed -i "s@\(.*/snapshot-controller:\)@\1${snapshot_controller_tag}@" ${KUBESPRAY_DIR}/contrib/offline/temp/images.list || exit 1 #fi } . ./target-scripts/venv.sh generate_list mkdir -p $FILES_DIR cp ${KUBESPRAY_DIR}/contrib/offline/temp/files.list $FILES_DIR/ cp ${KUBESPRAY_DIR}/contrib/offline/temp/images.list $IMAGES_DIR/ # download files files=$(cat ${FILES_DIR}/files.list) for i in $files; do get_url $i done # download images ./download-images.sh || exit 1 ((3.12) ) root@admin:~/kubespray-offline# cp download-kubespray-files.sh download-kubespray-files.bak ((3.12) ) root@admin:~/kubespray-offline# sed -i '/generate_list$/,$ { /generate_list/!d }' download-kubespray-files.sh ((3.12) ) root@admin:~/kubespray-offline# diff download-kubespray-files.sh download-kubespray-files.bak 104a105,118 > > mkdir -p $FILES_DIR > > cp ${KUBESPRAY_DIR}/contrib/offline/temp/files.list $FILES_DIR/ > cp ${KUBESPRAY_DIR}/contrib/offline/temp/images.list $IMAGES_DIR/ > > # download files > files=$(cat ${FILES_DIR}/files.list) > for i in $files; do > get_url $i > done > > # download images > ./download-images.sh || exit 1 ((3.12) ) root@admin:~/kubespray-offline# sed -i 's|offline/generate_list.sh|offline/generate_list.sh -e kube_version=1.33.7|g' download-kubespray-files.sh ((3.12) ) root@admin:~/kubespray-offline# cat download-kubespray-files.sh | grep kube_version LANG=C /bin/bash ${KUBESPRAY_DIR}/contrib/offline/generate_list.sh -e kube_version=1.33.7 || exit 1 ((3.12) ) root@admin:~/kubespray-offline# ./download-kubespray-files.sh python3 = python3.12 VENV_DIR = /root/.venv/3.12 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [Collect container images for offline deployment] ***************************************************************** Friday 13 February 2026 23:07:52 +0900 (0:00:00.016) 0:00:00.016 ******* Friday 13 February 2026 23:07:52 +0900 (0:00:00.013) 0:00:00.030 ******* Friday 13 February 2026 23:07:52 +0900 (0:00:00.015) 0:00:00.045 ******* Friday 13 February 2026 23:07:52 +0900 (0:00:00.015) 0:00:00.061 ******* Friday 13 February 2026 23:07:52 +0900 (0:00:00.014) 0:00:00.075 ******* Friday 13 February 2026 23:07:53 +0900 (0:00:00.012) 0:00:00.088 ******* Friday 13 February 2026 23:07:53 +0900 (0:00:00.021) 0:00:00.110 ******* Friday 13 February 2026 23:07:53 +0900 (0:00:00.013) 0:00:00.123 ******* Friday 13 February 2026 23:07:53 +0900 (0:00:00.016) 0:00:00.139 ******* Friday 13 February 2026 23:07:53 +0900 (0:00:00.011) 0:00:00.150 ******* Friday 13 February 2026 23:07:53 +0900 (0:00:00.423) 0:00:00.574 ******* TASK [Collect container images for offline deployment] ***************************************************************** changed: [localhost] => (item=files) changed: [localhost] => (item=images) PLAY RECAP ************************************************************************************************************* localhost : ok=1 changed=1 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 Friday 13 February 2026 23:07:54 +0900 (0:00:00.937) 0:00:01.512 ******* =============================================================================== Collect container images for offline deployment ----------------------------------------------------------------- 0.94s download : Download | Download files / images ------------------------------------------------------------------- 0.42s download : Prep_download | Register docker images info ---------------------------------------------------------- 0.02s download : Prep_download | Create local cache for files and images on control node ------------------------------ 0.02s download : Prep_download | On localhost, check if passwordless root is possible --------------------------------- 0.02s download : Prep_download | On localhost, check if user has access to the container runtime without using sudo --- 0.02s download : Prep_download | Parse the outputs of the previous commands ------------------------------------------- 0.01s download : Prep_download | Set a few facts ---------------------------------------------------------------------- 0.01s download : Prep_download | Create staging directory on remote node ---------------------------------------------- 0.01s download : Prep_download | Check that local user is in group or can become root --------------------------------- 0.01s download : Download | Get kubeadm binary and list of required images -------------------------------------------- 0.01s ((3.12) ) root@admin:~/kubespray-offline# cd cache/kubespray-2.30.0/contrib/offline/temp ((3.12) ) root@admin:~/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/temp# diff files-2.list files.list 1,3c1,3 < https://dl.k8s.io/release/v1.34.3/bin/linux/amd64/kubelet < https://dl.k8s.io/release/v1.34.3/bin/linux/amd64/kubectl < https://dl.k8s.io/release/v1.34.3/bin/linux/amd64/kubeadm --- > https://dl.k8s.io/release/v1.33.7/bin/linux/amd64/kubelet > https://dl.k8s.io/release/v1.33.7/bin/linux/amd64/kubectl > https://dl.k8s.io/release/v1.33.7/bin/linux/amd64/kubeadm 9,10c9,10 < https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.34.0/crictl-v1.34.0-linux-amd64.tar.gz < https://storage.googleapis.com/cri-o/artifacts/cri-o.amd64.v1.34.4.tar.gz --- > https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.33.0/crictl-v1.33.0-linux-amd64.tar.gz > https://storage.googleapis.com/cri-o/artifacts/cri-o.amd64.v1.33.8.tar.gz
K8S 관련 폐쇄망 실습
샘플앱배포
nginx[alpine]
((3.12) ) root@admin:~/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/temp# cat << EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:alpine # docker.io/library/nginx:alpine ports: - containerPort: 80 EOF ##배포 시도해도 실패 ((3.12) ) root@admin:~/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/temp# kubectl describe pod Warning Failed 8s kubelet Failed to pull image "nginx:alpine": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "docker.io/library/nginx:alpine": failed to resolve image: failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/alpine": dial tcp 98.95.137.26:443: i/o timeout Warning Failed 8s kubelet Error: ErrImagePull Normal BackOff 8s kubelet Back-off pulling image "nginx:alpine" Warning Failed 8s kubelet Error: ImagePullBackOff(컨테이너) 이미지 저장소에 이미지 push
((3.12) ) root@admin:~/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/temp# podman pull nginx:alpine ✔ docker.io/library/nginx:alpine Trying to pull docker.io/library/nginx:alpine... Getting image source signatures Copying blob 955a8478f9ac done | Copying blob 589002ba0eae done | Copying blob 3e2c181db1b0 done | Copying blob bca5d04786e1 done | Copying blob 6b7b6c7061b7 done | Copying blob 399d0898a94e done | Copying blob 6d397a54a185 done | Copying blob 5e7756927bef done | Copying config b76de378d5 done | Writing manifest to image destination b76de378d57272a1dd9091a05dd548a3639dfb792ebdbf95d06704d2950afdea ((3.12) ) root@admin:~/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/temp# podman images | grep nginx docker.io/library/nginx alpine b76de378d572 8 days ago 63.5 MB docker.io/library/nginx 1.29.4 248d2326f351 10 days ago 164 MB registry.k8s.io/ingress-nginx/controller v1.13.3 c44d76c3213e 4 months ago 334 MB docker.io/library/nginx 1.28.0-alpine c318e336065b 9 months ago 49.7 MB ((3.12) ) root@admin:~# curl -s 192.168.10.10:35000/v2/library/nginx/tags/list | jq { "name": "library/nginx", "tags": [ "1.28.0-alpine", "1.29.4" ] }kubespary 에 containerd_registries_mirrors values 설정 후 적용 --tags containerd
((3.12) ) root@admin:~# cat /root/kubespray-offline/outputs/kubespray-2.30.0/inventory/mycluster/group_vars/all/offline.yml | head -n 15 # # offline.yml sample # http_server: "http://192.168.10.10" registry_host: "192.168.10.10:35000" # Insecure registries for containerd containerd_registries_mirrors: - prefix: "{{ registry_host }}" mirrors: - host: "http://{{ registry_host }}" capabilities: ["pull", "resolve"] skip_verify: true ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.34.3" --tags containerd ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 tree /etc/containerd /etc/containerd ├── certs.d │ ├── 192.168.10.10:35000 │ │ └── hosts.toml │ ├── docker.io │ │ └── hosts.toml │ ├── quay.io │ │ └── hosts.toml │ └── registry-1.docker.io │ └── hosts.toml ├── config.toml └── cri-base.json ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# ssh k8s-node2 cat /etc/containerd/certs.d/quay.io/hosts.toml server = "https://quay.io" [host."http://192.168.10.10:35000"] capabilities = ["pull","resolve"] skip_verify = true override_path = false
헬름 차트 저장소
nginx helm 차트 작성, 배포 및 tgz 패키징
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/nginx alpine b76de378d572 8 days ago 63.5 MB 192.168.10.10:35000/library/nginx alpine ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# podman pull nginx:1.28.0-alpine Trying to pull docker.io/library/nginx:1.28.0-alpine... ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# podman tag nginx:1.28.0-alpine 192.168.10.10:35000/library/nginx:1.28.0-alpine ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0# curl -s 192.168.10.10:35000/v2/library/nginx/tags/list | jq { "name": "library/nginx", "tags": [ "1.28.0-alpine", "1.29.4", "alpine" ] } ##차트작성 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# tree . ├── Chart.yaml ├── templates │ ├── deployment.yaml │ └── service.yaml └── values.yaml 2 directories, 4 files ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# helm package . Successfully packaged chart and saved it to: /root/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart/nginx-chart-1.0.0.tgz ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# tar -tzf nginx-chart-1.0.0.tgz nginx-chart/Chart.yaml nginx-chart/values.yaml nginx-chart/templates/deployment.yaml nginx-chart/templates/service.yaml ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# zcat nginx-chart-1.0.0.tgz | tar -xOf - nginx-chart/Chart.yaml apiVersion: v2 appVersion: 1.28.0-alpine description: A Helm chart for deploying Nginx with custom index.html name: nginx-chart type: application version: 1.0.0 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# helm uninstall dev-nginx release "dev-nginx" uninstalled ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION외부 공용 차트 tgz 패키징 다운로드 후 배포 서버에 복사 후 사용
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# podman pull docker.io/bitnami/nginx:latest Trying to pull docker.io/bitnami/nginx:latest... Getting image source signatures Copying blob 39137f68a782 done | Copying config f335e22d6b done | Writing manifest to image destination f335e22d6b5abdfc5de5c5d8fd54f6af452e0e4cf7c6c003e7ddf1649e2d0387 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# podman tag bitnami/nginx:latest 192.168.10.10:35000/bitnami/nginx:latest ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# podman push 192.168.10.10:35000/bitnami/nginx:latest Getting image source signatures Copying blob 83eeb01b5597 done | Copying config f335e22d6b done | Writing manifest to image destination ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# helm get metadata my-nginx NAME: my-nginx CHART: nginx VERSION: 22.4.7 APP_VERSION: 1.29.5 ANNOTATIONS: fips=true,images=- name: git version: 2.53.0 image: registry-1.docker.io/bitnami/git:latest - name: nginx version: 1.29.5 image: registry-1.docker.io/bitnami/nginx:latest - name: nginx-exporter version: 1.5.1 image: registry-1.docker.io/bitnami/nginx-exporter:latest ,licenses=Apache-2.0,tanzuCategory=clusterUtility DEPENDENCIES: common NAMESPACE: default REVISION: 1 STATUS: deployed DEPLOYED_AT: 2026-02-13T23:38:51+09:00 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION my-nginx default 1 2026-02-13 23:38:51.442908021 +0900 KST deployed nginx-22.4.7 1.29.5 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# kubectl get deploy -owide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR my-nginx 1/1 1 1 14s nginx registry-1.docker.io/bitnami/nginx:latest app.kubernetes.io/instance=my-nginx,app.kubernetes.io/name=nginx nginx 1/1 1 1 15m nginx nginx:alpine app=nginx ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# helm get manifest my-nginx | grep 'image:' image: registry-1.docker.io/bitnami/nginx:latest image: registry-1.docker.io/bitnami/nginx:latest내부망에 Helm Chart 저장소(ChartMuseum) 구축 후 nginx 차트 업로드 후 사용
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# chmod 777 /data/chartmuseum/charts ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# podman run -d \ --name chartmuseum \ -p 8080:8080 \ -v /data/chartmuseum/charts:/charts \ -e STORAGE=local \ -e STORAGE_LOCAL_ROOTDIR=/charts \ -e DEBUG=true \ ghcr.io/helm/chartmuseum:v0.16.4 Trying to pull ghcr.io/helm/chartmuseum:v0.16.4... Getting image source signatures Copying blob 589002ba0eae skipped: already exists Copying blob aa06029dd384 done | Copying blob c1c9b34da041 done | Copying config 281626b9b5 done | Writing manifest to image destination fb47663c6e175287500ddfffbf200aead634d15fb00e719adf8716953d462841 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# helm repo add internal http://192.168.10.10:8080 "internal" has been added to your repositories ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "internal" chart repository Update Complete. ⎈Happy Helming!⎈ ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-oci-reg# helm repo list NAME URL internal http://192.168.10.10:8080 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# curl -s http://192.168.10.10:8080/api/charts | jq { "nginx-chart": [ { "name": "nginx-chart", "version": "1.0.0", "description": "A Helm chart for deploying Nginx with custom index.html", "apiVersion": "v2", "appVersion": "1.28.0-alpine", "type": "application", "urls": [ "charts/nginx-chart-1.0.0.tgz" ], "created": "2026-02-13T14:44:08.758241429Z", "digest": "35ce014a79d3c92287d0784aef47e0fc021f42371141536fbaf410da2271c6fd" } ] } ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "internal" chart repository Update Complete. ⎈Happy Helming!⎈ ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# helm install my-nginx internal/nginx-chart NAME: my-nginx LAST DEPLOYED: Fri Feb 13 23:44:34 2026 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION my-nginx default 1 2026-02-13 23:44:34.773014407 +0900 KST deployed nginx-chart-1.0.0 1.28.0-alpine
Private PyPI(Python Package Index) Mirror
[k8s-node] pip 설정 및 사용
root@k8s-node1:~# curl http://192.168.10.10/pypi/ <!DOCTYPE html> <html> <head> <title>Simple index</title> </head> <body> <a href="ansible/index.html">ansible</a> <a href="ansible-core/index.html">ansible-core</a> <a href="cffi/index.html">cffi</a> <a href="cryptography/index.html">cryptography</a> <a href="cython/index.html">Cython</a> <a href="distro/index.html">distro</a> <a href="flit-core/index.html">flit_core</a> <a href="jinja2/index.html">Jinja2</a> <a href="jmespath/index.html">jmespath</a> <a href="markupsafe/index.html">MarkupSafe</a> <a href="netaddr/index.html">netaddr</a> <a href="packaging/index.html">packaging</a> <a href="pip/index.html">pip</a> <a href="pycparser/index.html">pycparser</a> <a href="pyyaml/index.html">PyYAML</a> <a href="resolvelib/index.html">resolvelib</a> <a href="ruamel-yaml/index.html">ruamel.yaml</a> <a href="selinux/index.html">selinux</a> <a href="setuptools/index.html">setuptools</a> <a href="wheel/index.html">wheel</a> </body>
root@k8s-node1:~# pip list | grep -i netaddr
root@k8s-node1:~# pip install netaddr
Looking in indexes: http://192.168.10.10/pypi
Collecting netaddr
Downloading http://192.168.10.10/pypi/netaddr/netaddr-1.3.0-py3-none-any.whl (2.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 46.0 MB/s eta 0:00:00
Installing collected packages: netaddr
Successfully installed netaddr-1.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@k8s-node1:~# pip list | grep -i netaddr
netaddr 1.3.0
root@k8s-node1:~# pip install httpx
Looking in indexes: http://192.168.10.10/pypi
ERROR: Could not find a version that satisfies the requirement httpx (from versions: none)
ERROR: No matching distribution found for httpx
```pypi 추가패키지설치후 사용
((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# cat /root/.config/pip/pip.conf [global] index = http://localhost/pypi/ index-url = http://localhost/pypi/ trusted-host = localhost ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# mv /root/.config/pip/pip.conf /root/.config/pip/pip.bak ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# pip install httpx Collecting httpx Downloading httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB) Collecting anyio (from httpx) Downloading anyio-4.12.1-py3-none-any.whl.metadata (4.3 kB) Collecting certifi (from httpx) Downloading certifi-2026.1.4-py3-none-any.whl.metadata (2.5 kB) Collecting httpcore==1.* (from httpx) Downloading httpcore-1.0.9-py3-none-any.whl.metadata (21 kB) Collecting idna (from httpx) Downloading idna-3.11-py3-none-any.whl.metadata (8.4 kB) Collecting h11>=0.16 (from httpcore==1.*->httpx) Downloading h11-0.16.0-py3-none-any.whl.metadata (8.3 kB) Collecting typing_extensions>=4.5 (from anyio->httpx) Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) Downloading httpx-0.28.1-py3-none-any.whl (73 kB) Downloading httpcore-1.0.9-py3-none-any.whl (78 kB) Downloading h11-0.16.0-py3-none-any.whl (37 kB) Downloading anyio-4.12.1-py3-none-any.whl (113 kB) Downloading idna-3.11-py3-none-any.whl (71 kB) Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) Downloading certifi-2026.1.4-py3-none-any.whl (152 kB) Installing collected packages: typing_extensions, idna, h11, certifi, httpcore, anyio, httpx Successfully installed anyio-4.12.1 certifi-2026.1.4 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 idna-3.11 typing_extensions-4.15.0 ((3.12) ) root@admin:~/kubespray-offline/outputs/kubespray-2.30.0/nginx-chart# pip list | grep httpx httpx 0.28.1 ((3.12) ) root@admin:~/kubespray-offline# curl http://192.168.10.10/pypi/ <!DOCTYPE html> <html> <head> <title>Simple index</title> </head> <body> <a href="ansible/index.html">ansible</a> <a href="ansible-core/index.html">ansible-core</a> <a href="cffi/index.html">cffi</a> <a href="cryptography/index.html">cryptography</a> <a href="cython/index.html">Cython</a> <a href="distro/index.html">distro</a> <a href="flit-core/index.html">flit_core</a> <a href="httpx/index.html">httpx</a> <a href="jinja2/index.html">Jinja2</a> <a href="jmespath/index.html">jmespath</a> <a href="markupsafe/index.html">MarkupSafe</a> <a href="netaddr/index.html">netaddr</a> <a href="packaging/index.html">packaging</a> <a href="pip/index.html">pip</a> <a href="pycparser/index.html">pycparser</a> <a href="pyyaml/index.html">PyYAML</a> <a href="resolvelib/index.html">resolvelib</a> <a href="ruamel-yaml/index.html">ruamel.yaml</a> <a href="selinux/index.html">selinux</a> <a href="setuptools/index.html">setuptools</a> <a href="wheel/index.html">wheel</a> </body> </html>((3.12) ) root@admin:~/kubespray-offline# pip install httpx Requirement already satisfied: httpx in /root/.venv/3.12/lib64/python3.12/site-packages (0.28.1) Requirement already satisfied: anyio in /root/.venv/3.12/lib64/python3.12/site-packages (from httpx) (4.12.1) Requirement already satisfied: certifi in /root/.venv/3.12/lib64/python3.12/site-packages (from httpx) (2026.1.4) Requirement already satisfied: httpcore==1.* in /root/.venv/3.12/lib64/python3.12/site-packages (from httpx) (1.0.9) Requirement already satisfied: idna in /root/.venv/3.12/lib64/python3.12/site-packages (from httpx) (3.11) Requirement already satisfied: h11>=0.16 in /root/.venv/3.12/lib64/python3.12/site-packages (from httpcore==1.*->httpx) (0.16.0) Requirement already satisfied: typing_extensions>=4.5 in /root/.venv/3.12/lib64/python3.12/site-packages (from anyio->httpx) (4.15.0)