5주차 - ClusterMesh

본 내용은 gasida님의 Cilium 스터디를 진행 후 정리한 내용입니다.
꾸준히 업데이트 하면서 작성됩니다

 

 

환경 구성

Mac 기준

1. Docker Desktop 설치

2. kind 설치

# Install Kind
brew install kind
kind --version

# Install kubectl
brew install kubernetes-cli
kubectl version --client=true

## kubectl -> k 단축키 설정
echo "alias kubectl=kubecolor" >> ~/.zshrc

# Install Helm
brew install helm
helm version

 

(권장) 유용한 툴 설치

# 툴 설치
brew install krew
brew install kube-ps1
brew install kubectx

# kubectl 출력 시 하이라이트 처리
brew install kubecolor
echo "alias kubectl=kubecolor" >> ~/.zshrc
echo "compdef kubecolor=kubectl" >> ~/.zshrc

 

 

 

ClusterMesh

https://docs.cilium.io/en/latest/installation/kind/

 

kind k8s 클러스터 west, east 배포

#
kind create cluster --name west --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000 # sample apps
    hostPort: 30000
  - containerPort: 30001 # hubble ui
    hostPort: 30001
- role: worker
  extraPortMappings:
  - containerPort: 30002 # sample apps
    hostPort: 30002
networking:
  podSubnet: "10.0.0.0/16"
  serviceSubnet: "10.2.0.0/16"
  disableDefaultCNI: true
  kubeProxyMode: none
EOF


# 설치 확인
kubectl ctx
kubectl get node 
kubectl get pod -A

# 노드에 기본 툴 설치
docker exec -it west-control-plane sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
docker exec -it west-worker sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'

 

#
kind create cluster --name east --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 31000 # sample apps
    hostPort: 31000
  - containerPort: 31001 # hubble ui
    hostPort: 31001
- role: worker
  extraPortMappings:
  - containerPort: 31002 # sample apps
    hostPort: 31002
networking:
  podSubnet: "10.1.0.0/16"
  serviceSubnet: "10.3.0.0/16"
  disableDefaultCNI: true
  kubeProxyMode: none
EOF

# 설치 확인
kubectl config get-contexts 
CURRENT   NAME        CLUSTER     AUTHINFO    NAMESPACE
*         kind-east   kind-east   kind-east   
          kind-west   kind-west   kind-west 

kubectl config set-context kind-east
kubectl get node -v=6 --context kind-east
kubectl get node -v=6
kubectl get node -v=6 --context kind-west
cat ~/.kube/config

kubectl get pod -A
kubectl get pod -A --context kind-west

# 노드에 기본 툴 설치
docker exec -it east-control-plane sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
docker exec -it east-worker sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'

 

alias 설정

# alias 설정
alias kwest='kubectl --context kind-west'
alias keast='kubectl --context kind-east'

# 확인
kwest get node -owide
keast get node -owide

 

도전과제6

kind 와 같은 네트워크로 frr 컨테이너를 배포(bgp config 주입)하여, cilium 과 BGP(ECMP) + LB IPAM 로 인입 설정해보기

 

 

 

Cilium CNI 배포 - ClusterMesh 

https://docs.cilium.io/en/stable/network/clustermesh/clustermesh/

https://nomadxd.github.io/blog/multi-cluster-networking-with-cilium-cluster-mesh

Multi cluster networking with Cilium Cluster Mesh

 

clustermesh로 연결하기위해 아래 두 옵션이 포함됩니다.

--set cluster.name=west --set cluster.id=1 

--set cluster.name=east --set cluster.id=12

# cilium cli 설치
brew install cilium-cli # macOS
https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli # Windwos WSL2 , 아래 명령어로 설치
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}


# cilium cli 로 cilium cni 설치해보기 : dry-run
cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.0.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.1.0.0/16}' \
--set cluster.name=west --set cluster.id=1 \
--context kind-west --dry-run-helm-values

# cilium cli 로 cilium cni 설치
cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.0.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.1.0.0/16}' \
--set cluster.name=west --set cluster.id=1 \
--context kind-west

watch kubectl get pod -n kube-system --context kind-west


# dry-run
cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.1.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.0.0.0/16}' \
--set cluster.name=east --set cluster.id=2 \
--context kind-east --dry-run-helm-values

cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.1.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.0.0.0/16}' \
--set cluster.name=east --set cluster.id=2 \
--context kind-east

watch kubectl get pod -n kube-system --context kind-east


# 확인
kwest get pod -A && keast get pod -A
cilium status --context kind-west
cilium status --context kind-east
cilium config view --context kind-west
cilium config view --context kind-east
kwest exec -it -n kube-system ds/cilium -- cilium status --verbose
keast exec -it -n kube-system ds/cilium -- cilium status --verbose

kwest -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf ipmasq list
keast -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf ipmasq list

# coredns 확인 : 둘 다, cluster.local 기본 도메인 네임 사용 중
kubectl describe cm -n kube-system coredns --context kind-west | grep kubernetes
    kubernetes cluster.local in-addr.arpa ip6.arpa {

kubectl describe cm -n kube-system coredns --context kind-west | grep kubernetes
    kubernetes cluster.local in-addr.arpa ip6.arpa {

# k9s 사용 시
k9s --context kind-west
k9s --context kind-east


# 삭제 시
# cilium uninstall --context kind-west
# cilium uninstall --context kind-east

 

 

 

Setting up Cluster Mesh

# 라우팅 정보 확인
docker exec -it west-control-plane ip -c route
docker exec -it west-worker ip -c route
docker exec -it east-control-plane ip -c route
docker exec -it east-worker ip -c route


# Specify the Cluster Name and ID : 이미 설정 되어 있음

# Shared Certificate Authority
keast get secret -n kube-system cilium-ca
keast delete secret -n kube-system cilium-ca

kubectl --context kind-west get secret -n kube-system cilium-ca -o yaml | \
kubectl --context kind-east create -f -

keast get secret -n kube-system cilium-ca

 

# 모니터링 : 신규 터미널 2개
cilium clustermesh status --context kind-west --wait  
cilium clustermesh status --context kind-east --wait


# Enable Cluster Mesh : 간단한 실습 환경으로 NodePort 로 진행
cilium clustermesh enable --service-type NodePort --enable-kvstoremesh=false --context kind-west
cilium clustermesh enable --service-type NodePort --enable-kvstoremesh=false --context kind-east
 

# 32379 NodePort 정보 : clustermesh-apiserver 서비스 정보
kwest get svc,ep -n kube-system clustermesh-apiserver --context kind-west
NAME                            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/clustermesh-apiserver   NodePort   10.2.216.182   <none>        2379:32379/TCP   65s

NAME                              ENDPOINTS         AGE
endpoints/clustermesh-apiserver   10.0.0.195:2379   65s # 대상 파드는 clustermesh-apiserver 파드 IP

kwest get pod -n kube-system -owide | grep clustermesh


keast get svc,ep -n kube-system clustermesh-apiserver --context kind-east
NAME                            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/clustermesh-apiserver   NodePort   10.3.252.188   <none>        2379:32379/TCP   43s

NAME                              ENDPOINTS         AGE
endpoints/clustermesh-apiserver   10.1.0.206:2379   65s # 대상 파드는 clustermesh-apiserver 파드 IP

keast get pod -n kube-system -owide | grep clustermesh


# 모니터링 : 신규 터미널 2개
watch -d "cilium clustermesh status --context kind-west --wait"
watch -d "cilium clustermesh status --context kind-east --wait"


# Connect Clusters
cilium clustermesh connect --context kind-west --destination-context kind-east

# 확인
cilium clustermesh status --context kind-west --wait
cilium clustermesh status --context kind-east --wait

 

# 
kubectl exec -it -n kube-system ds/cilium -c cilium-agent --context kind-west -- cilium-dbg troubleshoot clustermesh
kubectl exec -it -n kube-system ds/cilium -c cilium-agent --context kind-east -- cilium-dbg troubleshoot clustermesh

# 확인
kwest get pod -A && keast get pod -A
cilium status --context kind-west
cilium status --context kind-east
cilium clustermesh status --context kind-west
cilium clustermesh status --context kind-east
cilium config view --context kind-west
cilium config view --context kind-east
kwest exec -it -n kube-system ds/cilium -- cilium status --verbose
keast exec -it -n kube-system ds/cilium -- cilium status --verbose
ClusterMesh:   1/1 remote clusters ready, 0 global-services
   east: ready, 2 nodes, 4 endpoints, 3 identities, 0 services, 0 MCS-API service exports, 0 reconnections (last: never)
   └  etcd: 1/1 connected, leases=0, lock leases=0, has-quorum=true: endpoint status checks are disabled, ID: c6ba18866da7dfd8
   └  remote configuration: expected=true, retrieved=true, cluster-id=2, kvstoremesh=false, sync-canaries=true, service-exports=disabled
   └  synchronization status: nodes=true, endpoints=true, identities=true, services=true

#
helm get values -n kube-system cilium --kube-context kind-west 
...
cluster:
  id: 1
  name: west
clustermesh:
  apiserver:
    kvstoremesh:
      enabled: false
    service:
      type: NodePort
    tls:
      auto:
        enabled: true
        method: cronJob
        schedule: 0 0 1 */4 *
  config:
    clusters:
    - ips:
      - 172.18.0.4
      name: east
      port: 32379
    enabled: true
  useAPIServer: true
...

helm get values -n kube-system cilium --kube-context kind-east 
...
cluster:
  id: 2
  name: east
clustermesh:
  apiserver:
    kvstoremesh:
      enabled: false
    service:
      type: NodePort
    tls:
      auto:
        enabled: true
        method: cronJob
        schedule: 0 0 1 */4 *
  config:
    clusters:
    - ips:
      - 172.18.0.3
      name: west
      port: 32379
    enabled: true
  useAPIServer: true
...


# 라우팅 정보 확인 : 클러스터간 PodCIDR 라우팅 주입 확인!
docker exec -it west-control-plane ip -c route
docker exec -it west-worker ip -c route
docker exec -it east-control-plane ip -c route
docker exec -it east-worker ip -c route

 

hubble enable

# 설정
helm upgrade cilium cilium/cilium --version 1.17.6 --namespace kube-system --reuse-values \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=30001 --kube-context kind-west
kwest -n kube-system rollout restart ds/cilium

## 혹은 cilium hubble enable --ui --relay --context kind-west
## kubectl --context kind-west patch svc -n kube-system hubble-ui -p '{"spec": {"type": "NodePort", "ports": [{"port": 80, "targetPort": 8081, "nodePort": 30001}]}}'


# 설정
helm upgrade cilium cilium/cilium --version 1.17.6 --namespace kube-system --reuse-values \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=31001 --kube-context kind-east
kwest -n kube-system rollout restart ds/cilium

## 혹은 cilium hubble enable --ui --relay --context kind-east
## kubectl --context kind-east patch svc -n kube-system hubble-ui -p '{"spec": {"type": "NodePort", "ports": [{"port": 80, "targetPort": 8081, "nodePort": 31001}]}}'


# 확인
kwest get svc,ep -n kube-system hubble-ui --context kind-west
keast get svc,ep -n kube-system hubble-ui --context kind-east

# hubble-ui 접속 시
open http://localhost:30001
open http://localhost:31001

 

west 파드 ↔ east 파드간 직접 통신 확인 with static routing : 파드간 직접 통신 tcpdump 로 확인

#
cat << EOF | kubectl apply --context kind-west -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF

cat << EOF | kubectl apply --context kind-east -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF


# 확인
kwest get pod -A && keast get pod -A
kwest get pod -owide && keast get pod -owide
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
curl-pod   1/1     Running   0          43s   10.0.0.144   west-control-plane   <none>           <none>
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
curl-pod   1/1     Running   0          36s   10.1.0.128   east-control-plane   <none>           <none>

#
kubectl exec -it curl-pod --context kind-west -- ping -c 1 10.1.0.128
64 bytes from 10.1.0.128: icmp_seq=1 ttl=62 time=0.877 ms

kubectl exec -it curl-pod --context kind-west -- ping 10.1.0.128

# 목적지 파드에서 tcpdump 로 확인 : NAT 없이 직접 라우팅.
kubectl exec -it curl-pod --context kind-east -- tcpdump -i eth0 -nn

# 목적지 k8s 노드?에서 icmp tcpdump 로 확인 : 다른곳 경유하지 않고 직접 노드에서 파드로 인입 확인
docker exec -it east-control-plane tcpdump -i any icmp -nn
docker exec -it east-worker tcpdump -i any icmp -nn

#
kubectl exec -it curl-pod --context kind-east -- ping -c 1 10.0.0.144
64 bytes from 10.0.0.144: icmp_seq=1 ttl=62 time=1.24 ms

 

 

Load-balancing & Service Discovery

#
cat << EOF | kubectl apply --context kind-west -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
  annotations:
    service.cilium.io/global: "true"
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF


#
cat << EOF | kubectl apply --context kind-east -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
  annotations:
    service.cilium.io/global: "true"
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF



# 확인
kwest get svc,ep webpod && keast get svc,ep webpod

kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend               Service Type   Backend
...
13   10.2.37.24:80/TCP       ClusterIP      1 => 10.1.1.89:80/TCP (active)      
                                            2 => 10.0.1.27:80/TCP (active)      
                                            3 => 10.1.1.213:80/TCP (active)     
                                            4 => 10.0.1.19:80/TCP (active) 
                                            
keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend               Service Type   Backend
...
13   10.3.156.11:80/TCP     ClusterIP      1 => 10.1.1.89:80/TCP (active)      
                                           2 => 10.0.1.27:80/TCP (active)      
                                           3 => 10.1.1.213:80/TCP (active)     
                                           4 => 10.0.1.19:80/TCP (active) 
#
kubectl exec -it curl-pod --context kind-west -- ping -c 1 10.1.0.128
kubectl exec -it curl-pod --context kind-east -- ping -c 1 10.0.0.144

#
kubectl exec -it curl-pod --context kind-west -- sh -c 'while true; do curl -s --connect-timeout 1 webpod ; sleep 1; echo "---"; done;'
kubectl exec -it curl-pod --context kind-east -- sh -c 'while true; do curl -s --connect-timeout 1 webpod ; sleep 1; echo "---"; done;'

# 현재 Service Annotations 설정
kwest describe svc webpod | grep Annotations -A1
Annotations:              service.cilium.io/global: true
Selector:                 app=webpod

keast describe svc webpod | grep Annotations -A1
Annotations:              service.cilium.io/global: true
Selector:                 app=webpod


# 모니터링 : 반복 접속해두기
kubectl exec -it curl-pod --context kind-west -- sh -c 'while true; do curl -s --connect-timeout 1 webpod ; sleep 1; echo "---"; done;'


#
kwest scale deployment webpod --replicas 1
kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity

# 로컬 k8s 에 목적지가 없을 경우 어떻게 되나요?
kwest scale deployment webpod --replicas 0
kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity

#
kwest scale deployment webpod --replicas 2
kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity


# tcpdump 확인 : dataplane flow 확인
docker exec -it west-control-plane tcpdump -i any tcp port 80 -nnq 
docker exec -it west-worker tcpdump -i any tcp port 80 -nnq
docker exec -it east-control-plane tcpdump -i any tcp port 80 -nnq 
docker exec -it east-worker tcpdump -i any tcp port 80 -nnq

 

 

 

 

업로드 중입니다