k8s HA 구성중 kubeadm init 실패 증상 - serv... - 인프런

_{k8s HA 구성을 하고 있는데요. kubeadm init 실패를 하고 있는 상황이고 해결방법을 찾지 못하고 있습니다.}

_{- 문제가 발생하고 있는 서버에서 kubeadm init를 실행하면 아래 로그와 같이 serviceaccount 부분에서 500 에러가 발생합니다.}

_{- 동일한 작업을 local에서 하면 문제가 없는데요}

_{- 둘다 hyper-v에서 우분투 서버를 생성하고 작업했고, 작업 내용을 동일합니다.}

_-serviceaccount 부분에서 500 에러가 발생하기 이전까지는 201 응답을 받고 정상 작업을 진행했는데 원인과 해결방법을 찾을 수 있을까요?

_{1. kubeadm init 실패 로그}

_{-------------}

request.go:1073] Request Body: {"kind":"ServiceAccount","apiVersion":"v1","metadata":{"name":"coredns","namespace":"kube-system","creationTimestamp":null}}
round_trippers.go:466] curl -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubeadm/v1.24.1 (linux/amd64) kubernetes/3ddd0f4" 'https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s'
round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s 500 Internal Server Error in 138 milliseconds
round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 137 ms Duration 138 ms
round_trippers.go:577] Response Headers:
round_trippers.go:580]     Date: Wed, 15 Jun 2022 15:49:22 GMT
round_trippers.go:580]     Audit-Id: 58b00d5f-5f4f-4ab2-bbb3-c3479189f4bc
round_trippers.go:580]     Cache-Control: no-cache, private
round_trippers.go:580]     Content-Type: application/json
round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 4383fc49-8e46-40c9-8a5e-7031390e956f
round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 5e6fba37-152d-4f54-bbf7-e6829d7f15b1
round_trippers.go:580]     Content-Length: 169
request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Unknown desc = malformed header: missing HTTP content-type","code":500}
rpc error: code = Unknown desc = malformed header: missing HTTP content-type
unable to create serviceaccount

안녕하세요.

답변주신 haproxy 가 쿠버네티스 내부에 내장되어 있을까요?

처음에 HA를 위해 haproxy 서버를 따로 두어 구축하고 있었는데 오류가 발생하자 master1 서버에서 kubeadm init 으로만 실행해도 동일한 오류가 발생했습니다.

첨부한 로그에 적혀있는 10.1.10.101는 master1 서버의 IP입니다.

I0615 15:49:22.038672    1224 request.go:1073] Request Body: {"kind":"ClusterRoleBinding","apiVersion":"rbac.authorization.k8s.io/v1","metadata":{"name":"system:coredns","creationTimestamp":null},"subjects":[{"kind":"ServiceAccount","name":"coredns","namespace":"kube-system"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system:coredns"}}
I0615 15:49:22.038725    1224 round_trippers.go:466] curl -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubeadm/v1.24.1 (linux/amd64) kubernetes/3ddd0f4" 'https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s'
I0615 15:49:22.063041    1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 24 milliseconds
I0615 15:49:22.063060    1224 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 24 ms Duration 24 ms
I0615 15:49:22.063070    1224 round_trippers.go:577] Response Headers:
I0615 15:49:22.063077    1224 round_trippers.go:580]     Audit-Id: f52a46cb-6ecb-4b79-aa62-1a22300a8d07
I0615 15:49:22.063084    1224 round_trippers.go:580]     Cache-Control: no-cache, private
I0615 15:49:22.063090    1224 round_trippers.go:580]     Content-Type: application/json
I0615 15:49:22.063096    1224 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 4383fc49-8e46-40c9-8a5e-7031390e956f
I0615 15:49:22.063102    1224 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 5e6fba37-152d-4f54-bbf7-e6829d7f15b1
I0615 15:49:22.063109    1224 round_trippers.go:580]     Content-Length: 604
I0615 15:49:22.063116    1224 round_trippers.go:580]     Date: Wed, 15 Jun 2022 15:49:22 GMT
I0615 15:49:22.063503    1224 request.go:1073] Response Body: {"kind":"ClusterRoleBinding","apiVersion":"rbac.authorization.k8s.io/v1","metadata":{"name":"system:coredns","uid":"133b3a67-dfac-4244-b1a7-0f27a314cb58","resourceVersion":"266","creationTimestamp":"2022-06-15T15:49:22Z","managedFields":[{"manager":"kubeadm","operation":"Update","apiVersion":"rbac.authorization.k8s.io/v1","time":"2022-06-15T15:49:22Z","fieldsType":"FieldsV1","fieldsV1":{"f:roleRef":{},"f:subjects":{}}}]},"subjects":[{"kind":"ServiceAccount","name":"coredns","namespace":"kube-system"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system:coredns"}}
I0615 15:49:22.063860    1224 request.go:1073] Request Body: {"kind":"ServiceAccount","apiVersion":"v1","metadata":{"name":"coredns","namespace":"kube-system","creationTimestamp":null}}
I0615 15:49:22.063921    1224 round_trippers.go:466] curl -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubeadm/v1.24.1 (linux/amd64) kubernetes/3ddd0f4" 'https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s'
I0615 15:49:22.201998    1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s 500 Internal Server Error in 138 milliseconds
I0615 15:49:22.202021    1224 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 137 ms Duration 138 ms
I0615 15:49:22.202028    1224 round_trippers.go:577] Response Headers:
I0615 15:49:22.202035    1224 round_trippers.go:580]     Date: Wed, 15 Jun 2022 15:49:22 GMT
I0615 15:49:22.202042    1224 round_trippers.go:580]     Audit-Id: 58b00d5f-5f4f-4ab2-bbb3-c3479189f4bc
I0615 15:49:22.202048    1224 round_trippers.go:580]     Cache-Control: no-cache, private
I0615 15:49:22.202054    1224 round_trippers.go:580]     Content-Type: application/json
I0615 15:49:22.202060    1224 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 4383fc49-8e46-40c9-8a5e-7031390e956f
I0615 15:49:22.202128    1224 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 5e6fba37-152d-4f54-bbf7-e6829d7f15b1
I0615 15:49:22.202134    1224 round_trippers.go:580]     Content-Length: 169
I0615 15:49:22.202152    1224 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Unknown desc = malformed header: missing HTTP content-type","code":500}
rpc error: code = Unknown desc = malformed header: missing HTTP content-type

missing HTTP content-type오류가 발생한 시점의 로그 일부입니다.
앞서 500 오류 전에는 201가 발생한 작업이 몇개 있습니다.

1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 108 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s 201 Created in 2 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s 201 Created in 16 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 20 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s 201 Created in 49 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s 201 Created in 167 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/secrets?timeout=10s 201 Created in 182 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?timeout=10s 201 Created in 10 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 18 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 36 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 109 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 50 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-public/configmaps?timeout=10s 201 Created in 223 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/roles?timeout=10s 201 Created in 12 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/rolebindings?timeout=10s 201 Created in 20 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 38 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?timeout=10s 201 Created in 16 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 24 milliseconds

1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s 500 Internal Server Error in 138 milliseconds

----

결국 ubuntu server 18.04 환경에서 kubeadm init가 성공하여 HA 구성을 완료하였습니다.

아직도 의문인게
local hyper-v에서는 ubuntu server 22 에서도 kubeadm init가 정상동작했습니다.
하지만 원격 hyper-v 서버는 ubuntu server 18 에서 kubeadm init가 왜 성공 했는지 원인을 찾지 못한 상태입니다.

인프런 커뮤니티 질문&답변

k8s HA 구성중 kubeadm init 실패 증상 - serviceaccount 부분부터 실패