Redian新闻
>
k8s集群中namespace状态一直显示Terminating如何解决?

k8s集群中namespace状态一直显示Terminating如何解决?

公众号新闻

一、问题现象

    今天在做测试时,在一个namespace下无法启动pod,查看ns状态一直显示Terminating

[root@node1 ~]# kubectl get nsNAME                   STATUS        AGEconfigmap              Terminating   135ddefault                Active        207dharbor                 Active        207dkube-flannel           Terminating   17mkube-node-lease        Active        207dkube-public            Active        207dkube-system            Active        207dkubekey-system         Active        207dkubernetes-dashboard   Active        207dlocal-path-storage     Active        187dnginx                  Active        146dtest                   Terminating   126d

使用–force 删除也是一直卡着的状态

[root@node1 ~]# kubectl delete ns test --force Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.namespace "test" force deleted



二、查看ns下的资源

    根据以上现象,怀疑是该ns下有未释放的资源,使用如下命令查看

1: 查看所有资源[root@node1 ~]# kubectl get all -n  testNo resources found in test namespace.[root@node1 ~]# 以上得知该ns下没有任何未释放的资源
##################我看网上有使用kubectl api-resources -o name --verbs=list --namespaced | xargs -n 1 kubectl get --show-kind --ignore-not-found -n test这个命令查看的,也是返回结果为空


三、问题处理

    以json格式导出ns的详细信息

[root@node1 ~]# kubectl get  ns test -o json  > test.json

    编辑test.json文件,确保spec中内容为空,如下:

    "spec": {        "finalizers": [    #########            "kubernetes"   ######### 删除这三行内容,告知k8s要删除的ns中内容为空        ]                  #########    },

将空ns通过调用k8s的api接口覆盖掉原来的ns



[root@node1 ~]# curl -k \> -H "Content-Type: application/json" \> -X PUT \> --data-binary @test.json \> http://127.0.0.1:8081/api/v1/namespaces/test/finalizecurl: (7) Failed connect to 127.0.0.1:8081; Connection refused

因为k8s主节点使用了认证,如果直接使用命令会拒绝连接,需要使用kube-proxy进行代理8081端口

使用kube-proxy开启端口

[root@node1 ~]# kubectl proxy --port=8081 Starting to serve on 127.0.0.1:8081

打开新的终端,再次执行上述命令,返回如下内容

[root@node1 ~]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @test.json http://127.0.0.1:8081/api/v1/namespaces/test/finalize{  "kind": "Namespace",  "apiVersion": "v1",  "metadata": {    "name": "test",    "uid": "f2676c45-a75b-49be-9e01-84958bedc4a0",    "resourceVersion": "33942216",    "creationTimestamp": "2023-06-28T06:02:07Z",    "deletionTimestamp": "2023-06-28T06:02:24Z",    "labels": {      "kubernetes.io/metadata.name": "test"    },    "annotations": {      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"creationTimestamp\":\"2023-06-28T06:02:07Z\",\"deletionTimestamp\":\"2023-06-28T06:02:24Z\",\"labels\":{\"kubernetes.io/metadata.name\":\"test\"},\"name\":\"test\",\"resourceVersion\":\"28081915\",\"uid\":\"f2676c45-a75b-49be-9e01-84958bedc4a0\"},\"spec\":{},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-06-28T06:02:29Z\",\"message\":\"Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: an error on the server (\\\"Internal Server Error: \\\\\\\"/apis/metrics.k8s.io/v1beta1\\\\\\\": the server could not find the requested resource\\\") has prevented the request from succeeding\",\"reason\":\"DiscoveryFailed\",\"status\":\"True\",\"type\":\"NamespaceDeletionDiscoveryFailure\"},{\"lastTransitionTime\":\"2023-06-28T06:02:29Z\",\"message\":\"All legacy kube types successfully parsed\",\"reason\":\"ParsedGroupVersions\",\"status\":\"False\",\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"},{\"lastTransitionTime\":\"2023-06-28T06:02:29Z\",\"message\":\"All content successfully deleted, may be waiting on finalization\",\"reason\":\"ContentDeleted\",\"status\":\"False\",\"type\":\"NamespaceDeletionContentFailure\"},{\"lastTransitionTime\":\"2023-06-28T06:02:29Z\",\"message\":\"All content successfully removed\",\"reason\":\"ContentRemoved\",\"status\":\"False\",\"type\":\"NamespaceContentRemaining\"},{\"lastTransitionTime\":\"2023-06-28T06:02:29Z\",\"message\":\"All content-preserving finalizers finished\",\"reason\":\"ContentHasNoFinalizers\",\"status\":\"False\",\"type\":\"NamespaceFinalizersRemaining\"}],\"phase\":\"Terminating\"}}\n"    },    "managedFields": [      {        "manager": "kubectl-create",        "operation": "Update",        "apiVersion": "v1",        "time": "2023-06-28T06:02:07Z",        "fieldsType": "FieldsV1",        "fieldsV1": {          "f:metadata": {            "f:labels": {              ".": {},              "f:kubernetes.io/metadata.name": {}            }          }        }      },      {        "manager": "kube-controller-manager",        "operation": "Update",        "apiVersion": "v1",        "time": "2023-09-27T07:02:23Z",        "fieldsType": "FieldsV1",        "fieldsV1": {          "f:status": {            "f:conditions": {              ".": {},              "k:{\"type\":\"NamespaceContentRemaining\"}": {                ".": {},                "f:lastTransitionTime": {},                "f:message": {},                "f:reason": {},                "f:status": {},                "f:type": {}              },              "k:{\"type\":\"NamespaceDeletionContentFailure\"}": {                ".": {},                "f:lastTransitionTime": {},                "f:message": {},                "f:reason": {},                "f:status": {},                "f:type": {}              },              "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {                ".": {},                "f:lastTransitionTime": {},                "f:message": {},                "f:reason": {},                "f:status": {},                "f:type": {}              },              "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {                ".": {},                "f:lastTransitionTime": {},                "f:message": {},                "f:reason": {},                "f:status": {},                "f:type": {}              },              "k:{\"type\":\"NamespaceFinalizersRemaining\"}": {                ".": {},                "f:lastTransitionTime": {},                "f:message": {},                "f:reason": {},                "f:status": {},                "f:type": {}              }            }          }        },        "subresource": "status"      },      {        "manager": "kubectl-client-side-apply",        "operation": "Update",        "apiVersion": "v1",        "time": "2023-11-02T02:46:01Z",        "fieldsType": "FieldsV1",        "fieldsV1": {          "f:metadata": {            "f:annotations": {              ".": {},              "f:kubectl.kubernetes.io/last-applied-configuration": {}            }          }        }      }    ]  },  "spec": {},  "status": {    "phase": "Terminating",    "conditions": [      {        "type": "NamespaceDeletionDiscoveryFailure",        "status": "True",        "lastTransitionTime": "2023-06-28T06:02:29Z",        "reason": "DiscoveryFailed",        "message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: an error on the server (\"Internal Server Error: \\\"/apis/metrics.k8s.io/v1beta1\\\": the server could not find the requested resource\") has prevented the request from succeeding"      },      {        "type": "NamespaceDeletionGroupVersionParsingFailure",        "status": "False",        "lastTransitionTime": "2023-06-28T06:02:29Z",        "reason": "ParsedGroupVersions",        "message": "All legacy kube types successfully parsed"      },      {        "type": "NamespaceDeletionContentFailure",        "status": "False",        "lastTransitionTime": "2023-06-28T06:02:29Z",        "reason": "ContentDeleted",        "message": "All content successfully deleted, may be waiting on finalization"      },      {        "type": "NamespaceContentRemaining",        "status": "False",        "lastTransitionTime": "2023-06-28T06:02:29Z",        "reason": "ContentRemoved",        "message": "All content successfully removed"      },      {        "type": "NamespaceFinalizersRemaining",        "status": "False",        "lastTransitionTime": "2023-06-28T06:02:29Z",        "reason": "ContentHasNoFinalizers",        "message": "All content-preserving finalizers finished"      }    ]  }}

查看ns是否被删除

[root@node1 ~]#kubectl get ns    ####名为test的ns已经被删除 NAME                   STATUS        AGEconfigmap              Terminating   135ddefault                Active        207dharbor                 Active        207dkube-flannel           Terminating   21mkube-node-lease        Active        207dkube-public            Active        207dkube-system            Active        207dkubekey-system         Active        207dkubernetes-dashboard   Active        207dlocal-path-storage     Active        187dnginx                  Active        146d
其他Terminating状态的ns也可以使用上述方法删除#######################注意,ns的名字不要用和资源类型一样,要不然会把ns名称识别成资源类型报错。上面名为configmap的ns就是这个问题,导致删除报错无法清理,只能调用etcd进行删除。

链接:https://blog.csdn.net/weixin_40579389/article/details/134186164?spm=1001.2014.3001.5502

(版权归原作者所有,侵删)


微信扫码关注该文公众号作者

戳这里提交新闻线索和高质量文章给我们。
相关阅读
k8s 多网卡方案之multus用法使用 kube-downscaler 降低Kubernetes集群成本K8s部署Jumpserver并使用Istio对外暴露服务【解字】安全一篇可供参考的 K8S 落地实践经验领先船不是领导船Amid Global Warming, El Niño Threatens Uncertain Winter in China弃亚马逊转戴尔,彻底下云、去 K8s 后,我们已经节省了 100 万美元当K8s出现问题时,我们可以从哪些方面排查出放弃亚马逊、谷歌,彻底告别 k8s!NeurIPS 2023 | MCUFormer: 可在内存有限微控制器上部署的Vison Transformer模型冰岛伊萨菲厄泽(Isafjordur),山山相依AI具体可以解决哪些营销和运营的问题?如何解决?Kubernetes 实战:使用 k8s+jenkins 实现 CICD火了132年!经典老牌𝙐𝙎𝙋𝘼(美国马球协会)打底衫来了!上身就是高级感,真香!画了一年多的油画《打球去》终于完成啦!KEDA:基于事件驱动扩展K8S应用的深度实践弃亚马逊转戴尔,彻底下云、去K8s后,我们已经节省了100万美元K8s多集群实践思考和探索Providing Long-Term Care for Shanghai’s Most Vulnerable ResidentJava近期新闻:Spring Framework 6.1、Spring Data 2023.1、Payara PlatformK8s 日常运维故障处理,80% 你可能都遇见过!?诺奖加持,量子点直显时代来临!利亚德携手Saphlux将诺奖技术带进下一代新型显示K8s留给我们一地鸡毛!裁判文书上网公开存在哪些问题?如何解决?Space Race: Why Young Chinese Are Cutting Ties with Relatives火了132年!经典老牌𝙐𝙎𝙋𝘼(美国马球协会)打底衫来了!3折抢!真香花卉摄影,静听绽放伦敦时装周中韩明星状态一言难尽!陈坤逐渐“周迅化”?女神全智贤初现老态!通俗易懂k8s——服务的注册与发现通俗易懂k8s——核心组件深入解析k8s 网络插件—AntreaNeurIPS 2023 | 结合脉冲神经网络和Transformer的纯加法Transformer云原生面经分享:精心整理的 K8s / Docker / DevOps 面试真题!| 极客时间NeurIPS 2023 | MixFormerV2:基于Transformer的高效跟踪器
logo
联系我们隐私协议©2024 redian.news
Redian新闻
Redian.news刊载任何文章,不代表同意其说法或描述,仅为提供更多信息,也不构成任何建议。文章信息的合法性及真实性由其作者负责,与Redian.news及其运营公司无关。欢迎投稿,如发现稿件侵权,或作者不愿在本网发表文章,请版权拥有者通知本网处理。