This repository has been archived by the owner on Jun 6, 2024. It is now read-only.
data node and node manager are repeatedly restarted on prod bed #1682
Assignees
Comments
|
Here is the duplicate issue #1658 |
|
data-node down for oom, we need to raise their memory. But node-mananger down for failing to connect resource-mananger, I think the casue is resource-manager down. |
|
@mzmssg , what is the root cause of RM down? also OOM? |
|
@fanyangCS The most possibility is Azure's issue. The processes |
|
close this issue as this is resolved. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
from prod bed.
core@paigcr-a-infra-1001:~$ kubectl get pods NAME READY STATUS RESTARTS AGE alertmanager-79b5488d8f-t8fxw 1/1 Running 0 1d drivers-one-shot-2g7zq 1/1 Running 0 8d drivers-one-shot-2phs9 1/1 Running 0 8d drivers-one-shot-4c2jm 1/1 Running 0 8d drivers-one-shot-4gr6n 1/1 Running 0 8d drivers-one-shot-5kdhf 1/1 Running 0 8d drivers-one-shot-5kmnx 1/1 Running 0 8d drivers-one-shot-5ml5t 1/1 Running 0 8d drivers-one-shot-5n86g 1/1 Running 0 8d drivers-one-shot-5xxnz 1/1 Running 0 8d drivers-one-shot-6rt75 1/1 Running 0 8d drivers-one-shot-74tz9 1/1 Running 0 8d drivers-one-shot-76xsb 1/1 Running 0 8d drivers-one-shot-7s22w 1/1 Running 0 8d drivers-one-shot-85pm5 1/1 Running 0 8d drivers-one-shot-8j9rr 1/1 Running 0 8d drivers-one-shot-8mtlf 1/1 Running 0 8d drivers-one-shot-8nsms 1/1 Running 0 8d drivers-one-shot-942k6 1/1 Running 0 8d drivers-one-shot-9d5wj 1/1 Running 0 8d drivers-one-shot-9dkqw 1/1 Running 0 8d drivers-one-shot-9qmtv 1/1 Running 0 8d drivers-one-shot-9xllj 1/1 Running 0 4d drivers-one-shot-b2k7j 1/1 Running 0 8d drivers-one-shot-bgpbb 1/1 Running 0 8d drivers-one-shot-btklz 1/1 Running 0 8d drivers-one-shot-bzkvt 1/1 Running 0 8d drivers-one-shot-c4bb7 1/1 Running 0 8d drivers-one-shot-cfpxt 1/1 Running 0 8d drivers-one-shot-cldwk 1/1 Running 0 8d drivers-one-shot-czt6j 1/1 Running 0 8d drivers-one-shot-djfcc 1/1 Running 0 8d drivers-one-shot-dvlz5 1/1 Running 0 8d drivers-one-shot-fc2qs 1/1 Running 9 8d drivers-one-shot-fdsm4 1/1 Running 0 8d drivers-one-shot-fpgc5 1/1 Running 0 8d drivers-one-shot-g8sgj 1/1 Running 0 8d drivers-one-shot-gl29g 1/1 Running 0 8d drivers-one-shot-gnkzs 1/1 Running 0 8d drivers-one-shot-jhhfc 1/1 Running 0 8d drivers-one-shot-jwjx9 1/1 Running 0 8d drivers-one-shot-jxbn2 1/1 Running 0 8d drivers-one-shot-krrgp 1/1 Running 0 8d drivers-one-shot-lt2kb 1/1 Running 0 8d drivers-one-shot-m4px6 1/1 Running 0 8d drivers-one-shot-m5qxq 1/1 Running 0 8d drivers-one-shot-m927n 1/1 Running 0 8d drivers-one-shot-m9tjp 1/1 Running 0 8d drivers-one-shot-mbhnh 1/1 Running 0 8d drivers-one-shot-mjjj7 1/1 Running 0 8d drivers-one-shot-n5nt7 1/1 Running 0 8d drivers-one-shot-ndcnp 1/1 Running 0 8d drivers-one-shot-p8p68 1/1 Running 0 8d drivers-one-shot-pvfw4 1/1 Running 0 8d drivers-one-shot-q58dp 1/1 Running 0 8d drivers-one-shot-q5nzq 1/1 Running 0 8d drivers-one-shot-rz2vg 1/1 Running 0 8d drivers-one-shot-s29n8 1/1 Running 0 8d drivers-one-shot-s8tkh 1/1 Running 0 8d drivers-one-shot-sbs8w 1/1 Running 0 8d drivers-one-shot-sgc8n 1/1 Running 0 8d drivers-one-shot-shspz 1/1 Running 0 8d drivers-one-shot-t24qm 1/1 Running 0 8d drivers-one-shot-t4phx 1/1 Running 7 8d drivers-one-shot-tk4wf 1/1 Running 0 8d drivers-one-shot-txrk9 1/1 Running 0 8d drivers-one-shot-vdccp 1/1 Running 0 8d drivers-one-shot-vfpjk 1/1 Running 0 8d drivers-one-shot-vsn8c 1/1 Running 0 8d drivers-one-shot-vtfbm 1/1 Running 1 8d drivers-one-shot-w9sxh 1/1 Running 0 8d drivers-one-shot-xg8vd 1/1 Running 0 8d drivers-one-shot-xlbqb 1/1 Running 0 8d drivers-one-shot-xr8j7 1/1 Running 0 8d drivers-one-shot-xw5lw 1/1 Running 0 8d end-to-end-test-deployment-7dc9d746cd-jhrkd 1/1 Running 0 8d etcd-server-10.151.40.179 1/1 Running 0 23h etcd-server-10.151.40.180 1/1 Running 3 1d etcd-server-10.151.40.181 1/1 Running 3 1d grafana-68f6cc767-v9vxx 1/1 Running 0 1d hadoop-data-node-ds-2fmn2 1/1 Running 0 8d hadoop-data-node-ds-2pws9 1/1 Running 0 8d hadoop-data-node-ds-2tqjd 1/1 Running 0 8d hadoop-data-node-ds-2zcrs 1/1 Running 0 8d hadoop-data-node-ds-4bzdw 1/1 Running 1 8d hadoop-data-node-ds-4f8hg 1/1 Running 0 8d hadoop-data-node-ds-4gkrs 1/1 Running 0 8d hadoop-data-node-ds-4l6jt 1/1 Running 0 8d hadoop-data-node-ds-4tznl 1/1 Running 2 8d hadoop-data-node-ds-5gbzf 1/1 Running 2 8d hadoop-data-node-ds-5krcc 1/1 Running 0 8d hadoop-data-node-ds-6b5ns 1/1 Running 0 8d hadoop-data-node-ds-6df45 1/1 Running 1 8d hadoop-data-node-ds-77q5x 1/1 Running 0 8d hadoop-data-node-ds-7qplm 1/1 Running 1 8d hadoop-data-node-ds-9brpw 1/1 Running 0 8d hadoop-data-node-ds-9vztk 1/1 Running 0 8d hadoop-data-node-ds-b2xrz 1/1 Running 0 8d hadoop-data-node-ds-bk5r6 1/1 Running 0 8d hadoop-data-node-ds-bzkh5 1/1 Running 1 8d hadoop-data-node-ds-ctkx2 1/1 Running 10 8d hadoop-data-node-ds-d6654 1/1 Running 1 8d hadoop-data-node-ds-fzh9s 1/1 Running 0 8d hadoop-data-node-ds-glz6r 1/1 Running 38 8d hadoop-data-node-ds-hbf86 1/1 Running 6 8d hadoop-data-node-ds-hhbg8 1/1 Running 4 8d hadoop-data-node-ds-hp4zg 1/1 Running 3 8d hadoop-data-node-ds-j46bj 1/1 Running 0 8d hadoop-data-node-ds-jk4sd 1/1 Running 0 8d hadoop-data-node-ds-jm2zz 1/1 Running 0 8d hadoop-data-node-ds-js7zs 1/1 Running 1 8d hadoop-data-node-ds-jzgdw 1/1 Running 0 8d hadoop-data-node-ds-jzjvn 1/1 Running 0 8d hadoop-data-node-ds-kb76s 1/1 Running 0 8d hadoop-data-node-ds-kdlqj 1/1 Running 0 8d hadoop-data-node-ds-kj5p7 1/1 Running 1 8d hadoop-data-node-ds-ktwhw 1/1 Running 0 8d hadoop-data-node-ds-kxjth 1/1 Running 2 8d hadoop-data-node-ds-l44wk 1/1 Running 0 8d hadoop-data-node-ds-l664d 1/1 Running 1 8d hadoop-data-node-ds-m24b9 1/1 Running 0 8d hadoop-data-node-ds-m74l2 1/1 Running 0 8d hadoop-data-node-ds-nch46 1/1 Running 37 8d hadoop-data-node-ds-nkbkh 1/1 Running 0 8d hadoop-data-node-ds-nl58x 1/1 Running 0 8d hadoop-data-node-ds-npxhn 1/1 Running 0 8d hadoop-data-node-ds-nshtc 1/1 Running 40 8d hadoop-data-node-ds-pbc9t 1/1 Running 23 8d hadoop-data-node-ds-pcj68 1/1 Running 0 8d hadoop-data-node-ds-pgbl4 1/1 Running 0 8d hadoop-data-node-ds-q5rqp 1/1 Running 19 8d hadoop-data-node-ds-qcxjh 1/1 Running 0 8d hadoop-data-node-ds-qgvvm 1/1 Running 0 8d hadoop-data-node-ds-qwdhn 1/1 Running 0 8d hadoop-data-node-ds-smrdl 1/1 Running 351 8d hadoop-data-node-ds-spbw9 1/1 Running 16 8d hadoop-data-node-ds-t2kqd 1/1 Running 0 8d hadoop-data-node-ds-t88c9 1/1 Running 23 8d hadoop-data-node-ds-tkhwd 1/1 Running 0 8d hadoop-data-node-ds-v8twh 1/1 Running 0 8d hadoop-data-node-ds-vjb4j 1/1 Running 3 8d hadoop-data-node-ds-vjfdw 1/1 Running 5 8d hadoop-data-node-ds-vxxlr 1/1 Running 1 8d hadoop-data-node-ds-w86gz 1/1 Running 4 8d hadoop-data-node-ds-w8wk9 1/1 Running 1 8d hadoop-data-node-ds-wjbwc 1/1 Running 48 8d hadoop-data-node-ds-wm5rc 1/1 Running 0 8d hadoop-data-node-ds-ws4dv 1/1 Running 7 8d hadoop-data-node-ds-wz7tq 1/1 Running 0 8d hadoop-data-node-ds-xvg69 1/1 Running 52 8d hadoop-jobhistory-service-bv6f2 1/1 Running 4 8d hadoop-name-node-ds-76kfx 1/1 Running 0 8d hadoop-node-manager-ds-2794b 1/1 Running 9 8d hadoop-node-manager-ds-49k45 1/1 Running 10 8d hadoop-node-manager-ds-4f8wv 1/1 Running 9 8d hadoop-node-manager-ds-4j88c 1/1 Running 10 8d hadoop-node-manager-ds-4pqcp 1/1 Running 9 8d hadoop-node-manager-ds-56sl5 1/1 Running 9 8d hadoop-node-manager-ds-5bqxc 1/1 Running 9 8d hadoop-node-manager-ds-6blkn 1/1 Running 10 8d hadoop-node-manager-ds-6tj4h 1/1 Running 10 8d hadoop-node-manager-ds-6vbhx 1/1 Running 10 8d hadoop-node-manager-ds-6wfls 1/1 Running 9 8d hadoop-node-manager-ds-86bq4 1/1 Running 9 8d hadoop-node-manager-ds-8rgn7 1/1 Running 11 8d hadoop-node-manager-ds-98v8k 1/1 Running 10 8d hadoop-node-manager-ds-9gvgn 1/1 Running 10 8d hadoop-node-manager-ds-bf4kk 1/1 Running 9 8d hadoop-node-manager-ds-bg646 1/1 Running 10 8d hadoop-node-manager-ds-bh7mv 1/1 Running 9 8d hadoop-node-manager-ds-c4bn7 1/1 Running 10 8d hadoop-node-manager-ds-dhl2j 1/1 Running 10 8d hadoop-node-manager-ds-gbghk 1/1 Running 9 8d hadoop-node-manager-ds-gqrbx 1/1 Running 9 8d hadoop-node-manager-ds-gtxzx 1/1 Running 10 8d hadoop-node-manager-ds-gvcjk 1/1 Running 10 8d hadoop-node-manager-ds-h4b92 1/1 Running 10 8d hadoop-node-manager-ds-h7jtc 1/1 Running 10 8d hadoop-node-manager-ds-hfsz2 1/1 Running 9 8d hadoop-node-manager-ds-hsnfl 1/1 Running 10 8d hadoop-node-manager-ds-hwk4d 1/1 Running 9 8d hadoop-node-manager-ds-j2gpw 1/1 Running 9 8d hadoop-node-manager-ds-j4jhb 1/1 Running 9 8d hadoop-node-manager-ds-jp5cs 1/1 Running 9 8d hadoop-node-manager-ds-jskhk 1/1 Running 9 8d hadoop-node-manager-ds-jztg8 1/1 Running 9 8d hadoop-node-manager-ds-k5rxf 1/1 Running 9 8d hadoop-node-manager-ds-kc46j 1/1 Running 9 8d hadoop-node-manager-ds-kp9th 1/1 Running 10 8d hadoop-node-manager-ds-ksv9m 1/1 Running 10 8d hadoop-node-manager-ds-kz4jb 1/1 Running 9 8d hadoop-node-manager-ds-ldzbv 1/1 Running 10 8d hadoop-node-manager-ds-lk9lh 1/1 Running 9 8d hadoop-node-manager-ds-m5hwq 1/1 Running 10 8d hadoop-node-manager-ds-mk24n 1/1 Running 9 8d hadoop-node-manager-ds-mrmg9 1/1 Running 9 8d hadoop-node-manager-ds-n68sj 1/1 Running 10 8d hadoop-node-manager-ds-nclhm 1/1 Running 9 8d hadoop-node-manager-ds-ns895 1/1 Running 10 8d hadoop-node-manager-ds-pj8p8 1/1 Running 10 8d hadoop-node-manager-ds-q5bgq 1/1 Running 9 8d hadoop-node-manager-ds-r6ph2 1/1 Running 9 8d hadoop-node-manager-ds-rgjb2 1/1 Running 9 8d hadoop-node-manager-ds-rmmx7 1/1 Running 9 8d hadoop-node-manager-ds-rspc4 1/1 Running 10 8d hadoop-node-manager-ds-sr5rb 1/1 Running 10 8d hadoop-node-manager-ds-swct8 1/1 Running 9 8d hadoop-node-manager-ds-t22b4 1/1 Running 10 8d hadoop-node-manager-ds-t8487 1/1 Running 9 8d hadoop-node-manager-ds-tgfnw 1/1 Running 10 8d hadoop-node-manager-ds-tjlbw 1/1 Running 10 8d hadoop-node-manager-ds-tm86p 1/1 Running 10 8d hadoop-node-manager-ds-v5npv 1/1 Running 10 8d hadoop-node-manager-ds-vbpbm 1/1 Running 10 8d hadoop-node-manager-ds-vxdmp 1/1 Running 10 8d hadoop-node-manager-ds-vxgzk 1/1 Running 10 8d hadoop-node-manager-ds-xt9dw 1/1 Running 9 8d hadoop-node-manager-ds-zmq4v 1/1 Running 10 8d hadoop-node-manager-ds-zqp4z 1/1 Running 10 8d hadoop-node-manager-ds-ztw4w 1/1 Running 9 8d hadoop-node-manager-ds-zvmcd 1/1 Running 10 8d hadoop-node-manager-ds-zxpvr 1/1 Running 9 8d hadoop-resource-manager-ds-wb8hx 2/2 Running 4 8d node-exporter-2fzpq 2/2 Running 0 8d node-exporter-2ml29 2/2 Running 0 8d node-exporter-2tkk8 2/2 Running 6 8d node-exporter-2vpzr 2/2 Running 0 8d node-exporter-486kf 2/2 Running 0 8d node-exporter-48jkz 2/2 Running 0 8d node-exporter-48wzt 2/2 Running 0 1d node-exporter-4chzc 2/2 Running 0 8d node-exporter-4rdvr 2/2 Running 0 8d node-exporter-4sbwk 2/2 Running 0 8d node-exporter-5tnjm 2/2 Running 0 8d node-exporter-65pq6 2/2 Running 0 8d node-exporter-6hfhx 2/2 Running 0 8d node-exporter-77xbb 2/2 Running 0 8d node-exporter-7hk46 2/2 Running 0 8d node-exporter-7kvz5 2/2 Running 0 8d node-exporter-7nvhq 2/2 Running 2 8d node-exporter-7rl8k 2/2 Running 0 8d node-exporter-7s4kx 2/2 Running 0 8d node-exporter-8krxc 2/2 Running 0 8d node-exporter-9bp79 2/2 Running 0 8d node-exporter-bbf4d 2/2 Running 0 8d node-exporter-bqdbn 2/2 Running 0 8d node-exporter-cp9kh 2/2 Running 0 8d node-exporter-d9fnq 2/2 Running 0 8d node-exporter-dxcdf 2/2 Running 0 8d node-exporter-dz98p 2/2 Running 0 8d node-exporter-fdpkm 2/2 Running 0 8d node-exporter-g6wv2 2/2 Running 8 8d node-exporter-g7dvg 2/2 Running 0 8d node-exporter-gdbql 2/2 Running 0 8d node-exporter-gng47 2/2 Running 0 8d node-exporter-h8ssb 2/2 Running 0 8d node-exporter-h8zbq 2/2 Running 0 8d node-exporter-hnlsg 2/2 Running 0 8d node-exporter-hrm6f 2/2 Running 0 8d node-exporter-hwxbf 2/2 Running 0 8d node-exporter-j9wrm 2/2 Running 0 8d node-exporter-jjr76 2/2 Running 0 8d node-exporter-kd2h4 2/2 Running 0 8d node-exporter-kgrvx 2/2 Running 0 8d node-exporter-ldstb 2/2 Running 0 8d node-exporter-lfw6q 2/2 Running 0 8d node-exporter-mqlsm 2/2 Running 0 8d node-exporter-mvj4p 2/2 Running 0 8d node-exporter-n922c 2/2 Running 0 8d node-exporter-nfmm9 2/2 Running 0 8d node-exporter-p8f5q 2/2 Running 0 8d node-exporter-pb8s2 2/2 Running 0 8d node-exporter-qhbjn 2/2 Running 0 8d node-exporter-qtq2j 2/2 Running 0 8d node-exporter-qtrlg 2/2 Running 0 8d node-exporter-r5ksc 2/2 Running 0 8d node-exporter-r7jr6 2/2 Running 0 8d node-exporter-r8ltt 2/2 Running 0 8d node-exporter-rbhpc 2/2 Running 0 8d node-exporter-rwr8p 2/2 Running 0 8d node-exporter-rwzh9 2/2 Running 0 8d node-exporter-s7bn9 2/2 Running 0 8d node-exporter-sfbll 2/2 Running 0 8d node-exporter-t2bwj 2/2 Running 0 8d node-exporter-t9gdx 2/2 Running 0 8d node-exporter-vrbrv 2/2 Running 0 8d node-exporter-ws2jg 2/2 Running 0 8d node-exporter-wvv6t 2/2 Running 0 8d node-exporter-xb4zc 2/2 Running 0 8d node-exporter-xbk8q 2/2 Running 0 8d node-exporter-xkbks 2/2 Running 0 8d node-exporter-xnj85 2/2 Running 0 8d node-exporter-xsdtf 2/2 Running 0 8d node-exporter-xsdv6 2/2 Running 0 8d node-exporter-xxhqb 2/2 Running 0 8d node-exporter-zsm5x 2/2 Running 0 8d node-exporter-zzq2t 2/2 Running 0 8d prometheus-deployment-67bf958dcb-p7qm6 1/1 Running 0 1d pylon-ds-k87k6 1/1 Running 0 8d rest-server-ds-w542d 1/1 Running 6 8d watchdog-7f85777f97-tp6m7 1/1 Running 0 1d webportal-ds-vth89 1/1 Running 0 3d yarn-frameworklauncher-ds-clpcx 1/1 Running 4 4d zookeeper-ds-9cgth 1/1 Running 0 1dmost of the data node and node manager has been restarted for more than 10 times.
Pod hadoop-data-node-ds-smrdl on 175 has been restarted for 351 times.
On 175 journal log:
killing is happening almost every day. Can we give a large memory size for those two services?
The text was updated successfully, but these errors were encountered: