[machine maintain]add machine and remove machine issue

[YanjieGao]

i follow to add and remove machine 10.0.0.7 to existing p100 cluster (v0.8.1 pai) but meet problem:
https://github.com/Microsoft/pai/blob/pai-0.8.y/docs/paictl/paictl-manual.md#Machine_Nodelist_Example

For add:

machine-list:

• hostname: next-a-gpu-0001
  hostip: 10.0.0.7
  machine-type: GENERIC
  k8s-role: worker
  pai-worker: "true"
  sshport: 22
  username: xx
  password: xx

But datanode and namenode IP address is error:
datanode

namenode log:

Driver meet rediness probe error, log seems right:

For remove machine: meet error

[ydye]

Node-manager issue:

1. Add the new node to the cluster-configuraiton
2. refresh the service cluster-configuraiton
3. refresh the service hadoop-data-node and hadoop-node-manager

please refer to this:
#253

drivers issue:
More detailed please.

remove problems: kind a bug

[YanjieGao]

for nm issue from your #253 suggestion:

hostmame is the same with machinelist.yaml hostname:

root@paidevbox:~/PaiDeployment# cd pai-cluster-msr-next-p100/

Currently i want to remove this node and try to deploy but also meet remove node error.... above

[YanjieGao]

For @253 i guess maybe the reason is i directly use config file and not gen config replace this lead to this issue. But this assumption currently not mentioned on our doc. If you have method tell me to remove this node and not throw exception, i could try to redeploy this.

https://github.com/Microsoft/pai/blob/3f7190dd185c75ca42a38a03a3c9def172798106/src/hadoop-data-node/deploy/hadoop-data-node-configuration/hdfs-site.xml#L271

[YanjieGao]

Yundong's fix for k8s remove: #1864