Job keep waiting, can't submit to YARN

[hao1939]

A lot of jobs in 'waiting':

Only the first one submitted to yarn:

[yqwang-ms]

The first job submission cannot be finished due to RM issues, so blocked the following submission.
However, even if we try to submit other jobs, they still cannot be submitted in time, so waiting is expected.
It is a platform issue (RM abnormal), DRI should handle this, and we should have watchdog for RM or Launcher for this case.

The YarnClient looping on this:
2018-09-04 13:08:03,770 INFO [pool-1-thread-1] org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Application submission is not finished, submitted application application_1536066266746_0001 is still in NEW_SAVING

[hao1939]

Hi @yqwang-ms ,

Could we raise a timeout error after a period of time waiting submitting? (I'm not against to figure out
and fix the root cause.)

My concern is that no matter what happens to YRAN or ZK or anything, should we keep user waiting forever? Especially there is no way to succeed.

[yqwang-ms]

It is YARN keep waiting forever, we should first to let YARN have a timeout instead of first make a workaround on top of it.
It is a livesite issue, launcher may help, but DRI / watchdog also need to notify customer.

[yqwang-ms]

YARN have a configuration to set the timeout, you can leverage it:
/**

• The duration that the yarn client library waits, cumulatively across polls,
• for an expected state change to occur. Defaults to -1, which indicates no
• limit.
  */
  public static final String YARN_CLIENT_APPLICATION_CLIENT_PROTOCOL_POLL_TIMEOUT_MS =
  YARN_PREFIX + "client.application-client-protocol.poll-timeout-ms";
  public static final long DEFAULT_YARN_CLIENT_APPLICATION_CLIENT_PROTOCOL_POLL_TIMEOUT_MS =
  -1;

[mzmssg]

Restart RM can fix it, but exception is in zookeeper log, need further investigation

[fanyangCS]

RM issue.