Pure K8S Alpha Release Plan

[abuccts]

Alpha release for k8s based PAI

Code complete date: Nov. 12

Plan Items

• RBAC enable K8S + KubeSpray [kubespray] Doc in contribute to guide user deploy k8s through kubespray #3757 @ydye (pending end-to-end test)
• HiveD test and fix
• Front-end changes [Rest Server] Update hived config validation #3812 @yqwang-ms @abuccts
• Job history Add api endpoint and webportal page of job retry history #3831 @debuggy
• User profile page [Web Portal] User profile page #3804 @sunqinzheng
• Token API [Rest Server] Application token api #3774 [Web Portal] fix update user password request #3834 [Web Portal] revoke browser tokens when user changes password / logout #3835

Deferred

• Internal storage and database Add limited internal storage and postgresql #3813 @hzy46

Finished

• Service Deployment:
  • Framework controller Add service deployment for framework controller #3435
  • Scheduler Add service deployment for hived scheduler #3495
  • Choose service list [Deployment] Support to manage a list of services in paictl #3432
  • Choose launcher type [Deployment] Choose services for different cluster type #3528
• Authentication (Azure Active Directory)
  • Login
  • Access control
• Webportal
  • Incompatible task role name; Cannot copy token
  • ❗️❗️ FC will crash if init container exit with error @yqwang-ms (use the private image for INT test first, switching to released FC later)
• ❗️ grafana cannot retrieve job status due to naming convention - Grafana no show job metrics in pure k8s version #3546
• ❗️ some marketplace jobs broken - Fix job issues on k8s based PAI #3555
• ❗️ wrong resources calculation - [Rest Server] Fix bug in hived resources calculation #3595 @abuccts
• failed to fetch stdout, stderr, and "full log" in job page after the job completes failed to fetch stdout, stderr, and "full log" in job page after the job completes #3576
• ❗️ Job Error Spec, Collection and Handling - [PureK8S]: PAI Job Error Spec, Collection and Handling #3428 @Binyang2014
• ❗️ job shown succeeded even it failed fix bug: job always show success even it failed #3556
• previous job config needs to add ssh plugin to use ssh - [pure k8s] Add SSH config to webportal for pure-k8s based PAI #3596 @wangdian
• ❗️ runtime breaks gang scheduling - Add sshbarrier to ssh plugin #3587 @wangdian
• ❗️❗️ backup plan investigation (K8S scheduler) - [Rest Server] Change to default k8s scheduler #3599 @abuccts @yqwang-ms @fanyangCS
• group in vc management Check groups every time restserver start #3458
• ❗️ log manager needs to be accessed through pylon - [Log manager] Access Log Manager through Pylon #3600 @TongboZhang
• GPU ID is empty, GPU counts in "Task role table" shows as 0. @abuccts [Rest Server] Add pod GPU number for default scheduler #3642
• "Retry Pending" --> "Waiting" @sunqinzheng
• Customization support for disable MPI (Default enable MPI), so that user can run elastic jobs. @wangdian (related PR SSH Generator for webportal #3644)
• Azure blob mount issue @wangdian ; (+ other storage test coverage)
• ❗️ HiveD schedule bug causing GPU not unavailable to use @abuccts @fanyangCS @yqwang-ms
• RESTful API compatibility [Pure K8S] API compatiblity #3206
• InfiniBand support
• cannot access grafana via an application gateway (Dashboard and Grafana cannot be visited behind Azure Gateway #3635)
• permanent token
• hide dashboard in admin mode (hide grafana dashboard in admin mode #3685)
• from homepage, click "view" button but no response (click the "view" but no response #3686)
• job detail page's task role header changes [Web Portal][Rest Server] Map rest server level job status completing, retry pending to running, waiting #3636 [Web Portal] Seperate 'waiting' and 'running' states on task role's statistics #3727 [Web Portal] display stopped task count in task role's header #3840
• fix virtual cluster list's chart [Web Portal] disable utilization charts' animation #3730
• job detail page's clone button should be rendered as a link (could be right clicked and opened in a new tab/window) [Web Portal] Render clone job button as a link if possible #3854
• Task should become stopping status after stop job button is clicked Add stopping status to task #3868
• Fix the bug that the job list page's status doesn't change after user clicks the stop button. [Web Portal] fix job list page's stopping status #3869

Backlogs

• k8s version Jenkins
• choose build images
• drivers installation
• 🕒REST API for VC (HiveD @abuccts and default VC @Binyang2014 )
• 🕒Implement v1 api for jobs

[mzmssg]

Details about AAD in #3378.
Besides, I think we need to add deployment about controller and scheduler

[mzmssg]

Need replace metrics from hadoop, like:

1. dashboard
   
2. hardware
   

[mzmssg]

Currently, default taskRole name contains capital letters and underline, which are illegel for k8s.

[wangdian]

Job keep retry and cannot be stopped.

[fanyangCS]

@yqwang-ms can you take a look at the "job keeps retrying" issue?

[abuccts]

Job keep retry and cannot be stopped.

@yqwang-ms can you take a look at the "job keeps retrying" issue?

This is caused by someone else's test, not related to the code, the whole bed is down.

[yqwang-ms]

All nodes becomes unknown, so controller and scheduler in statefulset is down.
I have manually restarted them, but all worker nodes are still down.
Need to investigate why all worker nodes in 15 and 16 beds are down.

[yqwang-ms]

Seems kubelet are removed in all worker nodes. (probably caused by someone cleaned the k8s cluster)

Even no kubelet history in 10.151.41.21 (15 bed worker)

core@paigcr-a-gpu-1115:~$ sudo docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                      PORTS               NAMES
71a06c62c9d5        hello-world                          "/hello"                 3 months ago        Exited (0) 3 months ago                         quizzical_kare
9882be1aa427        hello-world                          "/hello"                 3 months ago        Exited (0) 3 months ago                         nervous_einstein
b14e8b513c45        openpai.azurecr.io/dls/load_client   "python /load_client…"   6 months ago        Exited (137) 3 months ago                       ecstatic_leavitt
6fd11bd7983e        openpai.azurecr.io/dls/load_client   "python /load_client…"   6 months ago        Exited (137) 3 months ago                       zen_kapitsa
a54f9aa4fa7f        openpai.azurecr.io/dls/load_client   "python /load_client…"   6 months ago        Exited (137) 3 months ago                       agitated_khorana
c2ce59b04a8f        openpai.azurecr.io/dls/load_client   "python /load_client…"   6 months ago        Exited (137) 3 months ago                       jovial_boyd
adfca1e60e7e        openpai.azurecr.io/dls/load_client   "python /load_client…"   6 months ago        Exited (137) 3 months ago                       nostalgic_lovelace
d94900c2a098        openpai.azurecr.io/dls/load_client   "python /load_client…"   11 months ago       Exited (137) 3 months ago                       condescending_swirles
d38a16902aef        openpai.azurecr.io/dls/load_client   "python /load_client…"   11 months ago       Exited (137) 3 months ago                       fervent_dijkstra
0e1991722258        827fd3415f7f                         "/bin/bash"              11 months ago       Exited (0) 11 months ago                        wonderful_heisenberg
0bd2ec5f51dc        827fd3415f7f                         "/bin/bash"              11 months ago       Exited (0) 11 months ago                        dreamy_curran
cc2496d1fdad        827fd3415f7f                         "/bin/bash"              11 months ago       Exited (0) 11 months ago                        optimistic_euler
c5a9ebfc72a5        openpai.azurecr.io/dls/load_client   "python /load_client…"   11 months ago       Exited (137) 3 months ago                       admiring_davinci

[yqwang-ms]

Synced with Hanyu, seems it is caused by his paictl operations.
Let's redeploy 15 and 16.

[yqwang-ms]

15 is recovered.