Diawang/storage plugin

contrib/storage_plugin/README.MD

@@ -0,0 +1,114 @@
+# Team wise storage
+
+A tool to manage external storage in PAI.
+
+## Index
+- [ Storage data structure ](#Data_structure)
+    - [ Server data structure ](#Server_data)
+    - [ User data structure ](#User_data)
+- [ Team wise storage usages ](#Usages)
+    - [ Setup NFS server ](#Usages_nfs)
+    - [ Create server config in PAI storage plugin ](#Usages_server)
+    - [ Create user config in PAI storage plugin ](#Usages_user)
+    - [ Use Storage info in job container ](#Usages_job)
+
+
+## Team wise storage data structures <a name="Data_structure"></a>
+
+### Server data structure <a name="Server_data"></a>
+```json
+{
+	"type": "nfs",
+	"title": "default nfs",
+	"address": "10.0.0.1",
+	"rootPath": "/mnt",
+	"sharedFolders": ["data"],
+	"privateFolders": ["users"]
+}
+```
+
+- type: Remote server type, currently support nfs.
+- title: Shown name on server selection.
+- address: Remote server address.
+- rootPath: The root share folder on remote server.
+- sharedFolders: The shared folders under root path. Remote server will create folders like [rootPath]/[sharedFolder].
+- privateFolders: The private folders for users. The value is the parent folder before user name. Remote server will create folders like [rootPath]/[privateFolder]/[userName]
+
+
+### User data structure <a name="User_data"></a>
+```json
+{
+	"defaultStorage": "storage name",
+	"externalStorages": [
+		"storage name",
+		"storage name2"
+	]
+}
+```
+
+- defaultStorage: The default storage config used for user.
+- externalStorages: User will have permission and his own private folders on all servers in externamStorages according to storage settings.
+
+
+## Team wise storage usages <a name="Usages"></a>
+
+### Setup NFS server <a name="Usages_nfs"></a>
+- Remote server: 10.0.0.1
+- Root folder: /share/teamA
+
+Edit /etc/exports, export /share/teamA 
+```
+/share/teamA (rw, sync, no_root_squash) 
+```
+no_root_squash is needed for storage plugin to creae folders.
+
+
+### Create server config in PAI storage plugin <a name="Usages_server"></a>  
+In PAI dev-box, swith to folder contrib/storage-plugin
+
+Create server config using command:
+```
+python storagectl.py storage set teamA 10.0.0.1 /share/teamA --sharedfolders data --privatefolders users
+```
+
+- This will do the following things:
+	- Update k8s configmap, save server config
+	- Check remote nfs exports
+	- Create temp folder, mount remote folder, 
+		Check and create /share/teamA/data on server as public folder if needed
+		Create private folder /share/teamA/users/[user name] for all users who have access to the server
+	- Unmount and delete temp folder
+
+
+### Create user config in PAI storage plugin <a name="Usages_user"></a>
+In PAI dev-box, swith to folder contrib/storage-plugin
+
+Create user config using command:
+```
+python storagectl.py user setdefault user1 teamA
+```
+
+- This will do the following things:
+	- Set server teamA as user1's default external storage server
+	- Add teamA in to user1's available storage servers
+	- Create user1's private folder on teamA server
+
+
+### Use Storage info in job container <a name="Usages_job"></a>
+On webportal, we can get all server info and user info through URL:
+```TypeScript
+storageExternalUrl = `${api}/api/v1/kubernetes/api/v1/namespaces/default/configmaps/storage-external`;
+storageUserUrl = `${api}/api/v1/kubernetes/api/v1/namespaces/default/configmaps/storage-user`;
+
+fetch(storageExternalUrl).then(responseToData);
+fetch(storageUserUrl).then(responseToData);
+```
+
+Then we can get user's default storage config through user name, and get the server detail.
+```TypeScript
+const content = storageUserData[`${user}.json`];
+const { defaultStorage } = JSON.parse(content);
+storageKey = defaultStorage;
+const storageContent = storageExternalData[storageKey];
+const { type, address, rootPath } = JSON.parse(storageContent);
+```

contrib/storage_plugin/__init__.py

@@ -0,0 +1,16 @@
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

contrib/storage_plugin/examples/team-storage/servers/nfs_example.json

@@ -0,0 +1,8 @@
+{
+    "type": "nfs",
+    "title": "nfs_example",
+    "address": "10.0.0.1",
+    "rootPath": "/share/nfs",
+    "sharedFolders": ["data"],
+    "privateFolders": ["users"]
+    }

contrib/storage_plugin/examples/team-storage/users/user_example.json

@@ -0,0 +1,6 @@
+{
+    "defaultStorage": "nfs_example.json",
+    "externalStorages": [
+        "nfs_example.json"
+    ]
+}

contrib/storage_plugin/schemas/storage_server.schema.json

@@ -0,0 +1,37 @@
+{
+    "type": "object",
+    "properties": {
+        "type": {
+            "type": "string",
+            "description": "The type of external storage"
+        },
+        "title": {
+            "type": "string",
+            "description": "Shown name of external storage"
+        },
+        "address": {
+            "type": "string",
+            "description": "The ip address of external storage"
+        },
+        "rootPath": {
+            "type": "string",
+            "description": "The root path of external storage"
+        },
+        "sharedFolders": {
+            "type": "array",
+            "description": "Shared folder under root path",
+            "items": { "type": "string" }
+        },
+        "privateFolders": {
+            "type": "array",
+            "description": "The base of user private folder under root path, represent rootPath/$base/$username",
+            "items": { "type": "string" }
+        }
+    },
+    "required": [
+        "type",
+        "title",
+        "address",
+        "rootPath"
+    ]
+}

contrib/storage_plugin/schemas/storage_user.schema.json

@@ -0,0 +1,18 @@
+{
+    "type": "object",
+    "properties": {
+        "defaultStorage": {
+            "type": "string",
+            "description": "User default external storage"
+        },
+        "externalStorages": {
+            "type": "array",
+            "description": "All external storages that the user has permission to access",
+            "items": { "type": "string" }
+        }
+    },
+    "required": [
+        "defaultStorage",
+        "externalStorages"
+    ]
+}

contrib/storage_plugin/storagectl.md

@@ -0,0 +1,101 @@
+# storagectl
+
+A tool to manage your storage config.
+
+## Index
+- [ Init storage settings ](#Init)
+    - [Init with no default external storage](#Init_None)
+    - [Init with a nfs server as default external storage](#Init_Nfs)
+
+- [ Manage Server Config ](#Server_config)
+    - [ Set server config for nfs ](#Server_set)
+    - [ List server config ](#Server_list)
+	- [ Create path for server ](#Server_createpath)
+
+- [ Manage user config ](#User_config)
+    - [ Set user default server ](#User_setdefault)
+    - [ List user config ](#User_list) 
+
+- [ Push storage settings ](#Push)
+    - [ Push server settings ](#Push_server)
+    - [ Push user settings ](#Push_user)
+
+
+
+## Init storage settings <a name="Init"></a>
+
+### Init with no default external storage <a name="Init_None"></a>
+
+```
+python storagectl.py init [-f] none  
+```
+
+- Create default.json with no storage server.
+	- If -f was specified, it will override existing default.json on k8s server settings, else it will exit if default.json already exists.
+	- Create default user storage settings for all PAI users as well
+
+### Init with a nfs server as default external storage <a name="Init_Nfs"></a>
+```
+python storagectl.py init [-f] nfs address rootpath  
+```
+- Create default.json with nfs server.
+	- If -f was specified, it will override existing default.json on k8s server settings, else it will exit if default.json already exists.
+	- Create default user storage settings for all PAI users as well
+
+
+## Manage Server Config <a name="Server_config"></a> 
+
+### Set server config for nfs <a name="Server_set"></a> 
+```
+python storagectl.py server set SERVER_NAME nfs ADDRESS ROOT_PATH [--sharedfolders SHARED_FOLDERS [SHARED_FOLDERS ...]] [--privatefolders PRIVATE_FOLDERS [PRIVATE_FOLDERS ...]] 
+```
+- Create or modify server config
+- If '--sharedfolders' is set, create shared folders ROOT_PATH/SHARED_FOLDERS on remote server.
+- If '--privatefolders' is set, create user private folders ROOT_PATH/PRIVATE_FOLDERS/USER_NAME for every user associated with the server on remote server.
+
+### List server config <a name="Server_list"></a> 
+```
+python storagectl.py server list
+```
+- List all servers
+
+### Create path for server <a name="Server_createpath"></a> 
+```
+python storagectl.py server createpath SERVER_NAME
+```
+- Check and create path for server if needed
+
+
+## Manage User Config <a name="User_config"></a> 
+
+### Set user default server <a name="User_setdefault"></a> 
+```
+python storagectl.py user setdefault USER_NAME SERVER_NAME
+```
+- Set default server for user
+	- If privatefolders was defined on server, create privae folders for user on ROOT_PATH/PRIVATE_FOLDERS/USER_NAME
+
+### List user config <a name="User_list"></a> 
+```
+python storagectl.py user list
+```
+- List all users
+
+
+## Push storage settings <a name="Push"></a>
+
+### Push server settings <a name="Push_server"></a>
+
+```
+python storagectl.py push server /path/to/server-config/dir/or/file
+```
+- Create or update server config
+
+
+### Push user settings <a name="Push_user"></a>
+
+```
+python storagectl.py push user /path/to/user-config/dir/or/file
+```
+- Create or update user storage config
+