Decouple hdfs storage from global storage, support config multiple folders for hdfs #1922
Conversation
2 similar comments
|
|
||
| After parsing, object model looks like: | ||
| ```yaml | ||
| storage_path: /path/to/folder1,/path/to/folder2,etc. |
There was a problem hiding this comment.
,etc. could remove which maybe misleading user. if the format is the same with hdfs-site.xml, could refer that config intro format
| #hadoop-data-node: | ||
| # # storage path for hdfs, support comma-delimited list of directories, eg. /path/to/folder1,/path/to/folder2 ... | ||
| # # if left empty, will use cluster.common.data-path/hdfs/data | ||
| # storage_path: |
There was a problem hiding this comment.
what's the default value?
why not make it contains config default value and also give user flexibility to config multi path?
storage_path: /datastorage
There was a problem hiding this comment.
@YanjieGao
Default value is cluster_config[common][data-path]/hdfs/data, which might not be /datastorage.
The logic is: if admin give a specific hdfs storage, then use it, if not, use global storage.
Ideally, if we allow introduce other services' config in yaml, then here we could set $cluser_config.common.data-path/hdfs/data, which would be more clear
There was a problem hiding this comment.
For user only view this config file can't find cluster.common.data-path position. It will be confused for user. Because not all user know to query github code base to find cluster_config[common][storage] cluster object model config.
It is better to tell user clear the default path is where.
We should assume user maybe only get the context of current config file.
There was a problem hiding this comment.
Actually, I don't think user should see here, the default value is for advanced user or dev. In our design, user should overwrite this value in services-configuration.yaml, which contains the necessary context.
If we introduce some hard-code path here(even only in comments), it will couple this file with other service.
There was a problem hiding this comment.
I see, misunderstanding this as end user yaml file & my intent is not hard code (intend is to tell user could find this config default value is this file's data-path config).
In my understand default value is for not advanced user and advanced user will know customized it.
There was a problem hiding this comment.
not big problem. Could continue
mzmssg
changed the title
| @@ -0,0 +1,44 @@ | |||
| ## Hadoop data node section parser | |||
|
|
|||
There was a problem hiding this comment.
Pls link from below doc data path config to this file
There was a problem hiding this comment.
the problem is the same with below, if this doc is just for config. Later could tell user config info at service-config doc or other part
| @@ -0,0 +1,44 @@ | |||
| ## Hadoop data node section parser | |||
|
|
|||
| - [Default Configuration](#D_Config) | |||
There was a problem hiding this comment.
lack
1 first deployed user how to do: step by step. [step's could refer some deployment doc]
2 upgrading user how to do: step by step. [step's could refer some deployment doc]
There was a problem hiding this comment.
@YanjieGao
I think in our design, this doc shoud follow some standard items, only for per service configuration introduction. Not for maintance or other purpose.
i.e.
https://github.com/Microsoft/pai/blob/master/src/rest-server/config/rest-server.md
https://github.com/Microsoft/pai/blob/master/src/grafana/config/grafana.md
Of course, I will update hdfs docs after this PR
No description provided.