feat: gdal concurrency TDE-457 #96
Conversation
scripts/standardising.py
Outdated
| from scripts.files.files_helper import get_file_name_from_path, is_tiff | ||
| from scripts.gdal.gdal_helper import run_gdal | ||
| from scripts.logging.time_helper import time_in_ms | ||
|
|
||
|
|
||
| def standardising(files: List[str]) -> List[str]: | ||
| def start_standardising(files: List[str], argo_env: bool) -> List[str]: |
There was a problem hiding this comment.
would be good to do something like cryptocurrency: int and specify the number of workers to use, as argo_env doesn't really make too much sense to the caller.
There was a problem hiding this comment.
Thanks, could you elaborate further on cryptocurrency please?
There was a problem hiding this comment.
LOL darn autocorrect! concurrency
so then this just takes the number of threads rather than a opaque is_in_argo
| @@ -85,7 +85,7 @@ def parse_path(path: str) -> S3Path: | |||
| path (str): A S3 path. | |||
|
|
|||
| Returns: | |||
| S3Path (NamedTupe): s3_path.bucket (str), s3_path.key (str) | |||
There was a problem hiding this comment.
not 100% related to this, but does vscode not detect automatically the types of all of these, we shouldnt be doubling up the typing in both the function definition and the docs as it often leads to them being out of sync.
scripts/standardising.py
Outdated
| output_files.append(tmp_file_path) | ||
|
|
||
| if concurrency: | ||
| with Pool(concurrency) as p: |
There was a problem hiding this comment.
is there any down sides to doing with Pool(1) as p instead of doing two different code paths for running the script?
There was a problem hiding this comment.
I didn't want to add any overhead for local file processing, but I could test to see if it is an issue.
There was a problem hiding this comment.
It does not seem to be an issue.
No description provided.