Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vdk-structlog: put vdk init logs config behind flag #3107

Merged
merged 1 commit into from
Feb 16, 2024

Conversation

DeltaMichael
Copy link
Contributor

@DeltaMichael DeltaMichael commented Feb 13, 2024

Why?

Configuring the logs in the vdk_initialize hook causes commands like vdk server and vdk info to have the same verbose logging format as data job logs.

What?

  • Put the log configuration for vdk_initialize behind a config option and disable it by default for local runs and enable it by default for cloud runs. Configuring the logging format in vdk_initialize might still be useful for cloud runs and we don't care about other vdk commands output in cloud environments

  • Move config classes and functions into separate file

How was this tested

(venv) 11:01:14 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=True 
(venv) 11:01:19 ~ $ vdk create
2024-02-15 11:01:21,922 [VDK] [INFO ] vdk.internal.builtin_plugins.b builtin_hook_impl.py :98   vdk_initialize   - Setting: OP_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681, ATTEMPT_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681-aa061, EXECUTION_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681
2024-02-15 11:01:21,923 [VDK] [INFO ] vdk.internal.cli_entry                 cli_entry.py :135  vdk_main         - Start CLI vdk with args ['create']
Job Name: 

(venv) 11:01:52 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=False
(venv) 11:01:58 ~ $ vdk create                                 
Job Name: 
(venv) 11:03:25 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=True
(venv) 11:03:27 ~ $ vdk version
2024-02-15 11:03:30,267 [VDK] [INFO ] vdk.internal.builtin_plugins.b builtin_hook_impl.py :98   vdk_initialize   - Setting: OP_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810, ATTEMPT_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810-3568d, EXECUTION_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810
2024-02-15 11:03:30,267 [VDK] [INFO ] vdk.internal.cli_entry                 cli_entry.py :135  vdk_main         - Start CLI vdk with args ['version']
2024-02-15 11:03:30,292 [VDK] [INFO ] vdk.internal.builtin_plugins.v           version.py :85   version          - Versatile Data Kit (VDK)
Version: 0.3.1177416838
Build details: RELEASE_VERSION=0.3.1177416838, BUILD_DATE=Thu Feb 15 07:57:26 UTC 2024, BUILD_MACHINE_INFO=Linux runner-trzkizxz-project-28359933-concurrent-0rvcnn 5.4.235-144.344.amzn2.x86_64 #1 SMP Sun Mar 12 12:50:22 UTC 2023 x86_64 GNU/Linux, GITLAB_CI_JOB_ID=6177494231, GIT_COMMIT_SHA=a4c5a23b69ed2bd135b7f14658abd18c143e8759, GIT_BRANCH=main
Python version: 3.11.7 64bit (/Users/mdilyan/Projects/versatile-data-kit/projects/vdk-plugins/vdk-structlog/venv/bin/python)

Installed plugins:
vdk-ingest-http (from package vdk-ingest-http, version 0.2.1156222304)
vdk-sqlite (from package vdk-sqlite, version 0.1.1156222304)
vdk-server (from package vdk-server, version 0.1.1174992336)
vdk-structlog (from package vdk-structlog, version 0.1.0)
vdk-control-service-properties (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-execution-skip (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-plugin-control-cli (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-ingest-file (from package vdk-ingest-file, version 0.1.948436673)
--------------------------------------------------------------------------------

(venv) 11:04:04 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=False
(venv) 11:04:09 ~ $ vdk version                                
Versatile Data Kit (VDK)
Version: 0.3.1177416838
Build details: RELEASE_VERSION=0.3.1177416838, BUILD_DATE=Thu Feb 15 07:57:26 UTC 2024, BUILD_MACHINE_INFO=Linux runner-trzkizxz-project-28359933-concurrent-0rvcnn 5.4.235-144.344.amzn2.x86_64 #1 SMP Sun Mar 12 12:50:22 UTC 2023 x86_64 GNU/Linux, GITLAB_CI_JOB_ID=6177494231, GIT_COMMIT_SHA=a4c5a23b69ed2bd135b7f14658abd18c143e8759, GIT_BRANCH=main
Python version: 3.11.7 64bit (/Users/mdilyan/Projects/versatile-data-kit/projects/vdk-plugins/vdk-structlog/venv/bin/python)

Installed plugins:
vdk-ingest-http (from package vdk-ingest-http, version 0.2.1156222304)
vdk-sqlite (from package vdk-sqlite, version 0.1.1156222304)
vdk-server (from package vdk-server, version 0.1.1174992336)
vdk-structlog (from package vdk-structlog, version 0.1.0)
vdk-control-service-properties (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-execution-skip (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-plugin-control-cli (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-ingest-file (from package vdk-ingest-file, version 0.1.948436673)
--------------------------------------------------------------------------------

CI/CD

What kind of change is this?

Bugfix

@DeltaMichael DeltaMichael linked an issue Feb 13, 2024 that may be closed by this pull request
@DeltaMichael DeltaMichael force-pushed the person/mdilyan/structlog-vdk-init branch from f7dcec6 to 3940f40 Compare February 13, 2024 12:28
@DeltaMichael DeltaMichael force-pushed the person/mdilyan/structlog-vdk-init branch 5 times, most recently from aaf50db to 11063df Compare February 14, 2024 14:34
@antoniivanov
Copy link
Collaborator

Please expand on the

How was this tested
Manually

What were the commands and their output and how it was verified. I always discourage saying "manually" without explanation in testing done.

It's good idea to verify it works well with a few commands . E.g vdk list, vdk create, vdk depoy, vdk version

@DeltaMichael DeltaMichael force-pushed the person/mdilyan/structlog-vdk-init branch 4 times, most recently from a614c44 to 7f18a7b Compare February 15, 2024 13:12
Why?

Configuring the logs in the vdk_initialize hook causes commands like
vdk server and vdk info to have the same verbose logging format as
data job logs.

What?

- Put the log configuration for vdk_initialize behind a config option
and disable it by default for local runs and enable it by default
for cloud runs. Configuring the logging format in vdk_initialize might
still be useful for cloud runs and we don't care about other vdk commands
output in cloud environments

- Move config classes and functions into separate file

How was this tested

Manually
CI/CD

What kind of change is this?
Feature/non-breaking

Signed-off-by: Dilyan Marinov <[email protected]>
@DeltaMichael DeltaMichael force-pushed the person/mdilyan/structlog-vdk-init branch from 7f18a7b to 0b33c47 Compare February 15, 2024 15:10
@DeltaMichael DeltaMichael merged commit 5feed5b into main Feb 16, 2024
9 checks passed
@DeltaMichael DeltaMichael deleted the person/mdilyan/structlog-vdk-init branch February 16, 2024 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

non-run CLI commands log output without context
4 participants