Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Add exists check for workflow & node executions #172

Merged
merged 2 commits into from
Mar 31, 2021
Merged

Conversation

katrogan
Copy link
Contributor

Signed-off-by: Katrina Rogan [email protected]

TL;DR

Add exists check for workflow & node executions to avoid fetching and passing around fully fledged executions in memory.

Additionally, only add the exists check for these parent executions when node and task executions, respectively, are not found and we create an entry in the db for them for the first time.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

Tracking Issue

flyteorg/flyte#880

Follow-up issue

NA

Signed-off-by: Katrina Rogan <[email protected]>
@codecov
Copy link

codecov bot commented Mar 30, 2021

Codecov Report

Merging #172 (158a063) into master (2db25e0) will increase coverage by 0.22%.
The diff coverage is 92.53%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #172      +/-   ##
==========================================
+ Coverage   62.82%   63.04%   +0.22%     
==========================================
  Files         100      100              
  Lines        7257     7295      +38     
==========================================
+ Hits         4559     4599      +40     
+ Misses       2116     2114       -2     
  Partials      582      582              
Flag Coverage Δ
unittests 63.04% <92.53%> (+0.22%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/manager/impl/launch_plan_manager.go 61.79% <ø> (ø)
pkg/repositories/gormimpl/common.go 57.14% <ø> (ø)
pkg/repositories/gormimpl/launch_plan_repo.go 63.63% <ø> (ø)
pkg/repositories/gormimpl/task_repo.go 68.11% <ø> (ø)
pkg/repositories/gormimpl/workflow_repo.go 69.11% <ø> (ø)
pkg/manager/impl/util/shared.go 62.99% <75.00%> (ø)
pkg/repositories/gormimpl/execution_repo.go 67.41% <84.61%> (+2.94%) ⬆️
pkg/repositories/gormimpl/node_execution_repo.go 64.60% <87.50%> (+3.77%) ⬆️
pkg/manager/impl/execution_manager.go 68.87% <100.00%> (ø)
pkg/manager/impl/node_execution_manager.go 68.57% <100.00%> (+1.66%) ⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2db25e0...158a063. Read the comment docs.

Copy link
Contributor

@kumare3 kumare3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!

@kumare3
Copy link
Contributor

kumare3 commented Mar 31, 2021

I tested this change, seems good to go, performance improvement is not great, but this is the right change nevertheless

@katrogan katrogan merged commit 55c3291 into master Mar 31, 2021
EngHabu pushed a commit that referenced this pull request Apr 26, 2021
EngHabu pushed a commit that referenced this pull request Apr 26, 2021
EngHabu added a commit that referenced this pull request Apr 30, 2021
…D and OAuth2 configs, OAuth2 Metadata over gRPC #minor (#168)

* wip: OAuth2 Support

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* wip

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* wip

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* tighten security of generated tokens

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Support storing form post values in auth code JWT

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* save secrets to k8s secrets

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Expose metadata endpoints over gRPC

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* trim OpenID Connect config further

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Selectively authenticate gRPC endpoints

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Support external oauth2 server and Okta Config

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* update config

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fix nil secrets data map

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fixed the pointer overwrite issue in oauthServer metadata (#183)

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Co-authored-by: Prafulla Mahindrakar <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Simplify config further and move auth package up

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fix clusterresource Project and domain(#167)

* Fix clusterresource Project

Signed-off-by: Anand Swaminathan <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Bump flyteidl version to pick up auth role field number fix (#169)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add option to use project name as namespace for the task pods (#166)

* Add option to use project name as namespace for the task pods

Signed-off-by: Jeev B <[email protected]>

* rename

Signed-off-by: Jeev B <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* GetExecution performance improvements (#171)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add exists check for workflow & node executions (#172)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Remove legacy fetch for workflow execution inputs (#173)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Added release workflow (#170)

Signed-off-by: yuvraj <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update Flyteidl version (#175)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Added version in flyteadmin (#154)

* wip: added version pkg

Signed-off-by: yuvraj <[email protected]>

* wip: resolve conflict

Signed-off-by: yuvraj <[email protected]>

* wip: added version in rpc

Signed-off-by: yuvraj <[email protected]>

* wip: small fixes

Signed-off-by: yuvraj <[email protected]>

* wip: Added panic cache in get version service

Signed-off-by: yuvraj <[email protected]>

* Added flytestdlib for version package

Signed-off-by: yuvraj <[email protected]>

* Added version service test

Signed-off-by: yuvraj <[email protected]>

* wip: added ldflags in goreleaser

Signed-off-by: yuvraj <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Propagate nesting and principal for child executions (#177)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Write workflow and node execution events asynchronously (#174)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add sensible flyteadmin config defaults (#179)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Lint

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* further cleanup

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Only register authserver when auth is enabled

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update to latest flyteidl and separate auth interfaces

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* dead code

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* merge master

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Move to authorizedUris

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update to released flyteidl

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fix response expiry and add unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update go mod

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* fix unit tests that broke because of identity changes

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Co-authored-by: pmahindrakar-oss <[email protected]>
Co-authored-by: Prafulla Mahindrakar <[email protected]>
Co-authored-by: Anand Swaminathan <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
Co-authored-by: Jeev B <[email protected]>
Co-authored-by: Yuvraj <[email protected]>
Co-authored-by: Flyte Bot <[email protected]>
eapolinario pushed a commit that referenced this pull request Sep 6, 2023
…D and OAuth2 configs, OAuth2 Metadata over gRPC #minor (#168)

* wip: OAuth2 Support

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* wip

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* wip

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* tighten security of generated tokens

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Support storing form post values in auth code JWT

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* save secrets to k8s secrets

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Expose metadata endpoints over gRPC

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* trim OpenID Connect config further

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Selectively authenticate gRPC endpoints

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Support external oauth2 server and Okta Config

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* update config

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fix nil secrets data map

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fixed the pointer overwrite issue in oauthServer metadata (#183)

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Co-authored-by: Prafulla Mahindrakar <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Simplify config further and move auth package up

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fix clusterresource Project and domain(#167)

* Fix clusterresource Project

Signed-off-by: Anand Swaminathan <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Bump flyteidl version to pick up auth role field number fix (#169)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add option to use project name as namespace for the task pods (#166)

* Add option to use project name as namespace for the task pods

Signed-off-by: Jeev B <[email protected]>

* rename

Signed-off-by: Jeev B <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* GetExecution performance improvements (#171)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add exists check for workflow & node executions (#172)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Remove legacy fetch for workflow execution inputs (#173)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Added release workflow (#170)

Signed-off-by: yuvraj <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update Flyteidl version (#175)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Added version in flyteadmin (#154)

* wip: added version pkg

Signed-off-by: yuvraj <[email protected]>

* wip: resolve conflict

Signed-off-by: yuvraj <[email protected]>

* wip: added version in rpc

Signed-off-by: yuvraj <[email protected]>

* wip: small fixes

Signed-off-by: yuvraj <[email protected]>

* wip: Added panic cache in get version service

Signed-off-by: yuvraj <[email protected]>

* Added flytestdlib for version package

Signed-off-by: yuvraj <[email protected]>

* Added version service test

Signed-off-by: yuvraj <[email protected]>

* wip: added ldflags in goreleaser

Signed-off-by: yuvraj <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Propagate nesting and principal for child executions (#177)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Write workflow and node execution events asynchronously (#174)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add sensible flyteadmin config defaults (#179)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Lint

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* further cleanup

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Only register authserver when auth is enabled

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update to latest flyteidl and separate auth interfaces

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* dead code

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* merge master

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Move to authorizedUris

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update to released flyteidl

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Fix response expiry and add unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update go mod

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* fix unit tests that broke because of identity changes

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Co-authored-by: pmahindrakar-oss <[email protected]>
Co-authored-by: Prafulla Mahindrakar <[email protected]>
Co-authored-by: Anand Swaminathan <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
Co-authored-by: Jeev B <[email protected]>
Co-authored-by: Yuvraj <[email protected]>
Co-authored-by: Flyte Bot <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants