From e1b7fc9dc7ab33ade9e981e52c193f258bf0905c Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Wed, 8 Apr 2020 16:20:30 -0700 Subject: [PATCH 1/5] Add admin service entity docs --- rsts/contributor/components/admin.rst | 208 ------------ rsts/contributor/components/admin_service.rst | 315 ++++++++++++++++++ rsts/contributor/components/index.rst | 1 + 3 files changed, 316 insertions(+), 208 deletions(-) create mode 100644 rsts/contributor/components/admin_service.rst diff --git a/rsts/contributor/components/admin.rst b/rsts/contributor/components/admin.rst index 26a1bed016..05721619c2 100644 --- a/rsts/contributor/components/admin.rst +++ b/rsts/contributor/components/admin.rst @@ -132,212 +132,4 @@ Workflowengine This directory contains interfaces to build and execute workflows leveraging flytepropeller compiler and client components. - -Admin Service -============= - -Making Requests to FlyteAdmin ------------------------------ - -Adding request filters -++++++++++++++++++++++ - -We use `gRPC Gateway `_ to reverse proxy http requests into gRPC. -While this allows for a single implementation for both HTTP and gRPC, an important limitation is that fields mapped to the path pattern cannot be -repeated and must have a primitive (non-message) type. Unfortunately this means that repeated string filters cannot use a proper protobuf message. Instead use -the internal syntax shown below:: - - func(field,value) or func(field, value) - -For example, multiple filters would be appended to an http request:: - - ?filters=ne(version, TheWorst)+eq(workflow.name, workflow) - -Timestamp fields use the RFC3339Nano spec (ex: "2006-01-02T15:04:05.999999999Z07:00") - -The fully supported set of filter functions are - -- contains -- gt (greater than) -- gte (greter than or equal to) -- lt (less than) -- lte (less than or equal to) -- eq (equal) -- ne (not equal) -- value_in (for repeated sets of values) - -"value_in" is a special case where multiple values are passed to the filter expression. For example:: - - value_in(phase, 1;2;3) - -Filterable fields vary based on entity types: - -- Task - - - project - - domain - - name - - version - - created_at -- Workflow - - - project - - domain - - name - - version - - created_at -- Launch plans - - - project - - domain - - name - - version - - created_at - - updated_at - - workflows.{any workflow field above} (for example: workflow.domain) - - state (you must use the integer enum e.g. 1) - - States are defined in `launch_plan.proto `_. -- Executions (Workflow executions) - - - project - - domain - - name - - workflow.{any workflow field above} (for example: workflow.domain) - - launch_plan.{any launch plan field above} (for example: launch_plan.name) - - phase (you must use the upper-cased string name e.g. RUNNING) - - Phases are defined in `execution.proto `_. - - execution_created_at - - execution_updated_at - - duration (in seconds) - - mode (you must use the integer enum e.g. 1) - - Modes are defined in `execution.proto `_. - -- Node Executions - - - node_id - - execution.{any execution field above} (for example: execution.domain) - - phase (you must use the upper-cased string name e.g. QUEUED) - - Phases are defined in `execution.proto `_. - - started_at - - node_execution_created_at - - node_execution_updated_at - - duration (in seconds) -- Task Executions - - - retry_attempt - - task.{any task field above} (for example: task.version) - - execution.{any execution field above} (for example: execution.domain) - - node_execution.{any node execution field above} (for example: node_execution.phase) - - phase (you must use the upper-cased string name e.g. SUCCEEDED) - - Phases are defined in `execution.proto `_. - - started_at - - task_execution_created_at - - task_execution_updated_at - - duration (in seconds) - -Putting it all together ------------------------ - -If you wanted to do query on specific executions that were launched with a specific launch plan for a workflow with specific attributes, you could do something like: - -:: - - gte(duration, 100)+value_in(phase,RUNNING;SUCCEEDED;FAILED)+eq(lauch_plan.project, foo) - +eq(launch_plan.domain, bar)+eq(launch_plan.name, baz) - +eq(launch_plan.version, 1234) - +lte(workflow.created_at,2018-11-29T17:34:05.000000000Z07:00) - - - -Adding sorting to requests -++++++++++++++++++++++++++ - -Only a subset of fields are supported for sorting list queries. The explicit list is below: - -- ListTasks - - - project - - domain - - name - - version - - created_at -- ListTaskIds - - - project - - domain -- ListWorkflows - - - project - - domain - - name - - version - - created_at -- ListWorkflowIds - - - project - - domain -- ListLaunchPlans - - - project - - domain - - name - - version - - created_at - - updated_at - - state (you must use the integer enum e.g. 1) - - States are defined in `launch_plan.proto `_. -- ListWorkflowIds - - - project - - domain -- ListExecutions - - - project - - domain - - name - - phase (you must use the upper-cased string name e.g. RUNNING) - - Phases are defined in `execution.proto `_. - - execution_created_at - - execution_updated_at - - duration (in seconds) - - mode (you must use the integer enum e.g. 1) - - Modes are defined `execution.proto `_. -- ListNodeExecutions - - - node_id - - retry_attempt - - phase (you must use the upper-cased string name e.g. QUEUED) - - Phases are defined in `execution.proto `_. - - started_at - - node_execution_created_at - - node_execution_updated_at - - duration (in seconds) -- ListTaskExecutions - - - retry_attempt - - phase (you must use the upper-cased string name e.g. SUCCEEDED) - - Phases are defined in `execution.proto `_. - - started_at - - task_execution_created_at - - task_execution_updated_at - - duration (in seconds) - -Sorting syntax --------------- - -Adding sorting to a request requires specifying the ``key``, e.g. the attribute you wish to sort on. Sorting can also optionally specify the direction (one of ``ASCENDING`` or ``DESCENDING``) where ``DESCENDING`` is the default. - -Example sorting http param: - -:: - - sort_by.key=created_at&sort_by.direction=DESCENDING - -Alternatively, since descending is the default, the above could be rewritten as - -:: - - sort_by.key=created_at - - .. [0] Unfortunately, given unique naming constraints, some models are redefined in `migration_models `__ to guarantee unique index values. diff --git a/rsts/contributor/components/admin_service.rst b/rsts/contributor/components/admin_service.rst new file mode 100644 index 0000000000..78df114409 --- /dev/null +++ b/rsts/contributor/components/admin_service.rst @@ -0,0 +1,315 @@ +.. _components-admin-service: + +############################# +FlyteAdmin Service Background +############################# + +Entities +======== +The `admin service definition `_ defines REST operations for the entities +flyteadmin administers. + +As a refresher, the primary :ref:`entities ` across Flyte map similarly to FlyteAdmin entities. + +Static entities ++++++++++++++++ + +These include: +- Workflows +- Tasks +- Launch Plans + +Permitted operations: +- Create +- Get +- List + +The above are designated by an `identifier `_ +which consists of a project, domain, name and version specification. These entities are for the most part immutable. To update one of these specific entities, the updated +version must be reregistered with a unique and new version identifier attribute. + +One caveat is that launch plan state can toggle between `ACTIVE or INACTIVE `_. +At most one launch plan version across a shared project, domain and name specification can be active at a time. The state affects scheduled launch plans only. +An inactive launch plan can still be used to launch individual executions. However, only an active launch plan runs on a schedule (if it has a schedule defined). + + +Static entities metadata (Named Entities ++++++++++++++++++++++++++++++++++++++++++ +A `named entity `_ includes metadata for one of the above entities +(workflow, task or launch plan) across versions. A named entity includes a resource type (workflow, task or launch plan) and an +`identifier `_ which is composed of project, domain and name. +A named entity also includes metadata, which are mutable attributes about the referenced entity. + +This metadata includes: +- Description: a human readable description for the Named Entity collection +- State (workflows only): this determines whether the workflow is shown on the overview list of workflows scoped by project and domain + +Permitted operations: +- Create +- Update +- Get +- List + + +Execution entities +++++++++++++++++++ + +These include: +- (Workflow) executions +- Node executions +- Task executions + +Permitted operations: +- Create +- Get +- List + +After an execution begins, flyte propeller monitors the execution and sends events which admin uses to update the above executions. + +These `events `_ include +- WorkflowExecutionEvent +- NodeExecutionEvent +- TaskExecutionEvent + +and include information about respective phase transitions, phase transition time and optional output data if the event concerns a terminal phase change. + +These events are the **only** way to update an execution. No raw Update endpoint exists. + +To track the lifecycle of an execution admin stores attributes such as duration, timestamp at which an execution transitioned to running, and end time. + +For debug purposes admin also stores Workflow and Node execution events in its database, but does not currently expose them through an API. Because array tasks can yield very many executions, +admin does **not** store TaskExecutionEvents. + + +Platform entities ++++++++++++++++++ +Projects: like named entities, project have mutable metadata such as human-readable names and descriptions, in addition to their unique string ids. + +Permitted project operations: +- Register +- List + +Matchable resources +A thorough background on :ref: `matchable resources ` explains their purpose and application logic. As a summary, these are used to override system level defaults +for kubernetes cluster resource management, default execution values, and more all across different levels of specificity. + +These entities consist of: +- ProjectDomainAttributes +- WorkflowAttributes + +Where ProjectDomainAttributes configure customizable overrides at the project and domain level and WorkflowAttributes configure customizable overrides at the project, domain and workflow level. + +Permitted attribute operations: +- Update (implicitly creates if there is no existing override) +- Get +- Delete + +Using the Admin Service +======================= + +Making Requests to FlyteAdmin +----------------------------- + +Adding request filters +++++++++++++++++++++++ + +We use `gRPC Gateway `_ to reverse proxy http requests into gRPC. +While this allows for a single implementation for both HTTP and gRPC, an important limitation is that fields mapped to the path pattern cannot be +repeated and must have a primitive (non-message) type. Unfortunately this means that repeated string filters cannot use a proper protobuf message. Instead use +the internal syntax shown below:: + + func(field,value) or func(field, value) + +For example, multiple filters would be appended to an http request:: + + ?filters=ne(version, TheWorst)+eq(workflow.name, workflow) + +Timestamp fields use the RFC3339Nano spec (ex: "2006-01-02T15:04:05.999999999Z07:00") + +The fully supported set of filter functions are + +- contains +- gt (greater than) +- gte (greter than or equal to) +- lt (less than) +- lte (less than or equal to) +- eq (equal) +- ne (not equal) +- value_in (for repeated sets of values) + +"value_in" is a special case where multiple values are passed to the filter expression. For example:: + + value_in(phase, 1;2;3) + +Filterable fields vary based on entity types: + +- Task + + - project + - domain + - name + - version + - created_at +- Workflow + + - project + - domain + - name + - version + - created_at +- Launch plans + + - project + - domain + - name + - version + - created_at + - updated_at + - workflows.{any workflow field above} (for example: workflow.domain) + - state (you must use the integer enum e.g. 1) + - States are defined in `launch_plan.proto `_. +- Named Entity Metadata + - state (you must use the integer enum e.g. 1) + - States are defined in `common.proto `_. +- Executions (Workflow executions) + + - project + - domain + - name + - workflow.{any workflow field above} (for example: workflow.domain) + - launch_plan.{any launch plan field above} (for example: launch_plan.name) + - phase (you must use the upper-cased string name e.g. RUNNING) + - Phases are defined in `execution.proto `_. + - execution_created_at + - execution_updated_at + - duration (in seconds) + - mode (you must use the integer enum e.g. 1) + - Modes are defined in `execution.proto `_. + +- Node Executions + + - node_id + - execution.{any execution field above} (for example: execution.domain) + - phase (you must use the upper-cased string name e.g. QUEUED) + - Phases are defined in `execution.proto `_. + - started_at + - node_execution_created_at + - node_execution_updated_at + - duration (in seconds) +- Task Executions + + - retry_attempt + - task.{any task field above} (for example: task.version) + - execution.{any execution field above} (for example: execution.domain) + - node_execution.{any node execution field above} (for example: node_execution.phase) + - phase (you must use the upper-cased string name e.g. SUCCEEDED) + - Phases are defined in `execution.proto `_. + - started_at + - task_execution_created_at + - task_execution_updated_at + - duration (in seconds) + +Putting it all together +----------------------- + +If you wanted to do query on specific executions that were launched with a specific launch plan for a workflow with specific attributes, you could do something like: + +:: + + gte(duration, 100)+value_in(phase,RUNNING;SUCCEEDED;FAILED)+eq(lauch_plan.project, foo) + +eq(launch_plan.domain, bar)+eq(launch_plan.name, baz) + +eq(launch_plan.version, 1234) + +lte(workflow.created_at,2018-11-29T17:34:05.000000000Z07:00) + + + +Adding sorting to requests +++++++++++++++++++++++++++ + +Only a subset of fields are supported for sorting list queries. The explicit list is below: + +- ListTasks + + - project + - domain + - name + - version + - created_at +- ListTaskIds + + - project + - domain +- ListWorkflows + + - project + - domain + - name + - version + - created_at +- ListWorkflowIds + + - project + - domain +- ListLaunchPlans + + - project + - domain + - name + - version + - created_at + - updated_at + - state (you must use the integer enum e.g. 1) + - States are defined in `launch_plan.proto `_. +- ListWorkflowIds + + - project + - domain +- ListExecutions + + - project + - domain + - name + - phase (you must use the upper-cased string name e.g. RUNNING) + - Phases are defined in `execution.proto `_. + - execution_created_at + - execution_updated_at + - duration (in seconds) + - mode (you must use the integer enum e.g. 1) + - Modes are defined `execution.proto `_. +- ListNodeExecutions + + - node_id + - retry_attempt + - phase (you must use the upper-cased string name e.g. QUEUED) + - Phases are defined in `execution.proto `_. + - started_at + - node_execution_created_at + - node_execution_updated_at + - duration (in seconds) +- ListTaskExecutions + + - retry_attempt + - phase (you must use the upper-cased string name e.g. SUCCEEDED) + - Phases are defined in `execution.proto `_. + - started_at + - task_execution_created_at + - task_execution_updated_at + - duration (in seconds) + +Sorting syntax +-------------- + +Adding sorting to a request requires specifying the ``key``, e.g. the attribute you wish to sort on. Sorting can also optionally specify the direction (one of ``ASCENDING`` or ``DESCENDING``) where ``DESCENDING`` is the default. + +Example sorting http param: + +:: + + sort_by.key=created_at&sort_by.direction=DESCENDING + +Alternatively, since descending is the default, the above could be rewritten as + +:: + + sort_by.key=created_at + diff --git a/rsts/contributor/components/index.rst b/rsts/contributor/components/index.rst index 6b6014e600..5b58f45edf 100644 --- a/rsts/contributor/components/index.rst +++ b/rsts/contributor/components/index.rst @@ -9,5 +9,6 @@ These sections provide implementation details about specific flyte components. T :name: componentstoc admin + admin_service catalog console From b501f1c2e938336d73368d2d0fc2981292cefda4 Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Wed, 8 Apr 2020 16:21:58 -0700 Subject: [PATCH 2/5] More --- rsts/contributor/components/admin.rst | 2 ++ rsts/contributor/components/index.rst | 1 - 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/rsts/contributor/components/admin.rst b/rsts/contributor/components/admin.rst index 05721619c2..9091314555 100644 --- a/rsts/contributor/components/admin.rst +++ b/rsts/contributor/components/admin.rst @@ -22,6 +22,8 @@ service `definition `. methods to process requests. +A detailed explanation of the service can be found in the :ref:`admin service ` page. + .. _components-admin-manager: Managers diff --git a/rsts/contributor/components/index.rst b/rsts/contributor/components/index.rst index 5b58f45edf..6b6014e600 100644 --- a/rsts/contributor/components/index.rst +++ b/rsts/contributor/components/index.rst @@ -9,6 +9,5 @@ These sections provide implementation details about specific flyte components. T :name: componentstoc admin - admin_service catalog console From d8e8e5897d50e8cb395048852d6724f7c6fa1990 Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Wed, 8 Apr 2020 16:30:31 -0700 Subject: [PATCH 3/5] editing --- rsts/contributor/components/admin_service.rst | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/rsts/contributor/components/admin_service.rst b/rsts/contributor/components/admin_service.rst index 78df114409..e3a70897be 100644 --- a/rsts/contributor/components/admin_service.rst +++ b/rsts/contributor/components/admin_service.rst @@ -15,11 +15,13 @@ Static entities +++++++++++++++ These include: + - Workflows - Tasks - Launch Plans Permitted operations: + - Create - Get - List @@ -37,14 +39,16 @@ Static entities metadata (Named Entities +++++++++++++++++++++++++++++++++++++++++ A `named entity `_ includes metadata for one of the above entities (workflow, task or launch plan) across versions. A named entity includes a resource type (workflow, task or launch plan) and an -`identifier `_ which is composed of project, domain and name. +`id `_ which is composed of project, domain and name. A named entity also includes metadata, which are mutable attributes about the referenced entity. This metadata includes: + - Description: a human readable description for the Named Entity collection - State (workflows only): this determines whether the workflow is shown on the overview list of workflows scoped by project and domain Permitted operations: + - Create - Update - Get @@ -55,11 +59,13 @@ Execution entities ++++++++++++++++++ These include: + - (Workflow) executions - Node executions - Task executions Permitted operations: + - Create - Get - List @@ -67,6 +73,7 @@ Permitted operations: After an execution begins, flyte propeller monitors the execution and sends events which admin uses to update the above executions. These `events `_ include + - WorkflowExecutionEvent - NodeExecutionEvent - TaskExecutionEvent @@ -86,20 +93,23 @@ Platform entities Projects: like named entities, project have mutable metadata such as human-readable names and descriptions, in addition to their unique string ids. Permitted project operations: + - Register - List Matchable resources -A thorough background on :ref: `matchable resources ` explains their purpose and application logic. As a summary, these are used to override system level defaults +A thorough background on :ref:`matchable resources ` explains their purpose and application logic. As a summary, these are used to override system level defaults for kubernetes cluster resource management, default execution values, and more all across different levels of specificity. These entities consist of: + - ProjectDomainAttributes - WorkflowAttributes Where ProjectDomainAttributes configure customizable overrides at the project and domain level and WorkflowAttributes configure customizable overrides at the project, domain and workflow level. Permitted attribute operations: + - Update (implicitly creates if there is no existing override) - Get - Delete From e66863f9f29582764fe677d6cd15f7d9403a8e2f Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Wed, 8 Apr 2020 16:35:26 -0700 Subject: [PATCH 4/5] more editing --- rsts/contributor/components/admin_service.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/rsts/contributor/components/admin_service.rst b/rsts/contributor/components/admin_service.rst index e3a70897be..300459e629 100644 --- a/rsts/contributor/components/admin_service.rst +++ b/rsts/contributor/components/admin_service.rst @@ -117,9 +117,6 @@ Permitted attribute operations: Using the Admin Service ======================= -Making Requests to FlyteAdmin ------------------------------ - Adding request filters ++++++++++++++++++++++ From f27afd9338cb93f3420a08247dd728287735003d Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Thu, 9 Apr 2020 10:25:07 -0700 Subject: [PATCH 5/5] index --- rsts/contributor/components/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/rsts/contributor/components/index.rst b/rsts/contributor/components/index.rst index 6b6014e600..5b58f45edf 100644 --- a/rsts/contributor/components/index.rst +++ b/rsts/contributor/components/index.rst @@ -9,5 +9,6 @@ These sections provide implementation details about specific flyte components. T :name: componentstoc admin + admin_service catalog console