Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add admin service and entity docs #252

Merged
merged 5 commits into from
Apr 9, 2020
Merged

Add admin service and entity docs #252

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e1b7fc9
Add admin service entity docs
katrogan Apr 8, 2020
b501f1c
More
katrogan Apr 8, 2020
d8e8e58
editing
katrogan Apr 8, 2020
e66863f
more editing
katrogan Apr 8, 2020
f27afd9
index
katrogan Apr 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 2 additions & 208 deletions rsts/contributor/components/admin.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ service `definition <https://github.com/lyft/flyteidl/blob/master/protos/flyteid
RPC handlers are a thin shim that enforce some request structure validation and call out to appropriate :ref:`manager <components-admin-manager>`.
methods to process requests.

A detailed explanation of the service can be found in the :ref:`admin service <components-admin-service>` page.

.. _components-admin-manager:

Managers
Expand Down Expand Up @@ -132,212 +134,4 @@ Workflowengine

This directory contains interfaces to build and execute workflows leveraging flytepropeller compiler and client components.


Admin Service
=============

Making Requests to FlyteAdmin
-----------------------------

Adding request filters
++++++++++++++++++++++

We use `gRPC Gateway <https://github.com/grpc-ecosystem/grpc-gateway>`_ to reverse proxy http requests into gRPC.
While this allows for a single implementation for both HTTP and gRPC, an important limitation is that fields mapped to the path pattern cannot be
repeated and must have a primitive (non-message) type. Unfortunately this means that repeated string filters cannot use a proper protobuf message. Instead use
the internal syntax shown below::

func(field,value) or func(field, value)

For example, multiple filters would be appended to an http request::

?filters=ne(version, TheWorst)+eq(workflow.name, workflow)

Timestamp fields use the RFC3339Nano spec (ex: "2006-01-02T15:04:05.999999999Z07:00")

The fully supported set of filter functions are

- contains
- gt (greater than)
- gte (greter than or equal to)
- lt (less than)
- lte (less than or equal to)
- eq (equal)
- ne (not equal)
- value_in (for repeated sets of values)

"value_in" is a special case where multiple values are passed to the filter expression. For example::

value_in(phase, 1;2;3)

Filterable fields vary based on entity types:

- Task

- project
- domain
- name
- version
- created_at
- Workflow

- project
- domain
- name
- version
- created_at
- Launch plans

- project
- domain
- name
- version
- created_at
- updated_at
- workflows.{any workflow field above} (for example: workflow.domain)
- state (you must use the integer enum e.g. 1)
- States are defined in `launch_plan.proto <https://github.com/lyft/flyteidl/blob/2c17791170ece1cced3e96daa08ea2692efe3d07/protos/flyteidl/admin/launch_plan.proto#L23>`_.
- Executions (Workflow executions)

- project
- domain
- name
- workflow.{any workflow field above} (for example: workflow.domain)
- launch_plan.{any launch plan field above} (for example: launch_plan.name)
- phase (you must use the upper-cased string name e.g. RUNNING)
- Phases are defined in `execution.proto <https://github.com/lyft/flyteidl/blob/223537e15e05bc6925403a8c11c5a09d91008a80/protos/flyteidl/core/execution.proto#L11,L21>`_.
- execution_created_at
- execution_updated_at
- duration (in seconds)
- mode (you must use the integer enum e.g. 1)
- Modes are defined in `execution.proto <https://github.com/lyft/flyteidl/blob/182eeb9e1d0f2369e479a981f30cb51c2d7a0672/protos/flyteidl/admin/execution.proto#L96>`_.

- Node Executions

- node_id
- execution.{any execution field above} (for example: execution.domain)
- phase (you must use the upper-cased string name e.g. QUEUED)
- Phases are defined in `execution.proto <https://github.com/lyft/flyteidl/blob/223537e15e05bc6925403a8c11c5a09d91008a80/protos/flyteidl/core/execution.proto#L26,L36>`_.
- started_at
- node_execution_created_at
- node_execution_updated_at
- duration (in seconds)
- Task Executions

- retry_attempt
- task.{any task field above} (for example: task.version)
- execution.{any execution field above} (for example: execution.domain)
- node_execution.{any node execution field above} (for example: node_execution.phase)
- phase (you must use the upper-cased string name e.g. SUCCEEDED)
- Phases are defined in `execution.proto <https://github.com/lyft/flyteidl/blob/223537e15e05bc6925403a8c11c5a09d91008a80/protos/flyteidl/core/execution.proto#L42,L49>`_.
- started_at
- task_execution_created_at
- task_execution_updated_at
- duration (in seconds)

Putting it all together
-----------------------

If you wanted to do query on specific executions that were launched with a specific launch plan for a workflow with specific attributes, you could do something like:

::

gte(duration, 100)+value_in(phase,RUNNING;SUCCEEDED;FAILED)+eq(lauch_plan.project, foo)
+eq(launch_plan.domain, bar)+eq(launch_plan.name, baz)
+eq(launch_plan.version, 1234)
+lte(workflow.created_at,2018-11-29T17:34:05.000000000Z07:00)



Adding sorting to requests
++++++++++++++++++++++++++

Only a subset of fields are supported for sorting list queries. The explicit list is below:

- ListTasks

- project
- domain
- name
- version
- created_at
- ListTaskIds

- project
- domain
- ListWorkflows

- project
- domain
- name
- version
- created_at
- ListWorkflowIds

- project
- domain
- ListLaunchPlans

- project
- domain
- name
- version
- created_at
- updated_at
- state (you must use the integer enum e.g. 1)
- States are defined in `launch_plan.proto <https://github.com/lyft/flyteidl/blob/2c17791170ece1cced3e96daa08ea2692efe3d07/protos/flyteidl/admin/launch_plan.proto#L23>`_.
- ListWorkflowIds

- project
- domain
- ListExecutions

- project
- domain
- name
- phase (you must use the upper-cased string name e.g. RUNNING)
- Phases are defined in `execution.proto <https://github.com/lyft/flyteidl/blob/223537e15e05bc6925403a8c11c5a09d91008a80/protos/flyteidl/core/execution.proto#L11,L21>`_.
- execution_created_at
- execution_updated_at
- duration (in seconds)
- mode (you must use the integer enum e.g. 1)
- Modes are defined `execution.proto <https://github.com/lyft/flyteidl/blob/182eeb9e1d0f2369e479a981f30cb51c2d7a0672/protos/flyteidl/admin/execution.proto#L96>`_.
- ListNodeExecutions

- node_id
- retry_attempt
- phase (you must use the upper-cased string name e.g. QUEUED)
- Phases are defined in `execution.proto <https://github.com/lyft/flyteidl/blob/223537e15e05bc6925403a8c11c5a09d91008a80/protos/flyteidl/core/execution.proto#L26,L36>`_.
- started_at
- node_execution_created_at
- node_execution_updated_at
- duration (in seconds)
- ListTaskExecutions

- retry_attempt
- phase (you must use the upper-cased string name e.g. SUCCEEDED)
- Phases are defined in `execution.proto <https://github.com/lyft/flyteidl/blob/223537e15e05bc6925403a8c11c5a09d91008a80/protos/flyteidl/core/execution.proto#L42,L49>`_.
- started_at
- task_execution_created_at
- task_execution_updated_at
- duration (in seconds)

Sorting syntax
--------------

Adding sorting to a request requires specifying the ``key``, e.g. the attribute you wish to sort on. Sorting can also optionally specify the direction (one of ``ASCENDING`` or ``DESCENDING``) where ``DESCENDING`` is the default.

Example sorting http param:

::

sort_by.key=created_at&sort_by.direction=DESCENDING

Alternatively, since descending is the default, the above could be rewritten as

::

sort_by.key=created_at


.. [0] Unfortunately, given unique naming constraints, some models are redefined in `migration_models <https://github.com/lyft/flyteadmin/blob/master/pkg/repositories/config/migration_models.go>`__ to guarantee unique index values.
Loading