[Feature] Support Styx style schedule in flyteidl

[honnix]

Motivation: Why do you think this is important?
When integrating with our scheduler Styx, we will need to extend the semantics of schedule to support Styx cron syntax, cron aliases, as well as offset.

Goal: What should the final outcome look like, ideally?
Scheduleprotobuf definition is extended to support Styx style schedule.

Describe alternatives you've considered
We tried reusing existing cron but the syntax is a bit different and flytekit doesn't support Styx style cron aliases; and more importantly, we need to support offset which is one the most important concepts in Styx.

Flyte component

• Overall
• Flyte Setup and Installation scripts
• Flyte Documentation
• Flyte communication (slack/email etc)
• FlytePropeller
• FlyteIDL (Flyte specification language)
• Flytekit (Python SDK)
• FlyteAdmin (Control Plane service)
• FlytePlugins
• DataCatalog
• FlyteStdlib (common libraries)
• FlyteConsole (UI)
• Other

[Optional] Propose: Link/Inline
We could extend Schedule protobuf in the following way:

// Styx style schedule
// More details can be found at https://github.com/spotify/styx#schedule-string
message StyxSchedule {
    string schedule = 1;
    string offset = 2;
}

// Defines complete set of information required to trigger an execution on a schedule.
message Schedule {

    oneof ScheduleExpression {
        // Uses AWS syntax: "Minutes Hours Day-of-month Month Day-of-week Year"
        // e.g. for a schedule that runs every 15 minutes: "0/15 * * * ? *"
        string cron_expression = 1;
        FixedRate rate = 2;
        StyxSchedule styx_schedule = 4;       // <-----------------
    }

    // Name of the input variable that the kickoff time will be supplied to when the workflow is kicked off.
    string kickoff_time_input_arg = 3;
}

Additional context
The way we integrate Styx with Flyte is having a deployment service talking flyteadmin interface in the northbound and deploying entities to both Flyte and Styx.

flytekit -> deployment service -> Flyteadmin
                              \
                                -> Styx                     

The deployment service does not forward schedule to flyteadmin, but only to Styx.

Is this a blocker for you to adopt Flyte
Yes unfortunately.

[schottra]

This will require some additional work in flyteconsole in order to correctly display the schedule.
As things currently are, I think the schedule string will not render at all because it will exist in a field that Console is not aware of.

[honnix]

@schottra Thanks for pointing out! We will work on a PR to fix this if the proposed IDL change makes sense and gets merged.

[honnix]

We are keeping the existing cron_expression supported and that should not fail anything on flyteconsole. To support the newly added proto message, we will need to make some changes that will come later.

[kumare3]

@honnix I think this can be closed right?

[honnix]

Sort of I think. Do we want to fix console to be able to render the new message?

[honnix]

@kumare3 @schottra I proposed a small change in flyteorg/flyteconsole#107. PTAL, thanks.

[kumare3]

@honnix, @schottra and I had a discussion today about the use of offsets.

Firstly, let us define offsets
Offsets are used to shift the passed in time parameter for a schedule using a duration field. The shift is done by actually changing the time of execution. Thus an hourly schedule offset by -P1H implies that the schedule is shifted by 1 hour in the past. This is useful to indicate that the execution should work on a partition or a dataset that was generated 1H in the past. It is independent of the window itself. For example, in the case where data lands after 3H's, but we run the schedule every hour and consider the data for 1H only, the window is 1H and the offset is 3H.

NOTE
Offsets can be completely implemented using just a cron schedule and default input parameters to a launch-plan. But, to achieve this the code would need to change. So to achieve the delayed window computation like the second case defined above, we could create a launch plan like this

  lp = wf.create_launch_plan(
         cron="@hourly",
         inputs={
                       "window": "P1H",
                       "delay": "P3H",
        },
)

Here window refers to the window of computation for the data. delay refers to the actual delay in computation of the data. Thus every hour, the computation may look back 3H and use data in the (T-3H, T-2H) period.

The same can be achieved using an offset

  lp = wf.create_launch_plan(
         cron=("@hourly", offset="P3H"),
         inputs={
                       "window": "P1H",
        },
)

Thus the schedule could be varied based on the data.

User interface consideration
In the UI we will show this as

*schedules*
every hour (offset (-) 3h)

Conclusion
We have decided to implement an offset in the IDL, but, use will be discouraged except for when using Styx, till we figure out the right set of usecases

[honnix]

I think we can close this now. Thanks for everyone's help.