Add job status endpoint to server

[val500]

Description

This adds an endpoint to Testflinger server that receives status updates from Testflinger agents. These updates require a webhook that the server will forward the rest of the update message to.

Documentation

Documentation was added to job_schema.rst to reflect new job_status_webhook option in the job yaml. The server ReadMe was updated with details on the new status endpoint.

Web service API changes

• Interface: [POST] /v1/status

• Request Structure:

  {
      "agent_id": "<string>",
      "job_queue": "<string>",
      "job_status_webhook": "<URL as string>",
      "events": [
          {
          "event_name": "<string enum of events>",
          "timestamp": "<string>",
          "detail_msg": "<string>"
          },
          ...
       ]
  }
  

• Response Status Codes:

  • HTTP 200 (OK)
  • HTTP 400 (Bad request) - The arguments could not be processed by the server
  • HTTP 504 (Gateway Timeout) - The webhook URL timed out

• Rationale: This endpoint receives status updates from the agent if there is a webhook specified in the job yaml and posts the update to this webhook.

Tests

For the server, unit tests were added to test_v1.py. For the agents, unit tests were added to both test_client.py and test_agent.py to test both webhook posting behavior and overall status updating.

[mz2]

@val500 @plars please consider documentation in all implementation changes: if it's the case that documentation is missing on the topic affected altogether, consider it a part of the scope to address that too (so we amortise that implementation cost across PRs as opposed to expect some future documentation catch-up).

[omar-selo]

Main concern is reliability, as in I wonder if we should add this to a queue rather than forward immediately

[plars]

Looks like you're missing the enums.py at the very least. Can you add that here or tell me where to find it if it's external?

[omar-selo]

I did have a few comments left, but to avoid blocking anything I'll approve. As I think this is a good first step

[plars]

I have a few more comments below. I think it would be nice to add a bit more on the unit tests, especially on the possible error/failure conditions. And I would definitely recommend and end-to-end test of this now that you have the test-observer side prototyped.

[omar-selo]

It occurred to me that it would be useful for TO if TF informs it about the last event. As in TF can mark an event as last to inform listeners that there will be no more events emitted after it. That way TO can say that this test execution has ended. Otherwise TO will need to know in detail which events would be final events. I prefer if TO doesn't actually know what the events mean for now. wdyt @plars @val500 ?

Update:
Just had a meeting with Andrej and Matias about these events. Here are some notes:

• Would be great to have traceability, such that we can get to a job from TO through the events. This could be in the form of a url.
• TO needs more information about the events, namely it needs to know if an event is the last event, and if an event is a failure/cancellation. We don't want to deduce these on TO by the event name because that couples TO to TF. Instead it would be useful if along with every event perhaps a couple of booleans are set, something like isFinal and isFailure.

wdyt @plars @val500

[val500]

It occurred to me that it would be useful for TO if TF informs it about the last event. As in TF can mark an event as last to inform listeners that there will be no more events emitted after it. That way TO can say that this test execution has ended. Otherwise TO will need to know in detail which events would be final events. I prefer if TO doesn't actually know what the events mean for now. wdyt @plars @val500 ?

Update: Just had a meeting with Andrej and Matias about these events. Here are some notes:

* Would be great to have traceability, such that we can get to a job from TO through the events. This could be in the form of a url.

* TO needs more information about the events, namely it needs to know if an event is the last event, and if an event is a failure/cancellation. We don't want to deduce these on TO by the event name because that couples TO to TF. Instead it would be useful if along with every event perhaps a couple of booleans are set, something like `isFinal` and `isFailure`.

wdyt @plars @val500

@omar-selo I think every job will end with the cleanup phase regardless of whether the job was cancelled or something threw an error, so the last event will always be ended_cleanup - @plars can confirm

[plars]

@omar-selo I think every job will end with the cleanup phase regardless of whether the job was cancelled or something threw an error, so the last event will always be ended_cleanup - @plars can confirm

That's true for testflinger specifically, but the idea is to make it more generic on both sides. We could set something like a boolean "IsFinal" but for us, it would always be on that cleanup event because that's the last thing that would happen, regardless of whether any part of the job failed.

However, that means that something like an event showing that the job timed out, cancelled, etc... would have already happened. If this is what you are really interested in, then perhaps we should mark those with an "IsTerminal" event for event types that will wind up terminating the whole job? would that be enough for you to pick those out as the reason for the overall job termination?

[plars]

Thanks for the updates! we should definitely go ahead and implement the things agreed to on the card unless you have further input on it. A few more comments, but looking better :)

[plars]

Meeting notes: need resource_url added - we discussed adding a JOB_START event at the beginning like the JOB_END event we have at the end. The detail section for that JOB_START event can contain the resource_url which would just be https://testflinger.canonical.com/jobs/<JOB_ID>. That isn't going to change throughout the job, so there's no need to send it with every single event.

[plars]

This looks like it should be good, and thanks for the video showing your testing of it against the TO changes on the other PR.

As discussed yesterday, please wait until the TO changes land so that we can land these PRs at the same time, and test them in staging together.