Master issue for new 'pacts for verification' endpoint

[bethesque]

Audience

This issue is for Pact library maintainers, to document and discuss the adoption of the new 'pacts for verification' endpoint.

Summary

The new 'pacts for verification' endpoint allows you to specify the provider name, the consumer version 'selectors', (and later on, the provider version tags that will be published with the verification results), and the Pact Broker will return a list of pacts to verify. This will replace the current approach of making multiple individual calls to the latest provider pacts and latest provider pacts with tag endpoints.

Why make this change?

• To support pending pacts. One of the reasons that provider teams do not enjoy using Pact (or refuse to use Pact at all), is that the current workflow causes the provider build to fail when the pact verification fails, and pact verification may fail for many reasons other than an actual integration bug (for example, when adding a new interaction to the pact before implementing it in the provider because you are following a "consumer driven" workflow). This new endpoint allows allows the Pact Broker to indicate whether or not a particular pact is in "pending" state - that is, whether it should be run without failing the build. This will reduce the number of false negative build failures experienced by the provider team, and make them happier to use Pact.
• The new endpoint will allow support for the upcoming "work in progress" workflow (allowing changed pacts to be automatically picked up by the provider verification build without having to manually set the tags in the provider configuration)
• The new endpoint de-duplicates pacts to allow verifications to run faster (eg. if 'master' and 'prod' pacts have the same content, there is no need to verify the same pact twice).
• It will allow support for future features such as specifying different pact types (http or message).

What changes need to be made to the code that uses this new API?

For implementations that wrap the ruby standalone

• The first thing is to upgrade to the latest ruby standalone. That will get you 90% of the way there. The last bit is to update the interface that is displayed to the user to support the consumer version selectors feature described in the first dot point of "For native implementations" below.

Note that the --consumer-version-selector argument is a json string, due to the complexity of the field. If your language allows it, please do not validate this hash - just pass it through. This will allow new features to be added to the selectors without having to make code updates to the wrapper implementations.

You can have a play around with an example CLI here https://github.com/pact-foundation/pact-provider-verifier/blob/master/script/dev/broker.sh

For native implementations

• The interface shown to the users will remain mostly the same, however, there is one new concept that should be supported. The 'pacts for verification' endpoint supports list of "consumer version selectors" rather than "consumer version tags". A selector consists of a tag name, and a flag to indicate a scope (actually, it could also consist of a hard coded version as well, but this would be an unlikely use case in this scenario). This allows the user to say "I want to verify all production pacts", rather than just "the latest production pact", which is useful when there are multiple versions of a consumer in production (eg. mobile consumers). This has been a feature requested by multiple people over the last few months.

  Please read this page to understand how selectors work before continuing https://docs.pact.io/pact_broker/advanced_topics/selectors

  At the moment, if your consumer library supports specifying tags, it probably allows a list of string consumer version tag names. eg consumerVersionTags: ["master", "prod"]. These indicate that the latest pact for each of these tag names should be verified.

  To support the "all pacts for a tag" feature (and future features that will be outlined below) I would suggest adding a consumerVersionSelectors configuration object that accepts a list of hashes (while also allowing people to continue using the simple tag list if they want). Please use the term consumerVersionSelectors and don't make up a custom one so that the documentation can be consistent across all the implementations.

  eg.

  configuration.consumerVersionTags = ["master"] //existing functionality
  configuration.consumerVersionSelectors = [{"tag": "master", "latest": true}, {"tag": "prod"}]
  

  As you will have noticed, in the API, a selector with a tag name, but without a "latest": true indicates that ALL of the versions for that tag should be selected. This may be unintuitive to users who are familiar with the existing consumer version tag configuration for Pact, who would expect that a selector without any scope qualifier would return the latest by default. This will need to be documented clearly.

  If your language allows it, please pass through any other keys that the user specifies in the selector though to the API directly, as this will allow new features to be added without having to change the code. Please don't get overly exacting about client side validation - just make sure that any errors in the response get shown to the user in full, so they can get the error messages from the server themselves.

• The calling code should make use of the "pb:provider-pacts-for-verification" relation that will be available in the HAL index to retrieve the list of pacts. If it does not exist, please fall back to using the existing methods of retrieving pacts for backwards compatibility.

• The returned list of pacts will include metadata about the pact, including a list of "notices" that will include the reason that this particular pact was returned (eg. is it the latest 'prod' pact, is it a 'work in progress' pact). If possible, please print out, or somehow make available, these messages. For this feature, just print out the notices where "when": "before_verification". The "when": "after_verification" notices will become relevant in the "pending pacts" feature.
  It is an array so that we can change the notices without having to make any changes to the calling code. Being as descriptive as possible about what's going on should help people understand and transition to the new workflow.

API documentation

URL

POST pacts/provider/{provider}/for-verification (but please, use the pb:provider-pacts-for-verification relation in the HAL index rather than hardcoding this, so that you can detect when this feature is available in the version of the Pact Broker that is being used. The request uses POST because the input parameters need to be structured data, and that just gets messy in GET query params.

Body

Example: This data structure represents the way a user might specify "I want to verify the latest 'master' pact, all 'prod' pacts, and when I publish the verification results, I'm going to tag the provider version with 'master'"

{
  "consumerVersionSelectors": [
    {
      "tag": "master",
      "latest": true
    },{
      "tag": "prod"
    }
  ],
  "providerVersionTags": ["master"]
}

consumerVersionSelectors.tag: the tag name(s) of the consumer versions to get the pacts for.

consumerVersionSelectors.fallbackTag: the name of the tag to fallback to if the specified tag does not exist. This is useful when the consumer and provider use matching branch names to coordinate the development of new features.

consumerVersionSelectors.latest: true. If the latest flag is omitted, all the pacts with the specified tag will be returned. (This might seem a bit weird, but it's done this way to match the syntax used for the matrix query params. See https://docs.pact.io/selectors)

consumerVersionSelectors.consumer: allows a selector to only be applied to a certain consumer. This is used when there is an API that has multiple consumers, one of which is a deployed service, and one of which is a mobile consumer. The deployed service only needs the latest production pact verified, where as the mobile consumer may want all the production pacts verified.

providerVersionTags: the tag name(s) for the provider application version that will be published with the verification results. This is used by the Broker to determine whether or not a particular pact is in pending state or not. This parameter can be specified multiple times.

Response body

pending flag and the "pending reason" notice will only be included if includePendingStatus is set to true.

{
  "_embedded": {
    "pacts": [
      {
        "verificationProperties": {
          "notices": [
            {
              "text": "This pact is being verified because it is the pact for the latest version of Foo tagged with 'dev'"
            }
          ],
        },
        "_links": {
          "self": {
            "href": "http://localhost:9292/pacts/provider/Bar/consumer/Foo/pact-version/0e3369199f4008231946e0245474537443ccda2a",
            "name": "Pact between Foo (v1.0.0) and Bar"
          }
        }
      }
    ]
  },
  "_links": {
    "self": {
      "href": "http://localhost:9292/pacts/provider/Bar/for-verification",
      "title": "Pacts to be verified"
    }
  }
}

Testing

This feature is only released in the Pactflow code, until we have tested and finalised the API. You can use this instance to test with.

Broker base URL: https://test.pact.dius.com.au
username: dXfltyFMgNOFZAxr8io9wJ37iUpY42M
password: O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1

If you're writing a native implementation, please write me a pact and I'll add it to the verifications! Here's an example: https://github.com/pact-foundation/pact-ruby/blob/master/spec/service_providers/pact_ruby_fetch_pacts_for_verification_test.rb

Upcoming features

Just a heads up so that you can make informed decisions now about how to design the interface that your users see.

fallback: in the consumerVersionSelector, the user will be able to specify a "fallback" tag to use if the tag that they have specified does not exist. This will support "feature branch" development across projects where the branch names for a particular feature match for consumers and providers, to help them coordinate development.

includePendingStatus: true|false. This is to support the "pending pacts" feature. When true, a pending boolean will be added to the verificationProperties in the response, and an extra message will appear in the notices array to indicate why this pact is/is not in pending state.

includeWipPactsSince: Date. The "work in progress" pacts feature will allow newly changed pacts to be automatically included in the verification task without having to modify the list of consumer tags. More on this later.

excludeWipConsumerVersionTags: When the WIP feature is implemented, and the newly changed pacts are automatically included in this response, this will allow providers to skip specific verification for specific tags if they have decided that that pact will never be verified, and the consumer team has not removed that pact from the broker yet.

[mefellows]

@bethesque is this statement still true? My understanding is that by omitting latest now, it will indeed fetch all pacts with that tag, and eventually, scoped to a specific user if specified.

consumerVersionSelectors.latest: true. For now, this is mandatory, and must be set to true, as the other variations have not been implemented yet. When implemented, if the latest flag is omitted, all the pacts with the specified tag will be returned. (This might seem a bit weird, but it's done this way to match the syntax used for the matrix query params.) I suggest exposing a more intuitive interface to the users however.

[bethesque]

It's implemented now. I'll update that section. Thanks.