[EngSys] Allow Project Exclusions

[jsquire]

Summary

Extending the build pipeline to allow project exclusions to be specified within the service directory. This is intended to enable splitting the pipeline into multiple definitions for service directories that contain both T1 and T2 packages, particularly in the case where those are maintained by different teams.

Also included in this set of changes is a configuration set for the legacy Event Hubs packages.

Last Upstream Rebase

Monday, December 21, 11:55am (EST)

References

• [Event Hubs] Exclude Legacy Packages from Pipelines (#17659)
• Event Hubs Test Pipelines: Split out Management, Legacy Client, and Client Tests (#15101)

[jsquire]

The Manually filter existed in the tests archtype; it seemed like it should be applied here as well.

[jsquire]

Though the expression is used in a couple of call sites, I chose not to move it into a variable; my goal was to be explicit and clear on how the value is formed, since there is string parsing taking place in the project file. I feel that having the join inline improves locality and prevents having to scan the file to understand what is happening.

[jsquire]

The wildcard should be sufficient, but there's one outstanding observation from the initial round of testing where it's not 100% confirmed for one of our script calls.

My approach here was to explicitly define everything that exists today and leave the wild card to provide some level of filtering for anything new. I'm still working with members of the engineering team to investigate and will open a new PR to trim to just the wild card, if possible.

[jsquire]

Once the Event Hubs legacy configuration is merged, I'll create the corresponding pipelines for it and then exclude the T1 assets from the main ci.yml and tests.yml files.

[heaths]

Overall good, but with other T1s deprecated (e.g. Key Vault), should we maybe consider a simpler approach (easier to maintain) of moving them out to another directory or something?

[jsquire]

Overall good, but with other T1s deprecated (e.g. Key Vault), should we maybe consider a simpler approach (easier to maintain) of moving them out to another directory or something?

That's a wider conversation and beyond the scope of my mandate here. I would absolutely love to see us split up the service directories, reduce the clutter, and only group together assets that are truly related. Unfortunately, I've been making that suggestion for quite a while and it hasn't been well-received by the folks that own the guidelines for repository structure. Please feel free to add your voice to that plea.

[pakrym]

I'm a bit concerned with having to keep track of the projects/paths in multiple places and keep it all in sync without making mistakes. Is there maybe some import voodoo we can use to centralize this information or write a script that would make sure things are up-to-date and correct?

[jsquire]

These particular triggers are owned by DevOps, unfortunately, and aren't something that we have much influence over in terms of placement and syntax. We could use a sub template to import them into a variable, but this set is used in a single place, so there's little benefit.

The ExcludeProject parameter that we added has a different format with the goal of less ceremony and more flexibility. It's meant to be more of a "tell me the pattern of project names that you don't care about and we'll ignore anything under that." With the wildcard support, it shouldn't be necessary to change that often nor adapt to most changes in the repo paths.

To your point, the best way to centralize would likely be a script that could accept the wildcard patterns and generate the corresponding exclude paths. I think that's probably a bit of a premature optimization at this point, as there aren't many places where the T1 and T2 code is bound together under different ownership. For those that are, the T1 assets are in maintenance mode and it isn't likely that we'll see path changes. If we start to see people splitting pipelines in areas that are more volatile, I think that is a great time to revisit.

[jsquire]

I'm going to commit this in the current form for now, to unblock me from splitting the Messaging libraries out. I'll file an issue to track considering a script to automate at a later time.

[weshaggard]

Yes explicitly listing and maintaining these include/exclude paths has annoyed me as well. Unfortunately it is a devops limitation currently and so we just have to pay the cost until we can find a better solution.

I will say I do generally try and avoid mixing include and exclude path filters. I generally prefer we explicitly have an include list for the paths we want to trigger this. That way it is clearer what we expect to trigger this pipeline and only the people that care about this pipeline will need to edit it, as opposed to other folks that don't care about it but have to keep adding excludes to it. Of course another issue that will likely occur is that when a new directory is added someone will have to remember to add the path otherwise no pipeline will trigger.

[jsquire]

For this, I was following the example that had been shared with me during planning. Considering Wes' feedback, I'd prefer to have the inclusion by default with the rationale that its better to inadvertently include something that you need to exclude later than to forget to add something and not realize that it isn't included in CI.

[weshaggard]

Sorry I'm a little late to the party here but for project exclusions it was my goal that individual service directories could include a .projects file (example https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/search/service.projects). That way the services could apply whatever logic they wanted in msbuild to exclude/include different projects based on whatever condition was interesting to them. Doing it this way would prevent to pass a set of project paths but instead could just be controlled by a simple boolean condition.

[jsquire]

I'll take a look at this tomorrow and see if I can pivot to that, if that is the preference. Based on our initial short conversation, my goal was to mimic the exclude style from the CI pipelines so that it presented a consistent experience for the functionality.

[weshaggard]

We have tried to avoid using terms like legacy. If we are to follow our other naming conventions we should call these ci.data.yml to represent the track 1 data plane libraries.

[jsquire]

In this case, "legacy" is the terminology that was agreed upon with the service team. I'd prefer to keep the nomenclature that is used within the ecosystem to avoid confusion. When I looked through the pipelines, most services with multiples appear to be split between "data" and "mgmt"; I never would have guessed that "data" was referring to T1.

[weshaggard]

yeah we were trying to keep "empty" (i.e. ci.yml) as the forward looking track 2 set, also sometimes called client and then for any track1 data plane we were using "ci.data.yml" or "ci.mgmt.yml" for mgmt.

[jsquire]

If I'm being honest, I find that pattern confusing, but I don't feel strongly enough to swim upstream. I'll adjust to the recommendation.

[weshaggard]

There is no great convention but I believe it is better then using the term "legacy" and it at least has some consistency across different areas and languages.