Group Kiota PHP repositories

[andrueastman]

Related to microsoft/kiota#4636 and microsoft/kiota-dotnet#238

As a language specific concern, we should look into the feasibility of grouping the repos and outline if this is not feasible in this issue.

Sub-tasks:

Investigation

Created PoC monorepo org and projects for testing https://github.com/sample-monorepo/sample-monorepo

• Configuring composer to link multiple packages within same folder as dependencies of each other: sample config
• Tooling to manage mono-repo:
  Found monorepo-builder: Provides tooling to validate dependency versions across each sub-package in the monorepo
• Release automation from monorepo back to individual repositories for tagging & update to Packagist:
  Managed to use a GitHub app that can be used from the split workflow
• Find way to link Git history to allow merges from split folder:
  Currently, after splitting changes to indiv. repo, git history is unrelated, meaning no PR can be created: example

Implementation

• Create new repository
  • Add README, CHANGELOG etc
• Create GitHub app for pushing changes from mono-repo back to individual repos for release
  • Store APP ID in mono-repo variables
  • Store GitHub app private-key in Azure Key Vault
• Move code changes from individual repos to mono-repo preserving Git history
• Update composer config of each sub-project
• Update build workflows to run tests & static analysis etc across entire project
• Add release please configuration for mono-repo
• Add release workflow to isolate each sub-package's changes & create PR to individual repos for tagging
• Add release please to individual repos workflows to release on PR merge from mono-repo split workflow
• Add Kiota bundle package
• Port GitHub issues from individual repos to mono-repo
• Disable creation of new issues on individual repos
• Update README of individual repos to point customers to mono-repo
• Validate SonarCloud checks for each package

[baywet]

Whenever this is completed we should also:

• update the quickstart samples feat: upgrades samples to bundle kiota-samples#6457
• update the quickstart documenation feat: adds dependencies filter documentation MicrosoftDocs/openapi-docs#110

[Ndiritu]

(Subject to future edits as investigation continues, but edits will be listed here)

Composer (dependency manager) supports linking between sub-projects, however Packagist (dependency repository) doesn't support deployments from sub-packages. Packagist expects a package to be hosted within a GitHub repository whose root folder contains a composer.json file. Reference

Currently, package deployment works like this, our PHP repos have a Packagist GitHub app set up & a webhook that triggers publishing to Packagist when a tag is created. The version of the packages is determined by the tag or adding a "version" property to the composer.json.

With a mono-repo, our limitation would be in the deployment process.

Various tools exist that split the mono-repo into repositories during deployment & pushes commits & tags to the new repos so that deployment happens.

Two major tools seem viable for this:

Mono-repo builder

which provides various CLI utilities to validate our dependency versions across sub-projects & release automations which we shouldn't need courtesy of release-please. It does the repo split using a release workflow with the monorepo-split-github-action & providing it with repo:write permissions to split the repository and publish. Mono-repo builder has ~3M+ installs & seems stable at v11.x

Splitsh-lite

which does only the repo subtree split and has been used by Google Cloud PHP mono-repo to deploy their various service libraries as separate packages ref. They augment this with their own custom scripts.

Mono-repo builder seems most viable however it still leaves us potentially with our individual repositories for abstractions, http etc but we can manage everything via a kiota-php repository.

Pending:

• Prototype using Mono-repo builder to find any further pitfalls

[Ndiritu]

Open to further feedback here @andrueastman, @baywet, @shemogumbe

[baywet]

Thank you for the additional information.

So if I'm following things properly, in those scenarios we'd:

• create a mono-repo.
• migrate the "live" version of the code there.
• setup some kind of automation to replicate onto existing repos. (so that packagist webhook is triggered, and the platform has "something to refer to")

Or am I missing anything here?

[Ndiritu]

Yes, create a mono-repo. We could keep the current repos for the core libraries for to re-use the current Packagist webhook configs.
I will confirm all details once I run a PoC within the week.

[baywet]

Thanks! The two things I'd want us to get clarity on before we make any decision are:

• Is there a chance that the "packages from the monorepo" could appear in packagist by mistake? that people could take a dependency on them?
• Would "deleting" the existing repos be a breaking change to people who already have a reference to the packages on packagist? (I'm fairly confident the answer to this one is yes)

[Ndiritu]

Is there a chance that the "packages from the monorepo" could appear in packagist by mistake? that people could take a dependency on them?

No. We will not be deploying from the mono-repo and we do not need to set up the Packagist GitHub hook/app on the mono-repo.
We can tag and do a release on GitHub with a CHANGELOG to track changes but these won't get to Packagist without the config described.

Would "deleting" the existing repos be a breaking change to people who already have a reference to the packages on packagist? (I'm fairly confident the answer to this one is yes)

For this to work, we won't be deleting existing repos. We'll retain the various repos and re-use their existing Packagist configurations so that once changes are tagged on the mono-repo, we push the changes of each sub-project to the corresponding existing GitHub repo with a tag that will trigger an update on Packagist.

We could disable issues & PRs to the existing Kiota core PHP repos, dependabot etc and update the README to point customers to the mono-repo.

[Ndiritu]

Here's a sample set-up of the mono-repo with corresponding individual repos for http and abstractions.

Here's a hacky workflow that simulates how release-please could create a tag that is cascaded to the individual repos. Individual repos are tagged & the update to Packagist happens: abstractions, http

The updates will require a fine-grained PAT set up under the Microsoft Graph org with content read/write permissions on the specific Kiota core PHP repos. We'd store this as a secret that the mono-repo's release workflow can access.

[Ndiritu]

If there are no further concerns/questions @baywet, we can create the kiota-php mono-repo & I can start making the changes.

[baywet]

Thank you for the additional information.

I do have some concerns about any solution requiring PATs after the latest security requirements announcements. Additionally, with the forced roll out of policies, we won't be able to sync things without a pull request.

Do you think we could use an application for the sync instead? How much setup is that going to require?

[Ndiritu]

Realized we can use git subtree split to separate the changes in the individual folders and sync the changes to the individual repos via git using the contents:read permissions - buggy sample but illustrates this

We can push release updates to a new branch on the individual repos & either auto-merge or merge those manually which would trigger release-please on the specific repos & publishing happens.

[Ndiritu]

Running into some trouble pushing to the individual repositories due to access issues. The GitHub token doesn't have scopes to push across repos in an org. Looking for a GitHub app that might be able to do this.

[Ndiritu]

@baywet finally got the PoC working.

To summarize the investigation:

• Retain all Kiota core PHP repos and use their existing Packagist configurations for releases
• Create kiota-php mono-repo
• Create sub-projects within mono-repo with existing Git history (git subtree add -P packages/abstractions [email protected]:microsoft/kiota-abstractions-php.git) etc

Release process:

• We'll need to split changes from individual sub-projects back to the main repos
• Create GitHub app with permissions to write to all existing Kiota core PHP repos.
• On new tag creation on the mono-repo, trigger a workflow to split sub-project changes & push to parent repo - example workflow
• Workflow splits changes & creates a new branch on the parent repo
• Add automation to parent repos to auto-create PR on push to some release branch pattern - sample diff
• On merging PR to main in the parent repo, review & merge release-please PR

Lmk if you have any additional thoughts/concerns

[baywet]

Thanks for the deep dive here!
I did have a follow up question: why would we use subtrees instead of submodules?

[Ndiritu]

Submodules would mean that each change to a sub-folder is first pushed to the sub-project's remote & merged to the sub-project's main branch... meaning we'd still be reviewing PRs on the sub-projects. Probably cleaner when the sub-project is a dependency - pulling latest changes is easier. Here's a sample monorepo with submodule setup

Subtrees pull the history of the sub-project making the monorepo the new source of truth. The individual repos would be only for release purposes. Cleaner to make changes

[baywet]

Right, this is not an aggregation case but rather a "publishing" case. We want the source of truth to be on the main repo. Sorry about the confusion here for a second.

Thank you for this deep investigation here! (I just saw you created an org & co)

I'm happy with the solution, @andrueastman ?

[andrueastman]

Thanks for this as well @Ndiritu

I'm happy with this as well. Just one question though on this.

Add automation to parent repos to auto-create PR on push to some release branch pattern - sample diff

Will this mean that multiple PRs are created on each of the existing repo? If so, would it make sense to have these repos only writable by the automation(so that it just writes the source and tag directly to the main branch) so that we have less overhead of approving/merging the PRs?

[Ndiritu]

@andrueastman I think that would be more ideal but I think @baywet had mentioned that there may be a requirement to have PRs as mandatory on our repos. Could you please clarify?

Also, when ready, could you please create the kiota-php repository?

[baywet]

The security teams are eventually going to roll out branch policies across all repositories mandating the use of pull requests to merge anything to the main branches. The timelines are not clear at this point, but the last time I heard of it, they were talking about this quarter.

[baywet]

Repo created https://github.com/microsoft/kiota-php/