Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V3 alias refactoring #3872

Merged
merged 10 commits into from
Jan 22, 2025
Merged

V3 alias refactoring #3872

merged 10 commits into from
Jan 22, 2025

Conversation

danielrbradley
Copy link
Member

Try to get the right information into the right places in order to implement robust auto-aliasing to address #3848

@danielrbradley danielrbradley self-assigned this Jan 17, 2025
Copy link

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

Copy link

codecov bot commented Jan 17, 2025

Codecov Report

Attention: Patch coverage is 76.34146% with 97 lines in your changes missing coverage. Please review.

Project coverage is 57.26%. Comparing base (3a9c2b5) to head (4a9ba92).
Report is 11 commits behind head on master.

Files with missing lines Patch % Lines
provider/pkg/versioning/gen.go 8.00% 23 Missing ⚠️
provider/pkg/versioning/build_schema.go 0.00% 20 Missing ⚠️
provider/pkg/resources/resources.go 76.78% 9 Missing and 4 partials ⚠️
provider/pkg/openapi/discover.go 88.00% 10 Missing and 2 partials ⚠️
provider/pkg/openapi/versioner.go 69.56% 6 Missing and 1 partial ⚠️
provider/pkg/versioning/defaultVersion.go 88.67% 5 Missing and 1 partial ⚠️
provider/pkg/gen/schema.go 95.00% 3 Missing and 1 partial ⚠️
provider/pkg/versioning/deprecations.go 55.55% 4 Missing ⚠️
provider/pkg/versioning/specs.go 50.00% 4 Missing ⚠️
provider/pkg/gen/properties.go 75.00% 0 Missing and 2 partials ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3872      +/-   ##
==========================================
+ Coverage   57.12%   57.26%   +0.13%     
==========================================
  Files          79       79              
  Lines       12423    12469      +46     
==========================================
+ Hits         7097     7140      +43     
  Misses       4799     4799              
- Partials      527      530       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Use the full word for clarity (and make my spell checker happy)!
Include spec file path and version information so it's available for context when manipulating the specification into the schema.
- Use a proper struct to represent all the information we parse to decide the name.
- Fix matching on mixed-case folder names.
- Add test case to cover mismatched namespace from file path vs spec path.
Separate alias logic from main genResources method.
This is the name we use for organisation into modules, not how Azure organises it.
These modules are only sometimes, but not always based on the Azure "Resource Provider" name.

- Change type aliases to type definitions.
- Rename associated variables.
- Switch from codegen.SortedKeys to util.SortedKeys which is generic.
API version is convertable to SDK version easier than the other way round so keep it as API version as much as possible.

- Add helper functions to ApiVersion type.
We want the whole namespace - with the prefix. Therefore we need to copy the original findNamespaceWithoutPrefixFromPath method into a method that doesn't remove the prefix.
- Validate the newly extracted namespace against the old method of extraction.
- Removing the old method of extraction is very tricky as it's not clear what's bugs or desired behaviour when skipping resources.
- Add test coverage for "well known" namespaces
- "Lock" wasn't an accurate description as it's always regenerated from the spec.
- Rename "DefaultConfigToDefaultVersionLock" to "DefaultVersionsFromConfig"
- Name as the major version number as the whole folder is the "default versions" files and this is the main result.
- Use yaml over json for less visual noise and so we can include a warning about being generated.
- Manually converted v1 & verified that the data in v2 is identical.
Copy link
Contributor

@thomas11 thomas11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except maybe the "namespace from path" logic change?

provider/pkg/gen/schema.go Outdated Show resolved Hide resolved
provider/pkg/gen/schema.go Show resolved Hide resolved
}
}
return "", nil, false
}

// Identify the first segment of the path that contains a namespace.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't taking the first namespace a change in behavior? Previously, the code used findNamespaceWithoutPrefixFromPath which takes the last.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I think the old logic is wrong (working right to left) and only works because we're only considering the explicit prefixes "Microsoft." and "PaloAltoNetworks.". For resource URIs, there's other segments with dots (in Microsoft.EventGrid has segments like .../{a}.{b}/...).

I left the old method in with an inline comparison to prove that these approaches are equivelent for the time being:

// Sanity check the new and old methods return consistent results for now.

I don't like having two methods doing nearly the same thing. If we're confident the new approach is solid, then we can take out the old approach and leave the "less hard-coded" new approach in.

The other half of the broken logic I started addressing then reverted out was the "resolved module name mismatch" errors which seems overly aggressive and is excluding real resources, but I think that needs to be a separate PR as it changes the computed resources.

provider/pkg/versioning/gen.go Show resolved Hide resolved
- Rename versionMap
- Document TokenModule
- Update docs on version files.
@danielrbradley danielrbradley merged commit ee1cc82 into master Jan 22, 2025
23 checks passed
@danielrbradley danielrbradley deleted the v3-alias-preparations branch January 22, 2025 18:32
@pulumi-bot
Copy link
Contributor

This PR has been shipped in release v2.84.0.

danielrbradley added a commit that referenced this pull request Feb 4, 2025
Due to low confidence on the migration path in v2, we added all resource
being used in the default version as explicit versions too.

With the improved aliasing approach from #3872, this is not required any
more.

See the preview generation against the v3 branch:
v3...dont-duplicate-default-as-explicit-v3-preview

This has a net reduction of 1,118,866 lines of SDK code and an
uncompressed size reduction of just under 7% to deliver a 48% overall
reduction for v3.

| SDK | v2 size | v3 size | new v3 size |
|-----|-------|------|---------|
| dotnet | 653M | 363M | 340M |
| nodejs | 300M | 166M | 156M |
| python | 717M | 401M | 376M |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants