Refactor some dependency handling to fix various API dependency issues

[Bo98]

This PR tries to tackle several of the "harder" issues involving API dependencies. Basically the stuff that required deeper refactoring rather than a hack-on fix.

There were several cases where the dependency list was incorrect. Usually this erred on the side of too many dependencies rather than too little, so hasn't seen widespread bug reports beyond a few comments here and there.

Notably:

• The dependency list had parts that varied depending on the machine that generated the API JSON rather than the consumer.
• Stable-only or HEAD-only dependencies were not correctly considered.
• uses_from_macos "example", since: ... did not work correctly.

This changes in this PR are:

• Removed TapDependency (this makes the rest of the PR a bit simpler), folding it into Dependency
• Refactored uses_from_macos handling. Instead of being a conditional depends_on, it now consistently adds a UsesFromMacOSDependency on all platforms with the checks now taking place in installed?. This provides a more consistent behaviour on all platforms, and has a bonus of reducing the bloat of the variations part of the JSON.
• Introduced an internal auto tag, which is now applied to all dependencies not explicitly added via a depends_on. This fixes an issue where some of the "if needed" deps leaked from the machine that generated the API JSON. For example, if a formula uses xz and the machine that generated the API JSON did not have xz, it will now no longer appear on the API dependency list and instead be determined by the API consumer.
• Changed the API hash and Formulary to support all these changes. Some fields in the hash were deprecated as a result as some changes were unavoidably backwards incompatible given the basic array of strings we had before simply does not have the flexibility to store the additional information added in this PR like a hash/dictionary does. (Requirements already used a hash so the existing field worked there.)

This is draft because rspec tests need adjusting and basic commands are probably very broken as is. But the general implementation as a whole should be done, with just bug/typo fixes needed.

[MikeMcQuaid]

Looking good so far!

My main concern is the migration path for API consumers here and avoiding making API consumers having to rewrite their code for a change that none of them having really complained about.

Somewhat relatedly: I've been thinking for a little while that I think there could be benefit in a JSON output/API subset that is strictly what we need and not what API consumers do. Now that we have formula.jws.json and cask.jws.json: I 100% guarantee that no-one except us is decoding and using these files except us so they seem like a very good candidate for slimming down API fields to the bare minimum that we need/consume to make those files smaller.

For other cases e.g. brew info --json output and the formula.json files: I'm tempted to say we don't deprecate those old fields and just leave them around indefinitely. They aren't really size-sensitive and not breaking our (pretty much unversioned) public APIs is much more appealing than slimming down these files.

Thanks again for the PR!

[carlocab]

Will have a look at this PR later, but since we're looking at refactoring dependency handling:

Some of the "auto" dependencies (e.g. xz when the source tarball is a .tar.xz, or curl when the url has using: :homebrew_curl) get added to the build environment by superenv, which leads to opportunistic linkage (see icecast.rb and vorbis-tools.rb for hacks we've used to try to avoid this). It would be nice if we could get rid of this (if this PR doesn't handle that already).

[Bo98]

Thanks for reminding me. I knew there was another reason for tagging those dependencies as this wasn't the first time I had the idea to do that but I had forgotten what that other reason was.
I will indeed make that change.

[MikeMcQuaid]

It would be nice if we could get rid of this (if this PR doesn't handle that already).

Agreed 👍🏻

I will indeed make that change.

🎉

[Bo98]

Some of the "auto" dependencies (e.g. xz when the source tarball is a .tar.xz, or curl when the url has using: :homebrew_curl) get added to the build environment by superenv, which leads to opportunistic linkage (see icecast.rb and vorbis-tools.rb for hacks we've used to try to avoid this). It would be nice if we could get rid of this (if this PR doesn't handle that already).

fd411c2 will probably handle this (untested).

Though that won't necesssarily stop opportunistic linkage from simply being installed, should any formula do that - but that's not specific to implicit dependencies. The change should however stop it from being added to the superenv variables.

[MikeMcQuaid]

@Bo98 been following along with this but just FYI I'll hold off another proper review until it's non-draft. Feel free to nudge me in Slack if I miss that!

[MikeMcQuaid]

Looks great, thanks again @Bo98!

[MikeMcQuaid]

Not blocking: given how often we do this all over the place: feels like we should have a better accessor for this sort of thing.

[Bo98]

Probably.

I've tweaked this slightly in the meantime to:

@tap = Tap.fetch(Regexp.last_match(1), Regexp.last_match(2)) if name =~ HOMEBREW_TAP_FORMULA_REGEX