portable_formula: use [email protected] on Linux #156

danielnachun · 2022-06-20T22:37:33Z

This PR should allow us to build the portable formulae on Linux runners with newer glibc. We won't need to rebuilt the bottles since they are currently build against glibc 2.13 in the Wheezy container - the goal here is to make sure they work.

danielnachun · 2022-06-20T22:53:36Z

I'm a little unsure how to add the [email protected] dependency here as on_linux is not defined in this class.

Bo98 · 2022-06-20T22:55:29Z

It'll need to be in self.inherited of PortableFormula, like the keg_only line.

danielnachun · 2022-06-21T01:42:05Z

Okay that got us past the first hurdle but now it's failing with this:

[email protected]: gawk 3.0 (or later) is required to build glibc.

I guess it's trying to build [email protected] from source? Is there a way to use the bottle or do we have to rebuild it here for the test?

Bo98 · 2022-06-21T01:54:46Z

You'll need to special case install [email protected] without the --build-bottle here:

homebrew-portable-ruby/cmd/portable-package.rb

Lines 37 to 40 in a213023

    
           deps = Utils.safe_popen_read("brew", "deps", "-n", "--include-build", name).split("\n") 
        
           # Avoid installing glibc. Bottles depend on glibc. 
        
           safe_system "brew", "install", "--build-bottle", *verbose, *deps

danielnachun · 2022-06-21T03:52:22Z

I realized we needed to test this with Ubuntu 16.04, so I changed the Dockerfile to use that instead. Now I'm stuck with

fatal: not in a git directory

Presumably I'm missing a step in the Dockerfile, though I'm not exactly sure what to change.

Bo98 · 2022-06-21T04:34:12Z

cmd/portable-package.rb

        deps = Utils.safe_popen_read("brew", "deps", "-n", "--include-build", name).split("\n")

+        if deps.include?("[email protected]")
+          safe_system "brew", "install", "--ignore-dependencies", *verbose, "[email protected]"
+          deps -= ["[email protected]"]
+        end


[email protected] will still be there.

Maybe try something like (variable names could maybe be better):

Suggested change

deps = Utils.safe_popen_read("brew", "deps", "-n", "--include-build", name).split("\n")

if deps.include?("[email protected]")

safe_system "brew", "install", "--ignore-dependencies", *verbose, "[email protected]"

deps -= ["[email protected]"]

end

deps_to_install_from_bottles = ["[email protected]"]

deps = Dependency.expand(Formula[name], cache_key: "portable-package-#{name}") do |dependent, dep|

Dependency.prune if dep.test? || dep.optional?

next if deps_to_install_from_bottles.include?(dep.name)

Dependency.keep_but_prune_recursive_deps

end.map(&:name)

bottled_deps, deps = deps.partition { |dep| deps_to_install_from_bottles.include?(dep) }

safe_system "brew", "install", *verbose, *bottled_deps

I did an array and maybe that's overkill but you get the point hopefully.

Bo98 · 2022-06-21T04:35:55Z

I realized we needed to test this with Ubuntu 16.04, so I changed the Dockerfile to use that instead. Now I'm stuck with
fatal: not in a git directory
Presumably I'm missing a step in the Dockerfile, though I'm not exactly sure what to change.

I'll take a look tomorrow but maybe try pass --debug for more information.

danielnachun · 2022-06-21T05:43:34Z

I added --debug to the output but it's still not clear to me what the cause of the problem is.

cmd/portable-package.rb

Bo98 · 2022-06-22T14:48:36Z

That error is really weird. The .git directory exists but git says we're not in a git directory? I'll see if I can try reproduce locally.

danielnachun · 2022-06-22T17:23:45Z

I can't reproduce it locally by just using docker run -it homebrew/ubuntu16.04:master, so I'm assuming there's something about the Dockerfile that's causing this problem.

iMichka · 2022-06-25T19:55:26Z

Maybe it has to do with the way we mount the core repo.
What does ${GITHUB_REPOSITORY,,} do? Is that a typo? https://github.com/Homebrew/homebrew-portable-ruby/blob/master/.github/workflows/build.yml#L45

danielnachun · 2022-06-26T01:10:13Z

Maybe it has to do with the way we mount the core repo.
What does ${GITHUB_REPOSITORY,,} do? Is that a typo? https://github.com/Homebrew/homebrew-portable-ruby/blob/master/.github/workflows/build.yml#L45

That does look very odd, but I pushed a commit to change it and it didn't seem to help. I'm still not sure how to try to reproduce this locally. I think I have to run the Dockerfile locally with docker but I'm still figuring that out.

Bo98 · 2022-06-26T02:35:49Z

What does ${GITHUB_REPOSITORY,,} do? Is that a typo?

No that's intentional - it converts it to lowercase.

$ GITHUB_REPOSITORY=Homebrew/homebrew-portable-ruby; echo ${GITHUB_REPOSITORY,,}
homebrew/homebrew-portable-ruby

Really we can ditch the Dockerfile. I can push the CI changes needed to do that, though it would be nice to know how we're managing to get past the several git repo checks in brew as that seems like a bug. Permissions?

Bo98 · 2022-07-05T14:31:06Z

Our "Test Portable Ruby" step apparently has never worked properly on macOS since it's impossible to specify an external Ruby to be used. It also didn't work on Linux because HOMEBREW_RUBY_PATH is ignored, but I've fixed that to use PATH now.

MikeMcQuaid · 2022-07-21T09:31:40Z

How do we restart CI here? I don't see a button like I've used with homebrew-core and brew.

I've rerun them. Presumably you don't have sufficient permissions.

Bo98 · 2022-07-21T12:43:36Z

Also the 10.11 build failed in the "Test Portable Ruby" step but didn't really output anything about why.

In the last push I fixed this step so it would actually test something - i.e. it now actually checks if it is using the just-built Ruby.

For months/years it's never actually tested that properly. For months/years it's actually been using the existing Portable Ruby that brew installs.

I've got it working on Linux but it seems to still be using it's own Portable Ruby on macOS for some reason. I think because the removal of the existing Portable Ruby in brew cleanup doesn't take HOMEBREW_USE_RUBY_FROM_PATH into consideration so either we change that or we remove the existing Portable Ruby install more directly in the workflow rather than relying on brew cleanup, which I'll probably do.

Bo98 · 2022-07-21T13:14:04Z

Looks like HOMEBREW_USE_RUBY_FROM_PATH doesn't actually work on macOS. I can probably just make the step Linux only for now.

danielnachun · 2022-07-22T18:40:38Z

I made the "Test Portable Ruby" step Linux-only and 10.11 now works, but now ARM isn't. I messed up pulling the other commits added here and overwrote some by accident and that may be why ARM isn't working, though I thought I managed to restore everything.

Bo98 · 2022-08-02T14:41:28Z

Sorry for the delay - I took the week off.

I think the proper fix is still on the brew side. Skipping tests is not really acceptable in the long term, even if it has always been an issue. I'll take a look later today.

danielnachun · 2022-08-04T04:48:33Z

Should we rerun this with testing of Portable Ruby enabled on macOS when Homebrew/brew#13644 is merged?

MikeMcQuaid · 2022-08-04T10:39:50Z

Should we rerun this with testing of Portable Ruby enabled on macOS when Homebrew/brew#13644 is merged?

Did this now @danielnachun 👍🏻

Bo98 · 2022-08-04T15:58:18Z

10.11 works now. 11-arm64 however refuses to use the Ruby on the path:

Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby

Bo98 · 2022-08-04T16:04:10Z

Looks like HOMEBREW_USE_RUBY_FROM_PATH doesn't actually prioritise from the user PATH as I thought - it prioritises /usr/bin over anything, and it just happens to work on 10.11 because /usr/bin/ruby there is so ancient it can't run the version check script.

github-actions · 2022-08-26T00:09:56Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

MikeMcQuaid · 2022-08-26T07:33:01Z

@danielnachun Is this needed for 3.6.0/the glibc changes?

Bo98 · 2022-08-26T11:53:32Z

This is only needed to deprecate the Debian 7 Docker image. No plans to ship a new version of portable-ruby - existing bottles should be fine.

This is ready to go, just need to fix HOMEBREW_USE_RUBY_FROM_PATH over in brew first.

MikeMcQuaid · 2022-08-26T11:55:05Z

This is only needed to deprecate the Debian 7 Docker image. No plans to ship a new version of portable-ruby - existing bottles should be fine.

Ok. Sounds like this may also be worth doing as part of 3.6.0.

danielnachun · 2022-08-26T17:29:41Z

It would be great to get this in with 3.6.0. I've tried to look into this and haven't figure anything out yet.

MikeMcQuaid · 2022-08-31T18:06:28Z

Hurrah, thanks for getting this over the line, both!

danielnachun force-pushed the use_glibc_2_13 branch from b3d5cee to e4387c9 Compare June 21, 2022 01:35

danielnachun force-pushed the use_glibc_2_13 branch 5 times, most recently from 9637eaa to eb3f429 Compare June 21, 2022 03:44

Bo98 reviewed Jun 21, 2022

View reviewed changes

danielnachun force-pushed the use_glibc_2_13 branch 3 times, most recently from 0e14042 to 7e961a7 Compare June 21, 2022 05:31

MikeMcQuaid reviewed Jun 21, 2022

View reviewed changes

cmd/portable-package.rb Outdated Show resolved Hide resolved

danielnachun mentioned this pull request Jun 22, 2022

binutils: Build using [email protected] on Linux Homebrew/homebrew-core#92329

Closed

6 tasks

Bo98 force-pushed the use_glibc_2_13 branch 2 times, most recently from 96db95a to 1eedd01 Compare July 5, 2022 14:29

Bo98 force-pushed the use_glibc_2_13 branch 3 times, most recently from eaa89ea to 6d9e543 Compare July 5, 2022 14:43

Bo98 added 2 commits July 21, 2022 13:48

Remove Dockerfile

b48dc91

portable-libxcrypt: drop Debian 7 check

5fefaa5

Bo98 force-pushed the use_glibc_2_13 branch from 5826d82 to 5fefaa5 Compare July 21, 2022 12:49

danielnachun force-pushed the use_glibc_2_13 branch 4 times, most recently from 0c20031 to a6a237c Compare July 22, 2022 17:53

Bo98 force-pushed the use_glibc_2_13 branch from a6a237c to 5fefaa5 Compare August 4, 2022 13:27

cmd/portable-package: use HOMEBREW_BREW_FILE

fbde149

github-actions bot added the stale label Aug 26, 2022

Bo98 removed the stale label Aug 26, 2022

Bo98 merged commit fbde149 into Homebrew:master Aug 31, 2022

danielnachun deleted the use_glibc_2_13 branch September 2, 2022 17:32

github-actions bot added the outdated label Oct 3, 2022

github-actions bot locked as resolved and limited conversation to collaborators Oct 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

portable_formula: use [email protected] on Linux #156

portable_formula: use [email protected] on Linux #156

danielnachun commented Jun 20, 2022

danielnachun commented Jun 20, 2022

Bo98 commented Jun 20, 2022

danielnachun commented Jun 21, 2022

Bo98 commented Jun 21, 2022 •

edited

Loading

danielnachun commented Jun 21, 2022

Bo98 Jun 21, 2022 •

edited

Loading

Bo98 commented Jun 21, 2022

danielnachun commented Jun 21, 2022

Bo98 commented Jun 22, 2022

danielnachun commented Jun 22, 2022

iMichka commented Jun 25, 2022

danielnachun commented Jun 26, 2022

Bo98 commented Jun 26, 2022 •

edited

Loading

Bo98 commented Jul 5, 2022

MikeMcQuaid commented Jul 21, 2022

Bo98 commented Jul 21, 2022 •

edited

Loading

Bo98 commented Jul 21, 2022

danielnachun commented Jul 22, 2022

Bo98 commented Aug 2, 2022

danielnachun commented Aug 4, 2022

MikeMcQuaid commented Aug 4, 2022

Bo98 commented Aug 4, 2022

Bo98 commented Aug 4, 2022 •

edited

Loading

github-actions bot commented Aug 26, 2022

MikeMcQuaid commented Aug 26, 2022

Bo98 commented Aug 26, 2022

MikeMcQuaid commented Aug 26, 2022

danielnachun commented Aug 26, 2022

MikeMcQuaid commented Aug 31, 2022

-        deps = Utils.safe_popen_read("brew", "deps", "-n", "--include-build", name).split("\n")
-        if deps.include?("[email protected]")
-          safe_system "brew", "install", "--ignore-dependencies", *verbose, "[email protected]"
-          deps -= ["[email protected]"]
-        end
+        deps_to_install_from_bottles = ["[email protected]"]
+        deps = Dependency.expand(Formula[name], cache_key: "portable-package-#{name}") do |dependent, dep|
+          Dependency.prune if dep.test? || dep.optional?
+          next if deps_to_install_from_bottles.include?(dep.name)
+          Dependency.keep_but_prune_recursive_deps
+        end.map(&:name)
+        bottled_deps, deps = deps.partition { |dep| deps_to_install_from_bottles.include?(dep) }
+        safe_system "brew", "install", *verbose, *bottled_deps

portable_formula: use [email protected] on Linux #156

portable_formula: use [email protected] on Linux #156

Conversation

danielnachun commented Jun 20, 2022

danielnachun commented Jun 20, 2022

Bo98 commented Jun 20, 2022

danielnachun commented Jun 21, 2022

Bo98 commented Jun 21, 2022 • edited Loading

danielnachun commented Jun 21, 2022

Bo98 Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

Bo98 commented Jun 21, 2022

danielnachun commented Jun 21, 2022

Bo98 commented Jun 22, 2022

danielnachun commented Jun 22, 2022

iMichka commented Jun 25, 2022

danielnachun commented Jun 26, 2022

Bo98 commented Jun 26, 2022 • edited Loading

Bo98 commented Jul 5, 2022

MikeMcQuaid commented Jul 21, 2022

Bo98 commented Jul 21, 2022 • edited Loading

Bo98 commented Jul 21, 2022

danielnachun commented Jul 22, 2022

Bo98 commented Aug 2, 2022

danielnachun commented Aug 4, 2022

MikeMcQuaid commented Aug 4, 2022

Bo98 commented Aug 4, 2022

Bo98 commented Aug 4, 2022 • edited Loading

github-actions bot commented Aug 26, 2022

MikeMcQuaid commented Aug 26, 2022

Bo98 commented Aug 26, 2022

MikeMcQuaid commented Aug 26, 2022

danielnachun commented Aug 26, 2022

MikeMcQuaid commented Aug 31, 2022

Bo98 commented Jun 21, 2022 •

edited

Loading

Bo98 Jun 21, 2022 •

edited

Loading

Bo98 commented Jun 26, 2022 •

edited

Loading

Bo98 commented Jul 21, 2022 •

edited

Loading

Bo98 commented Aug 4, 2022 •

edited

Loading