Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arpack: use openblas on darwin; fix build; enable eigen #367594

Merged
merged 4 commits into from
Jan 2, 2025

Conversation

paparodeo
Copy link
Contributor

@paparodeo paparodeo commented Dec 23, 2024

change 07cdea2 uncovered a misconfiguration in the darwin build resulting in some tests failing / segfaulting. Switch to using openblas which has no issues with the tests. Additionally, enable eigen as it was part of the build inputs but was unused as EIGEN=ON was not set in the build flags. Finally, added an option to use Accelerate building with the flags -ff2c -fno-second-underscore found https://github.com/opencollab/arpack-ng/blob/804fa3149a0f773064198a8e883bd021832157ca/.github/workflows/jobs.yml#L184-L192.

FWIW: homebrew also uses openblas. https://github.com/Homebrew/homebrew-core/blob/master/Formula/a/arpack.rb

#357259
https://hydra.nixos.org/build/282505341

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Dec 23, 2024
@paparodeo paparodeo changed the title arpack: use openblas on darwin arpack: use openblas on darwin; fix build Dec 23, 2024
@ofborg ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Dec 23, 2024
@ofborg ofborg bot requested review from dotlambda and ttuegel December 23, 2024 22:29
@paparodeo
Copy link
Contributor Author

cc: @fedeinthemix @SuperSandro2000

@fedeinthemix
Copy link
Contributor

I'm not familiar with the Darwin platform and don't have access to it. Therefore, I can't provide meaningful comments. However, the information at https://developer.apple.com/documentation/accelerate/blas/ may be relevant.

@paparodeo
Copy link
Contributor Author

paparodeo commented Dec 27, 2024

I'm not familiar with the Darwin platform and don't have access to it. Therefore, I can't provide meaningful comments. However, the information at https://developer.apple.com/documentation/accelerate/blas/ may be relevant.

thanks. we've had this issue with other packages and the solution is to use the nixpkgs default (openblas). mostly cc'd cause this change either needs to move forward or the other one backed out given that darwin is currently broken.

cc: @emilazy

Copy link
Member

@emilazy emilazy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. (Is Accelerate.framework not handling single precision a known thing and definitely not something we’re holding wrong somehow?)

@paparodeo
Copy link
Contributor Author

paparodeo commented Dec 27, 2024

Sure. (Is Accelerate.framework not handling single precision a known thing and definitely not something we’re holding wrong somehow?)

https://github.com/Homebrew/homebrew-core/blob/master/Formula/a/arpack.rb is also using openblas tho they don't seem to be enabling the feature which caused tests to segfault.

@paparodeo
Copy link
Contributor Author

paparodeo commented Dec 28, 2024

Sure. (Is Accelerate.framework not handling single precision a known thing and definitely not something we’re holding wrong somehow?)

tho they seem to make it work: https://github.com/opencollab/arpack-ng/actions/runs/11434914625/job/31809312118

https://github.com/opencollab/arpack-ng/blob/804fa3149a0f773064198a8e883bd021832157ca/.github/workflows/jobs.yml#L184-L192

the FFLAGS="-ff2c -fno-second-underscore" seem to have something to do with making it not-segfault I think with accelerate. and EXAMPLES=on gets more tests however still fails 2 tests.

         83 - issue401_tst (Failed)
         84 - issue215_tst (Failed)

but using openblas (and without the FFLAGS) it fails the same tests. [edit] these failures only happen when EIGEN=ON

@paparodeo paparodeo marked this pull request as draft December 28, 2024 03:35
@paparodeo
Copy link
Contributor Author

paparodeo commented Dec 28, 2024

the FFLAGS="-ff2c -fno-second-underscore" seem to have something to do with making it not-segfault I think with accelerate. and EXAMPLES=on gets more tests however still fails 2 tests.

         83 - issue401_tst (Failed)
         84 - issue215_tst (Failed)

ok. so those tests fail on all platforms when EIGEN=ON which is what their test runners use but we don't set but i think we might've wanted to set as that uses the eigen library which is in the build inputs. -ff2c is required for Accelerate to not crash. So when FFLAGS=-ff2c and EIGEN=OFF (the default) the tests all pass with Accelerate. So that's good.

I suppose now to figure out why when EIGEN=ON the above tests fail on all platforms (linux & darwin)

[edit] tho https://gcc.gnu.org/onlinedocs/gfortran/Code-Gen-Options.html -ff2c' implies '-fsecond-underscore' and with accelerate it seems to compile and run tests fine if -fno-second-underscore` is used or not.

@paparodeo
Copy link
Contributor Author

         83 - issue401_tst (Failed)
         84 - issue215_tst (Failed)

ok -- enableParallelChecking = false; fixes these.

Change 07cdea2 uncovered a bug in how
we build arpack on darwin with the Accelerate. Disable the Accelerate
framework and use openblas the default nixpkgs blas/lapack
implementation.
@paparodeo paparodeo changed the title arpack: use openblas on darwin; fix build arpack: use openblas on darwin; fix build; enable eigen Dec 28, 2024
@github-actions github-actions bot added 10.rebuild-linux: 101-500 and removed 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Dec 28, 2024
@paparodeo
Copy link
Contributor Author

paparodeo commented Dec 28, 2024

OK. so updated the PR to now enable EIGEN=ON and parallel tests are disabled. Darwin still defaults to openblas with an option to enable Accelerate. Tests pass with ether option.

this is basically what the initial PR did (default to openblas for darwin) but after spending some time looking at the derivation made a few updates.

@paparodeo
Copy link
Contributor Author

@ofborg build arpack.passthru.tests arpack

@paparodeo paparodeo marked this pull request as ready for review December 28, 2024 07:11
@paparodeo
Copy link
Contributor Author

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 367594


x86_64-linux

⏩ 9 packages marked as broken and skipped:
  • octavePackages.fem-fenics
  • octavePackages.level-set
  • octavePackages.parallel
  • octavePackages.sparsersb
  • octavePackages.tisean
  • octavePackages.vibes
  • octavePackages.vrml
  • python312Packages.textnets
  • python312Packages.textnets.dist
❌ 23 packages failed to build:
  • hal-hardware-analyzer
  • librsb
  • octavePackages.bim
  • octavePackages.fits
  • octavePackages.msh
  • python312Packages.skytemple-ssb-debugger
  • python312Packages.skytemple-ssb-debugger.dist
  • python313Packages.explorerscript
  • python313Packages.explorerscript.dist
  • python313Packages.igraph
  • python313Packages.igraph.dist
  • python313Packages.kmapper
  • python313Packages.kmapper.dist
  • python313Packages.leidenalg
  • python313Packages.leidenalg.dist
  • python313Packages.skytemple-dtef
  • python313Packages.skytemple-dtef.dist
  • python313Packages.skytemple-files
  • python313Packages.skytemple-files.dist
  • python313Packages.skytemple-ssb-debugger
  • python313Packages.skytemple-ssb-debugger.dist
  • skytemple
  • skytemple.dist
✅ 97 packages built:
  • LPCNet
  • arpack
  • arpack-mpi
  • calculix-ccx
  • checkov
  • checkov.dist
  • flattenReferencesGraph
  • flattenReferencesGraph.dist
  • gama
  • igraph
  • igraph.dev
  • igraph.doc
  • jupyter-all
  • libleidenalg
  • minc_widgets
  • octave
  • octaveFull
  • octavePackages.arduino
  • octavePackages.audio
  • octavePackages.bsltl
  • octavePackages.cgi
  • octavePackages.communications
  • octavePackages.control
  • octavePackages.data-smoothing
  • octavePackages.database
  • octavePackages.dataframe
  • octavePackages.dicom
  • octavePackages.divand
  • octavePackages.doctest
  • octavePackages.econometrics
  • octavePackages.financial
  • octavePackages.fpl
  • octavePackages.fuzzy-logic-toolkit
  • octavePackages.ga
  • octavePackages.general
  • octavePackages.generate_html
  • octavePackages.geometry
  • octavePackages.gsl
  • octavePackages.image
  • octavePackages.image-acquisition
  • octavePackages.instrument-control
  • octavePackages.interval
  • octavePackages.io
  • octavePackages.linear-algebra
  • octavePackages.lssa
  • octavePackages.ltfat
  • octavePackages.mapping
  • octavePackages.matgeom
  • octavePackages.miscellaneous
  • octavePackages.mvn
  • octavePackages.nan
  • octavePackages.ncarray
  • octavePackages.netcdf
  • octavePackages.nurbs
  • octavePackages.ocl
  • octavePackages.octclip
  • octavePackages.octproj
  • octavePackages.optics
  • octavePackages.optim
  • octavePackages.optiminterp
  • octavePackages.quaternion
  • octavePackages.queueing
  • octavePackages.signal
  • octavePackages.sockets
  • octavePackages.splines
  • octavePackages.statistics
  • octavePackages.stk
  • octavePackages.strings
  • octavePackages.struct
  • octavePackages.symbolic
  • octavePackages.tsa
  • octavePackages.video
  • octavePackages.windows
  • octavePackages.zeromq
  • octopus
  • openems
  • python312Packages.explorerscript
  • python312Packages.explorerscript.dist
  • python312Packages.igraph
  • python312Packages.igraph.dist
  • python312Packages.kmapper
  • python312Packages.kmapper.dist
  • python312Packages.leidenalg
  • python312Packages.leidenalg.dist
  • python312Packages.python-csxcad
  • python312Packages.python-csxcad.dist
  • python312Packages.python-openems
  • python312Packages.python-openems.dist
  • python312Packages.scikit-tda
  • python312Packages.scikit-tda.dist
  • python312Packages.skytemple-dtef
  • python312Packages.skytemple-dtef.dist
  • python312Packages.skytemple-files
  • python312Packages.skytemple-files.dist
  • python313Packages.python-csxcad
  • python313Packages.python-csxcad.dist
  • vpv
all failures currently broken on master
Build Status for hal-hardware-analyzer.x86_64-linux on master
✖ (Failed) hal-hardware-analyzer-4.4.1 from 2024-12-23 - https://hydra.nixos.org/build/282858541
Build Status for librsb.x86_64-linux on master
✖ (Failed) librsb-1.2.0.10 from 2024-12-23 - https://hydra.nixos.org/build/282862544
Build Status for octavePackages.bim.x86_64-linux on master
✖ (Dependency failed) octave-9.3.0-bim-1.1.6 from 2024-12-23 - https://hydra.nixos.org/build/282827223
Build Status for octavePackages.fits.x86_64-linux on master
✖ (Failed) octave-9.3.0-fits-1.0.7 from 2024-12-23 - https://hydra.nixos.org/build/282836762
Build Status for octavePackages.msh.x86_64-linux on master
✖ (Dependency failed) octave-9.3.0-msh-1.0.12 from 2024-12-23 - https://hydra.nixos.org/build/282906993
Build Status for python312Packages.skytemple-ssb-debugger.x86_64-linux on master
✖ (Dependency failed) python3.12-skytemple-ssb-debugger-1.8.2 from 2024-12-23 - https://hydra.nixos.org/build/282733000
Build Status for python313Packages.explorerscript.x86_64-linux on master
✖ (Dependency failed) python3.13-explorerscript-0.2.1.post2 from 2024-12-23 - https://hydra.nixos.org/build/282874744
Build Status for python313Packages.igraph.x86_64-linux on master
✖ (Dependency failed) python3.13-igraph-0.11.8 from 2024-12-23 - https://hydra.nixos.org/build/282752950
Build Status for python313Packages.kmapper.x86_64-linux on master
✖ (Dependency failed) python3.13-kmapper-2.1.0 from 2024-12-23 - https://hydra.nixos.org/build/282865902
Build Status for python313Packages.leidenalg.x86_64-linux on master
✖ (Dependency failed) python3.13-leidenalg-0.10.2 from 2024-12-23 - https://hydra.nixos.org/build/282892400
Build Status for python313Packages.skytemple-dtef.x86_64-linux on master
✖ (Dependency failed) python3.13-skytemple-dtef-1.6.1 from 2024-12-23 - https://hydra.nixos.org/build/282834043
Build Status for python313Packages.skytemple-files.x86_64-linux on master
✖ (Dependency failed) python3.13-skytemple-files-1.8.3 from 2024-12-23 - https://hydra.nixos.org/build/282735686
Build Status for python313Packages.skytemple-ssb-debugger.x86_64-linux on master
✖ (Dependency failed) python3.13-skytemple-ssb-debugger-1.8.2 from 2024-12-23 - https://hydra.nixos.org/build/282819748
Build Status for skytemple.x86_64-linux on master
✖ (Dependency failed) skytemple-1.8.3 from 2024-12-23 - https://hydra.nixos.org/build/282899697

use ninja, cmakeBool and remove unneeded install_name_tool usage in
postFixup as the library name is already properly set by the build
The arpack package included the eigen library and added it to the build
inputs but neglected to enable its use by setting `EIGEN=ON` in the
build flags.  Enable support for eigenvalue-problems solver based on ICB
and eigen and disable parallel checking as the tests fail when run in
parallel.
Copy darwin flags `-ff2c -fno-second-underscore` from workflow to use
Accelerate without segfaulting / failing tests and enable the Accelerate
framework when `useAccel` is true.
https://github.com/opencollab/arpack-ng/blob/804fa3149a0f773064198a8e883bd021832157ca/.github/workflows/jobs.yml#L184-L192
Copy link
Member

@emilazy emilazy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good – is there a reason not to default to Accelerate for Darwin though since apparently it’s what upstream defaults to and tests in CI? (And also maybe the last commit should be squashed into the first one given the different approach?)

@paparodeo
Copy link
Contributor Author

paparodeo commented Dec 31, 2024

Looks good – is there a reason not to default to Accelerate for Darwin though since apparently it’s what upstream defaults to and tests in CI? (And also maybe the last commit should be squashed into the first one given the different approach?)

I left openblas because it works on linux and that's what this was using before the SDK refactor and we've seen some sketchy Accelerate behavior so it seemed like a safer option. not married to the choice tho

@wegank wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Jan 1, 2025
@tomberek
Copy link
Contributor

tomberek commented Jan 2, 2025

Builds on aarch64-darwin.
Builds on aarch64-linux.
Builds on x86_64-darwin.

@tomberek tomberek merged commit b3adb5f into NixOS:master Jan 2, 2025
49 of 50 checks passed
@paparodeo paparodeo deleted the arpack-fix-darwin branch January 2, 2025 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: darwin Running or building packages on Darwin 10.rebuild-darwin: 11-100 10.rebuild-linux: 101-500 12.approvals: 1 This PR was reviewed and approved by one reputable person
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants