Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

matrix-synapse: only include used optional dependencies in check #369859

Merged
merged 2 commits into from
Jan 7, 2025

Conversation

xinyangli
Copy link
Contributor

Remove unnecessary optional dependencies from nativeCheckInputs. This will make pysaml2 an optional dependencies (#367976). Unfortunately, this still doesn't fix build because of element-hq/synapse#17878, but it seems like this is more likely to be fixed sooner than IdentityPython/pysaml2#975

Related: #367976 #369303
Ping maintainers: maybe @NickCao @fadenb
others: @jcaesar (Might be interested?)

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@xinyangli xinyangli requested a review from mweinelt January 1, 2025 05:04
@xinyangli xinyangli removed the request for review from mweinelt January 1, 2025 05:06
@github-actions github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Jan 1, 2025
@mweinelt
Copy link
Member

mweinelt commented Jan 1, 2025

I think this is the wrong approach.

AIUI this makes it so that everyone uses a customized package and therefore can't rely on the cached package.

What should be done instead is most likely to override pyopenssl to fix pysaml2 for synapse.

@mweinelt mweinelt requested a review from Ma27 January 1, 2025 07:23
@xinyangli
Copy link
Contributor Author

Oh, I agree that we should override pyopenssl to a previous version.

Correct me if I'm wrong, but I think this will only require a rebuild for those who override extras in the wrapper, as the default wrapper will be cached by hydra?

Will keep this open for a bit to see if anyone has other thoughts.

@jcaesar
Copy link
Contributor

jcaesar commented Jan 1, 2025

I was considering going for removing just pysaml2 from the nativeCheckInputs

diff --git a/pkgs/servers/matrix-synapse/default.nix b/pkgs/servers/matrix-synapse/default.nix
index a1f78d9a2..cfb8594f5 100644
--- a/pkgs/servers/matrix-synapse/default.nix
+++ b/pkgs/servers/matrix-synapse/default.nix
@@ -149,7 +149,8 @@ python3.pkgs.buildPythonApplication rec {
       mock
       parameterized
     ])
-    ++ lib.flatten (lib.attrValues optional-dependencies);
+    # TODO: re-add pysaml2 when it stops being broken
+    ++ lib.flatten (lib.attrValues (optional-dependencies // { saml2 = [ ]; }));
 
   doCheck = !stdenv.hostPlatform.isDarwin;
 

and then skipping that failing test…

On the surface, downgrading pyopenssl seems wrong to me, holding security-sensitive libraries at an older version and such. But I really don't know what that lib does.

@Ma27
Copy link
Member

Ma27 commented Jan 1, 2025

AIUI this makes it so that everyone uses a customized package and therefore can't rely on the cached package.

I think most people should still get a cached package since matrix-synapse is still built by Hydra, with a modified unwrapped package internally. But it's a pretty odd fix regardless. Especially since we're trying to fix an issue in synapse that absolutely isn't a synapse problem.

From what I understood, pysaml2 has not only tests broken, after a quick look it seems as if the functionality would be degraded with current pyopenssl (correct me if I'm wrong), so skipping the tests is not an option.

So my suggestion would be to

  • mark pysaml2 as broken for the time being
  • only add pkgs to nativeCheckInputs in matrix-synapse-unwrapped that aren't broken.
  • fail hard when trying to add pysaml2 to the synapse wrapper.

Does that make sense?

@mweinelt
Copy link
Member

mweinelt commented Jan 1, 2025

Given that SAML support is probably a weird niche I think that approach is fine.

@xinyangli
Copy link
Contributor Author

Fixed according to comments.

But this approach kind of assumes that no one uses matrix-synapse-unwrapped separately (I believe so), since if an optional dependency gets marked as broken later, the build might still pass, and a feature that was working perfectly fine in the previous build could just stop working silently in the new build.

@xinyangli
Copy link
Contributor Author

Not sure why the ci failed?

@wegank wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Jan 4, 2025
@Mic92 Mic92 merged commit 1d862bb into NixOS:master Jan 7, 2025
22 of 24 checks passed
@Mic92
Copy link
Member

Mic92 commented Jan 7, 2025

Still get tests errors in matrix-synapse itself though:

[web01] matrix-synapse> ===============================================================================
[web01] matrix-synapse> [ERROR]
[web01] matrix-synapse> Traceback (most recent call last):
[web01] matrix-synapse>   File "/nix/store/c9m6yd8fg1flz2j5r4bif1ib5j20a0cy-python3-3.12.8/lib/python3.12/unittest/mock.py", line 1861, in _inner
[web01] matrix-synapse>     return f(*args, **kw)
[web01] matrix-synapse>   File "/build/source/tests/http/test_proxyagent.py", line 869, in test_proxy_with_http_scheme
[web01] matrix-synapse>     self.assertEqual(proxy_ep._hostStr, "proxy.com")
[web01] matrix-synapse> builtins.AttributeError: 'HostnameEndpoint' object has no attribute '_hostStr'
[web01] matrix-synapse> tests.http.test_proxyagent.MatrixFederationAgentTests.test_proxy_with_http_scheme
[web01] matrix-synapse> ===============================================================================
[web01] matrix-synapse> [ERROR]
[web01] matrix-synapse> Traceback (most recent call last):
[web01] matrix-synapse>   File "/nix/store/c9m6yd8fg1flz2j5r4bif1ib5j20a0cy-python3-3.12.8/lib/python3.12/unittest/mock.py", line 1861, in _inner
[web01] matrix-synapse>     return f(*args, **kw)
[web01] matrix-synapse>   File "/build/source/tests/http/test_proxyagent.py", line 857, in test_proxy_with_no_scheme
[web01] matrix-synapse>     self.assertEqual(proxy_ep._hostStr, "proxy.com")
[web01] matrix-synapse> builtins.AttributeError: 'HostnameEndpoint' object has no attribute '_hostStr'
[web01] matrix-synapse> tests.http.test_proxyagent.MatrixFederationAgentTests.test_proxy_with_no_scheme
[web01] matrix-synapse> ===============================================================================
[web01] matrix-synapse> [ERROR]
[web01] matrix-synapse> Traceback (most recent call last):
[web01] matrix-synapse>   File "/nix/store/c9m6yd8fg1flz2j5r4bif1ib5j20a0cy-python3-3.12.8/lib/python3.12/unittest/mock.py", line 1861, in _inner
[web01] matrix-synapse>     return f(*args, **kw)
[web01] matrix-synapse>   File "/build/source/tests/http/test_proxyagent.py", line 876, in test_p
[web01] roxy_with_https_scheme
[web01] matrix-synapse>     self.assertEqual(proxy_ep._wrappedEndpoint._hostStr, "proxy.com")
[web01] matrix-synapse> builtins.AttributeError: 'HostnameEndpoint' object has no attribute '_hostStr'
[web01] matrix-synapse> tests.http.test_proxyagent.MatrixFederationAgentTests.test_proxy_with_https_scheme
[web01] matrix-synapse> ===============================================================================
[web01] matrix-synapse> [ERROR]
[web01] matrix-synapse> Traceback (most recent call last):
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/logger/_observer.py", line 81, in __call__
[web01] matrix-synapse>     observer(event)
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/logger/_legacy.py", line 90, in __call__
[web01] matrix-synapse>     self.legacyObserver(event)
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/trial/_dist/workertrial.py", line 44, in emit
[web01] matrix-synapse>     self.protocol.callRemote(managercommands.TestWrite, out=text)
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/protocols/amp.py", line 951, in callRemote
[web01] matrix-synapse>     return co._doCommand(self)
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/protocols/amp.py", line 1973, in _doCommand
[web01] matrix-synapse>     d = proto._sendBoxCommand(
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/protocols/amp.py", line 884, in _sendBoxCommand
[web01] matrix-synapse>     box._sendTo(self.boxSender)
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/protocols/amp.py", line 713, in _sendTo
[web01] matrix-synapse>     proto.sendBox(self)
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/protocols/amp.py", line 2358, in sendBox
[web01] matrix-synapse>     self.transport.write(box.serialize())
[web01] matrix-synapse>   File "/nix/store/60yi01wql4dnn1fwzk6ivp9d4h4yqy4y-python3.12-twisted-24.11.0/lib/python3.12/site-packages/twisted/protocols/amp.py", line 692, in serialize
[web01] matrix-synapse>     raise TooLong(False, True, v, k)
[web01] matrix-synapse> twisted.protocols.amp.TooLong:
[web01] matrix-synapse> tests.storage.databases.main.test_events_worker.DatabaseOutageTestCase.test_recovery

@Mic92
Copy link
Member

Mic92 commented Jan 7, 2025

Fix #371815

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: python 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 12.approvals: 1 This PR was reviewed and approved by one reputable person
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants