Merge remote-tracking branch 'origin/staging-next' into staging

.github/CODEOWNERS

@@ -27,6 +27,12 @@
 /lib/asserts.nix            @infinisil @Profpatsch
 /lib/path.*                 @infinisil @fricklerhandwerk
 /lib/fileset                @infinisil
+## Libraries / Module system
+/lib/modules.nix            @infinisil @roberth
+/lib/types.nix              @infinisil @roberth
+/lib/options.nix            @infinisil @roberth
+/lib/tests/modules.sh       @infinisil @roberth
+/lib/tests/modules          @infinisil @roberth
 
 # Nixpkgs Internals
 /default.nix                                     @Ericson2314

doc/languages-frameworks/nim.section.md

@@ -1,74 +1,38 @@
 # Nim {#nim}
 
-## Overview {#nim-overview}
-
-The Nim compiler, a builder function, and some packaged libraries are available
-in Nixpkgs. Until now each compiler release has been effectively backwards
-compatible so only the latest version is available.
-
-## Nim program packages in Nixpkgs {#nim-program-packages-in-nixpkgs}
-
-Nim programs can be built using `nimPackages.buildNimPackage`. In the
-case of packages not containing exported library code the attribute
-`nimBinOnly` should be set to `true`.
+The Nim compiler and a builder function is available.
+Nim programs are built using `buildNimPackage` and a lockfile containing Nim dependencies.
 
 The following example shows a Nim program that depends only on Nim libraries:
-
 ```nix
-{ lib, nimPackages, fetchFromGitHub }:
+{ lib, buildNimPackage, fetchFromGitHub }:
 
-nimPackages.buildNimPackage (finalAttrs: {
+buildNimPackage { } (finalAttrs: {
   pname = "ttop";
-  version = "1.0.1";
-  nimBinOnly = true;
+  version = "1.2.7";
 
   src = fetchFromGitHub {
     owner = "inv2004";
     repo = "ttop";
     rev = "v${finalAttrs.version}";
-    hash = "sha256-x4Uczksh6p3XX/IMrOFtBxIleVHdAPX9e8n32VAUTC4=";
+    hash = "sha256-oPdaUqh6eN1X5kAYVvevOndkB/xnQng9QVLX9bu5P5E=";
   };
 
-  buildInputs = with nimPackages; [ asciigraph illwill parsetoml zippy ];
+  lockFile = ./lock.json;
 
-})
-```
-
-## Nim library packages in Nixpkgs {#nim-library-packages-in-nixpkgs}
-
-
-Nim libraries can also be built using `nimPackages.buildNimPackage`, but
-often the product of a fetcher is sufficient to satisfy a dependency.
-The `fetchgit`, `fetchFromGitHub`, and `fetchNimble` functions yield an
-output that can be discovered during the `configurePhase` of `buildNimPackage`.
-
-Nim library packages are listed in
-[pkgs/top-level/nim-packages.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/nim-packages.nix) and implemented at
-[pkgs/development/nim-packages](https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/nim-packages).
-
-The following example shows a Nim library that propagates a dependency on a
-non-Nim package:
-```nix
-{ lib, buildNimPackage, fetchNimble, SDL2 }:
-
-buildNimPackage (finalAttrs: {
-  pname = "sdl2";
-  version = "2.0.4";
-  src = fetchNimble {
-    inherit (finalAttrs) pname version;
-    hash = "sha256-Vtcj8goI4zZPQs2TbFoBFlcR5UqDtOldaXSH/+/xULk=";
-  };
-  propagatedBuildInputs = [ SDL2 ];
+  nimFlags = [
+    "-d:NimblePkgVersion=${finalAttrs.version}"
+  ];
 })
 ```
 
 ## `buildNimPackage` parameters {#buildnimpackage-parameters}
 
-All parameters from `stdenv.mkDerivation` function are still supported. The
-following are specific to `buildNimPackage`:
+The `buildNimPackage` function takes an attrset of parameters that are passed on to `stdenv.mkDerivation`.
 
-* `nimBinOnly ? false`: If `true` then build only the programs listed in
-  the Nimble file in the packages sources.
+The following parameters are specific to `buildNimPackage`:
+
+* `lockFile`: JSON formatted lockfile.
 * `nimbleFile`: Specify the Nimble file location of the package being built
   rather than discover the file at build-time.
 * `nimRelease ? true`: Build the package in *release* mode.
@@ -77,6 +41,72 @@ following are specific to `buildNimPackage`:
   Use this to specify defines with arguments in the form of `-d:${name}=${value}`.
 * `nimDoc` ? false`: Build and install HTML documentation.
 
-* `buildInputs` ? []: The packages listed here will be searched for `*.nimble`
-  files which are used to populate the Nim library path. Otherwise the standard
-  behavior is in effect.
+## Lockfiles {#nim-lockfiles}
+Nim lockfiles are created with the `nim_lk` utility.
+Run `nim_lk` with the source directory as an argument and it will print a lockfile to stdout.
+```sh
+$ cd nixpkgs
+$ nix build -f . ttop.src
+$ nix run -f . nim_lk ./result | jq --sort-keys > pkgs/by-name/tt/ttop/lock.json
+```
+
+## Lockfile dependency overrides {#nimoverrides}
+
+The `buildNimPackage` function matches the libraries specified by `lockFile` to attrset of override functions that are then applied to the package derivation.
+The default overrides are maintained as the top-level `nimOverrides` attrset at `pkgs/top-level/nim-overrides.nix`.
+
+For example, to propagate a dependency on SDL2 for lockfiles that select the Nim `sdl2` library, an overlay is added to the set in the `nim-overrides.nix` file:
+```nix
+{ lib
+/* … */
+, SDL2
+/* … */
+}:
+
+{
+  /* … */
+  sdl2 =
+    lockAttrs:
+    finalAttrs:
+    { buildInputs ? [ ], ... }:
+    {
+      buildInputs = buildInputs ++ [ SDL2 ];
+    };
+  /* … */
+}
+```
+
+The annotations in the `nim-overrides.nix` set are functions that take three arguments and return a new attrset to be overlayed on the package being built.
+- lockAttrs: the attrset for this library from within a lockfile. This can be used to implement library version constraints, such as marking libraries as broken or insecure.
+- finalAttrs: the final attrset passed by `buildNimPackage` to `stdenv.mkDerivation`.
+- prevAttrs: the attrset produced by initial arguments to `buildNimPackage` and any preceding lockfile overlays.
+
+### Overriding an Nim library override {#nimoverrides-overrides}
+
+The `nimOverrides` attrset makes it possible to modify overrides in a few different ways.
+
+Override a package internal to its definition:
+```nix
+{ lib, buildNimPackage, nimOverrides, libressl }:
+
+let
+  buildNimPackage' = buildNimPackage.override {
+    nimOverrides = nimOverrides.override { openssl = libressl; };
+  };
+in buildNimPackage' (finalAttrs: {
+  pname = "foo";
+  # …
+})
+
+```
+
+Override a package externally:
+```nix
+{ pkgs }: {
+  foo = pkgs.foo.override {
+    buildNimPackage = pkgs.buildNimPackage.override {
+      nimOverrides = pkgs.nimOverrides.override { openssl = libressl; };
+    };
+  };
+}
+```

nixos/doc/manual/release-notes/rl-2311.section.md

@@ -665,7 +665,8 @@ The module update takes care of the new config syntax and the data itself (user
   designed to be easy and safe to use.
 
   This aims to be a replacement for `lib.sources`-based filtering.
-  To learn more about it, see [the tutorial](https://nix.dev/tutorials/file-sets).
+  To learn more about it, see [the blog post](https://www.tweag.io/blog/2023-11-28-file-sets/)
+  or [the tutorial](https://nix.dev/tutorials/file-sets).
 
 - [`lib.gvariant`](https://nixos.org/manual/nixpkgs/unstable#sec-functions-library-gvariant):
   A partial and basic implementation of GVariant formatted strings.

nixos/doc/manual/release-notes/rl-2405.section.md

@@ -14,7 +14,7 @@ In addition to numerous new and upgraded packages, this release has the followin
 
 <!-- To avoid merge conflicts, consider adding your item at an arbitrary place in the list instead. -->
 
-- Create the first release note entry in this section!
+- [maubot](https://github.com/maubot/maubot), a plugin-based Matrix bot framework. Available as [services.maubot](#opt-services.maubot.enable).
 
 ## Backward Incompatibilities {#sec-release-24.05-incompatibilities}
 
@@ -26,4 +26,6 @@ In addition to numerous new and upgraded packages, this release has the followin
 
 <!-- To avoid merge conflicts, consider adding your item at an arbitrary place in the list instead. -->
 
-- Create the first release note entry in this section!
+- Programs written in [Nim](https://nim-lang.org/) are built with libraries selected by lockfiles.
+  The `nimPackages` and `nim2Packages` sets have been removed.
+  See https://nixos.org/manual/nixpkgs/unstable#nim for more information.

nixos/modules/installer/tools/tools.nix

@@ -130,7 +130,7 @@ in
     '';
   };
 
-  config = lib.mkIf (config.nix.enable && !config.system.disableInstallerTools) {
+  config = lib.mkMerge [ (lib.mkIf (config.nix.enable && !config.system.disableInstallerTools) {
 
     system.nixos-generate-config.configuration = mkDefault ''
       # Edit this configuration file to define what should be installed on
@@ -257,10 +257,13 @@ in
 
     documentation.man.man-db.skipPackages = [ nixos-version ];
 
+  })
+
+  # These may be used in auxiliary scripts (ie not part of toplevel), so they are defined unconditionally.
+  ({
     system.build = {
       inherit nixos-install nixos-generate-config nixos-option nixos-rebuild nixos-enter;
     };
-
-  };
+  })];
 
 }

nixos/modules/module-list.nix

@@ -621,6 +621,7 @@
   ./services/matrix/appservice-irc.nix
   ./services/matrix/conduit.nix
   ./services/matrix/dendrite.nix
+  ./services/matrix/maubot.nix
   ./services/matrix/mautrix-facebook.nix
   ./services/matrix/mautrix-telegram.nix
   ./services/matrix/mautrix-whatsapp.nix

nixos/modules/profiles/macos-builder.nix

@@ -103,6 +103,19 @@ in
     # server that QEMU provides (normally 10.0.2.3)
     networking.nameservers = [ "8.8.8.8" ];
 
+    # The linux builder is a lightweight VM for remote building; not evaluation.
+    nix.channel.enable = false;
+    # remote builder uses `nix-daemon` (ssh-ng:) or `nix-store --serve` (ssh:)
+    # --force: do not complain when missing
+    # TODO: install a store-only nix
+    #       https://github.com/NixOS/rfcs/blob/master/rfcs/0134-nix-store-layer.md#detailed-design
+    environment.extraSetup = ''
+      rm --force $out/bin/{nix-instantiate,nix-build,nix-shell,nix-prefetch*,nix}
+    '';
+    # Deployment is by image.
+    # TODO system.switch.enable = false;?
+    system.disableInstallerTools = true;
+
     nix.settings = {
       auto-optimise-store = true;
 

nixos/modules/services/databases/redis.nix

@@ -393,9 +393,7 @@ in {
         ProtectKernelModules = true;
         ProtectKernelTunables = true;
         ProtectControlGroups = true;
-        RestrictAddressFamilies =
-          optionals (conf.port != 0) ["AF_INET" "AF_INET6"] ++
-          optional (conf.unixSocket != null) "AF_UNIX";
+        RestrictAddressFamilies = [ "AF_INET" "AF_INET6" "AF_UNIX" ];
         RestrictNamespaces = true;
         LockPersonality = true;
         MemoryDenyWriteExecute = true;

nixos/modules/services/matrix/maubot.md

@@ -0,0 +1,103 @@
+# Maubot {#module-services-maubot}
+
+[Maubot](https://github.com/maubot/maubot) is a plugin-based bot
+framework for Matrix.
+
+## Configuration {#module-services-maubot-configuration}
+
+1. Set [](#opt-services.maubot.enable) to `true`. The service will use
+   SQLite by default.
+2. If you want to use PostgreSQL instead of SQLite, do this:
+
+   ```nix
+   services.maubot.settings.database = "postgresql://maubot@localhost/maubot";
+   ```
+
+   If the PostgreSQL connection requires a password, you will have to
+   add it later on step 8.
+3. If you plan to expose your Maubot interface to the web, do something
+   like this:
+   ```nix
+   services.nginx.virtualHosts."matrix.example.org".locations = {
+     "/_matrix/maubot/" = {
+       proxyPass = "http://127.0.0.1:${toString config.services.maubot.settings.server.port}";
+       proxyWebsockets = true;
+     };
+   };
+   services.maubot.settings.server.public_url = "matrix.example.org";
+   # do the following only if you want to use something other than /_matrix/maubot...
+   services.maubot.settings.server.ui_base_path = "/another/base/path";
+   ```
+4. Optionally, set `services.maubot.pythonPackages` to a list of python3
+   packages to make available for Maubot plugins.
+5. Optionally, set `services.maubot.plugins` to a list of Maubot
+   plugins (full list available at https://plugins.maubot.xyz/):
+   ```nix
+   services.maubot.plugins = with config.services.maubot.package.plugins; [
+     reactbot
+     # This will only change the default config! After you create a
+     # plugin instance, the default config will be copied into that
+     # instance's config in Maubot's database, and further base config
+     # changes won't affect the running plugin.
+     (rss.override {
+       base_config = {
+         update_interval = 60;
+         max_backoff = 7200;
+         spam_sleep = 2;
+         command_prefix = "rss";
+         admins = [ "@chayleaf:pavluk.org" ];
+       };
+     })
+   ];
+   # ...or...
+   services.maubot.plugins = config.services.maubot.package.plugins.allOfficialPlugins;
+   # ...or...
+   services.maubot.plugins = config.services.maubot.package.plugins.allPlugins;
+   # ...or...
+   services.maubot.plugins = with config.services.maubot.package.plugins; [
+     (weather.override {
+       # you can pass base_config as a string
+       base_config = ''
+         default_location: New York
+         default_units: M
+         default_language:
+         show_link: true
+         show_image: false
+       '';
+     })
+   ];
+   ```
+6. Start Maubot at least once before doing the following steps (it's
+   necessary to generate the initial config).
+7. If your PostgreSQL connection requires a password, add
+   `database: postgresql://user:password@localhost/maubot`
+   to `/var/lib/maubot/config.yaml`. This overrides the Nix-provided
+   config. Even then, don't remove the `database` line from Nix config
+   so the module knows you use PostgreSQL!
+8. To create a user account for logging into Maubot web UI and
+   configuring it, generate a password using the shell command
+   `mkpasswd -R 12 -m bcrypt`, and edit `/var/lib/maubot/config.yaml`
+   with the following:
+
+   ```yaml
+   admins:
+       admin_username: $2b$12$g.oIStUeUCvI58ebYoVMtO/vb9QZJo81PsmVOomHiNCFbh0dJpZVa
+   ```
+
+   Where `admin_username` is your username, and `$2b...` is the bcrypted
+   password.
+9. Optional: if you want to be able to register new users with the
+   Maubot CLI (`mbc`), and your homeserver is private, add your
+   homeserver's registration key to `/var/lib/maubot/config.yaml`:
+
+   ```yaml
+   homeservers:
+       matrix.example.org:
+           url: https://matrix.example.org
+           secret: your-very-secret-key
+   ```
+10. Restart Maubot after editing `/var/lib/maubot/config.yaml`,and
+    Maubot will be available at
+    `https://matrix.example.org/_matrix/maubot`. If you want to use the
+    `mbc` CLI, it's available using the `maubot` package (`nix-shell -p
+    maubot`).