taxy/taxy-magit-section split

[tarsius]

Now that taxy-magit-section has been moved out of "taxy itself", shouldn't the examples that depend on taxy-magit-section be moved to the same place?

Speaking of "that place", please consider not using a branch for that but a separate repository. Keeping it in a separate branch complicates things for users and for the Emacsmirror (I would have to temporary patch the import tool, which is doable but also indicates that yours is the only package that makes this necessary), while also having no benefit (that I could see).

[alphapapa]

Now that taxy-magit-section has been moved out of "taxy itself", shouldn't the examples that depend on taxy-magit-section be moved to the same place?

I don't know. On one hand, yes, but on the other, they show how taxy can be usefully applied, which I think is important, because it can seem somewhat theoretical without showing practical application. I'd probably prefer to keep the examples in one place, and refer to them in multiple places. But I'll think about it some more.

Speaking of "that place", please consider not using a branch for that but a separate repository. Keeping it in a separate branch complicates things for users and for the Emacsmirror (I would have to temporary patch the import tool, which is doable but also indicates that yours is the only package that makes this necessary), while also having no benefit (that I could see).

The benefit for me is keeping the code in a single repository, which is helpful since the two libraries are tied together. I'd prefer to keep them in the same branch, but Stefan asked me to separate them for GNU ELPA's sake. Now you're asking me to separate them even further for Emacsmirror's sake.

I don't want to create more work for either of you, but I don't understand why this is so hard. AFAICT, magit-section resides in magit.git, and it's in nonGNU ELPA. I'd prefer to do the same with taxy-magit-section and taxy.git, which is in GNU ELPA. Am I missing something?

[tarsius]

I'd probably prefer to keep the examples in one place, and refer to them in multiple places

It leads to a circular dependency [when using the borg package manager (or similar)] because taxy (or rather the bundled dependencies) depend(s) on taxy-section, which depends on taxy... But that's not a problem, at least not for borg, which does not automatically resolve dependencies and while epkg displays dependency trees, it does not expand it eagerly. (If I ever teach borg to automatically install dependencies, then I will have to resolve circular dependencies anyway, because other packages have similar loops, just usually not such tight ones.)

Because borg installs a package by cloning its repository and then compiling and adding the relevant directories to the load-path, it cannot install parts of a repository separately. On Melpa a repository/package foo is usually split up into multiple packages when some of the contained files have additional dependencies, such as helm-foo that many users won't want or need because they don't actually use helm. While borg cannot split up packages like this, it allows the user to declare which libraries to not compile. IMO both approaches are elegant but have different tradeoffs.

I cannot say the same for the approach taken by [non]GNU Elpa. The immediate problem here is that it maps a repository directly to a package (like for borg and the Emacsmirror for that matter) but at the same time it does not want to bundle optional extensions like Melpa doesn't want to either. As a consequence it has to track the repository twice -- two identical branches that are checked out in two identical but separate worktrees. This is the approach I took with magit/magit-section and what I wish Stefan would suggest in such situations.

Its understandable that Stefan would like to avoid the duplicated branches/worktrees and one suggestion I made was that one of the worktrees could be replaced by a symlink. That wouldn't make the duplicated branch unnecessary but at least its a step in the right direction. Stefan found that suggestion interesting but of course it complicates the tooling because now something/someone has to decide whether to checkout or link.

I would go one step further and add an additional package property for use in elpa-packages: :use-worktree/branch-of-this-other-package. Then neither an additional worktree nor branch would be needed. I would expect that to be fairly easy to implement.

I'd prefer to keep them in the same branch, but Stefan asked me to separate them for GNU ELPA's sake.

I fully agree and my suggestion would be to revert the split (going as far as to rewind history even) and instead do what I did for magit/magit-section. You can point him to this conversation when you do. He might not be thrilled about it but when I did it, he was okay with it. IMO it is not right for him to ask package maintainers to split up repositories like this. The inconvenience of having duplicated branches/worktrees is very minor and only a handful of people who actually check out [non]GNU Elpa and more or less all the implied worktrees are affected by it, while the inconvenience of having to split up the upstream repositories is a rather large one, which, when looking at the number of such >1package/repositories in Melpa, potentially affects a rather larger number of packages and their authors.

I plan to write to Stefan/emacs-devel about this at a later time and maybe even do a blog post comparing the strategies used by [non]GNU Elpa / Melpa / Emacsmirror / borg / straight and the motivations behind it and the drawbacks. Doing that is a larger undertaking so for now I have only written this shorter summary, which may gloss over things that appear obvious to me but actually are not. I.e. you're my guinea pig to test how much I can leave out for my argument to still make sense. ;D

[tarsius]

Just a quick FYI; I've added taxy-magit-section to the Emacsmirror as it is currently distributed.

[alphapapa]

I'd probably prefer to keep the examples in one place, and refer to them in multiple places

It leads to a circular dependency [when using the borg package manager (or similar)] because taxy (or rather the bundled dependencies) depend(s) on taxy-section, which depends on taxy... But that's not a problem, at least not for borg, which does not automatically resolve dependencies and while epkg displays dependency trees, it does not expand it eagerly. (If I ever teach borg to automatically install dependencies, then I will have to resolve circular dependencies anyway, because other packages have similar loops, just usually not such tight ones.)

I'm not sure what you mean. taxy doesn't depend on taxy-magit-section. Some of the Elisp files in examples/ do require it, but those are only intended to be example applications. I've left some of them in the taxy package recipe by default, because they could be useful, but if they're causing a problem, they could be excluded.

Because borg installs a package by cloning its repository and then compiling and adding the relevant directories to the load-path, it cannot install parts of a repository separately. On Melpa a repository/package foo is usually split up into multiple packages when some of the contained files have additional dependencies, such as helm-foo that many users won't want or need because they don't actually use helm. While borg cannot split up packages like this, it allows the user to declare which libraries to not compile. IMO both approaches are elegant but have different tradeoffs.

Yes, e.g. in org-ql the helm-org-ql.el file is excluded from the org-ql package recipe, and it has its own MELPA recipe for the package helm-org-ql.

I didn't know that borg doesn't handle that case by default.

I think this is an example of why it would be beneficial for packages to declare their recipes in their own repos rather than in MELPA's repo. That has some potential downsides, of course, but the alternative is that other package managers end up having to clone MELPA's repo just to get the recipes, which seems a bit silly.

I cannot say the same for the approach taken by [non]GNU Elpa. The immediate problem here is that it maps a repository directly to a package (like for borg and the Emacsmirror for that matter) but at the same time it does not want to bundle optional extensions like Melpa doesn't want to either. As a consequence it has to track the repository twice -- two identical branches that are checked out in two identical but separate worktrees. This is the approach I took with magit/magit-section and what I wish Stefan would suggest in such situations.

Yes, a 1:1 mapping between repo:package doesn't seem like the best solution.

Its understandable that Stefan would like to avoid the duplicated branches/worktrees and one suggestion I made was that one of the worktrees could be replaced by a symlink. That wouldn't make the duplicated branch unnecessary but at least its a step in the right direction. Stefan found that suggestion interesting but of course it complicates the tooling because now something/someone has to decide whether to checkout or link.

That sounds like a good solution.

I would go one step further and add an additional package property for use in elpa-packages: :use-worktree/branch-of-this-other-package. Then neither an additional worktree nor branch would be needed. I would expect that to be fairly easy to implement.

Sounds nice to me.

I'd prefer to keep them in the same branch, but Stefan asked me to separate them for GNU ELPA's sake.

I fully agree and my suggestion would be to revert the split (going as far as to rewind history even) and instead do what I did for magit/magit-section. You can point him to this conversation when you do. He might not be thrilled about it but when I did it, he was okay with it. IMO it is not right for him to ask package maintainers to split up repositories like this. The inconvenience of having duplicated branches/worktrees is very minor and only a handful of people who actually check out [non]GNU Elpa and more or less all the implied worktrees are affected by it, while the inconvenience of having to split up the upstream repositories is a rather large one, which, when looking at the number of such >1package/repositories in Melpa, potentially affects a rather larger number of packages and their authors.

I tend to agree with you. While taxy and taxy-magit-section have settled a bit, I still consider them closely tied, and changes in one may require changes in the other. I'd much prefer for the two files to live side-by-side in the same repo, branch, and worktree. It would make my work easier, and in general, making the developer's work easier seems like it should be a higher priority than simplifying an automated build process.

I'm a bit tired of having to argue these things, in general, so I'm hesitant to start another thread on emacs-devel about it. But the more I think about having taxy-magit-section.el in a separate branch, the less enthusiastic about it I feel. So maybe I will.

I plan to write to Stefan/emacs-devel about this at a later time and maybe even do a blog post comparing the strategies used by [non]GNU Elpa / Melpa / Emacsmirror / borg / straight and the motivations behind it and the drawbacks. Doing that is a larger undertaking so for now I have only written this shorter summary, which may gloss over things that appear obvious to me but actually are not. I.e. you're my guinea pig to test how much I can leave out for my argument to still make sense. ;D

I don't mind too much being your guinea pig. :) I did find this explanation a bit hard to follow, but I think if you said the same things but organized differently, it would be easier.

[alphapapa]

Okay, I started a thread and requested to move taxy-magit-section.el back into this branch: https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00722.html

[monnier]

The benefit for me is keeping the code in a single repository, which is helpful since the two libraries are tied together. I'd prefer to keep them in the same branch, but Stefan asked me to separate them for GNU ELPA's sake. Now you're asking me to separate them even further for Emacsmirror's sake.

You can keep them in a single repository on your machine, while storing them in two different remote repositories.

[alphapapa]

The benefit for me is keeping the code in a single repository, which is helpful since the two libraries are tied together. I'd prefer to keep them in the same branch, but Stefan asked me to separate them for GNU ELPA's sake. Now you're asking me to separate them even further for Emacsmirror's sake.

You can keep them in a single repository on your machine, while storing them in two different remote repositories.

Do you mean to push the same repo and branch to two different remotes, identical but for URL?

[monnier]

git push usually only pushes a single branch, not a whole repository (contrary to git fetch which usually fetches a whole repository). So I mean you'd do:

% git clone https://github.com/alphapapa/taxy
% cd taxy
% git remote add -f taxy-magit-section https://github.com/alphapapa/taxy-magit-section
% git branch tms taxy-magit-section/master

and then you have locally a single repository that contains taxy in master and taxy-magit-section in the tms branch. Whenever you git pull or git push it will automatically go to the appropriate remote repository, based on which branch you have currently in your workspace.

[alphapapa]

Yes, I'm familiar with git push, and I could do that, but why? That still leaves me with the files taxy.el and taxy-magit-section.el in separate branches, and I still have to view them in separate worktrees, and I still have to push each branch separately. And that would leave me with an extra remote repository to keep track of, one which wouldn't correspond to a separate local one, which would be more confusing for me.

I really just want to keep these files in the same repo, branch, and worktree, and distribute them as separate packages on ELPA, as I do on MELPA for some other packages (like org-ql and helm-org-ql, bufler and helm-bufler, etc), because they are too closely related for separating their source files to make sense to me.

[monnier]

I was just pointing out that a separate repository (as Jonas suggested) wouldn't make things much worse to you compared to a separate branch.

[alphapapa]

Not much worse, no. :) Is there some reason you would prefer that for ELPA's sake?

[monnier]

Not much worse, no. :) Is there some reason you would prefer that for ELPA's sake?

[ I assume by "ELPA" you mean "GNU ELPA". ]
No, for GNU ELPA, separate branches or separate repositories don't make much difference. It's good either way.

[akirak]

FYI: Straight.el can't install taxy-magit-section right now, because it cannot check out multiple branches of the same repository. It is possible for users to work around this issue (e.g. by creating a mirror repository with a different name), but it would be better to either create a separate repository or put the two packages in the same branch. It seems that straight.el has a separate mirror repository for taxy-magit-section.

(I am personally fine with this, because I am currently transitioning from straight.el to my own package manager that can perfectly handle this situation. I can use your package anyway, and the issue gives me an advantage over straight.el. However, it wouldn't be good for many other people.)

[tarsius]

I am currently transitioning [...] to my own package manager

Welcome to the club. 😀

[akirak]

Welcome to the club.

I didn't even know it was a club.

[monnier]

FYI: Straight.el can't install `taxy-magit-section` right now, because it cannot check out multiple branches of the same repository. It is possible for users to work around this issue (e.g. by creating a mirror repository with a different name), but it would be better to either create a separate repository or put the two packages in the same branch.

It seems even better would be to let `straight.el` handle that case. I can't think of a good reason why that should be difficult. Stefan

[akirak]

It seems even better would be to let straight.el handle that case.

@monnier I guess the author chose to keep it simple stupid. Mirror repositories are a good compromise that solves many problems (e.g. non-git sources), with an extra maintenance effort (though it can be automated).

[chasecaleb]

I didn't even know it was a club.

Indeed it is. You can even go one level deeper and build an Emacs package manager on top of the one true package manager (:rofl:), Nix: https://gitlab.com/chasecaleb/emacs.nix/-/tree/master/emacs/nix. Fortunately my package manager can handle two different branches of the same repo so this isn't a big deal to me 😄, but for what it's worth this was quite the WTF to figure out while tracking down dependencies for ement. It definitely violates "principle of least astonishment", but I get that all of the different tool concerns are complicated. Just curious, any plans on changing things vs keeping it the way they are? (I imagine the latter, since this is a year old issue.)

P.S. Thanks for ement @alphapapa, I'm liking it so far now that I sorted out these dependencies 👍

[akirak]

@chasecaleb I know Nix and am using Nix. There are emacs-overlay and nix-emacs-ci (to which I have contributed trivial workarounds several times) where Nix is used to build Emacs. I use Nix to build Emacs Lisp packages but not to build Emacs executables for myself, because the two repositories are great. I personally don't think building with Nix is deeper, though. For me, it works like a shortcut.

[chasecaleb]

@akirak I've seen you around a few other repos beyond here (nixpkgs perhaps)... small world! Absolutely agree with your sentiment about Nix making life easier; I'm familiar with emacs-overlay and it's great (FYI for anyone else reading this: emacs-overlay also provides Emacs package builds to make life easy). I haven't seen your emacs-twist before though, I'll have to check that out on a rainy day.

I reinvented the wheel with building Emacs packages in Nix because I'm overly opinionated and missed how easy Borg's workflow made it to contribute to packages and switch to forks, so my design was heavily influenced by Borg's workflow (thanks, @tarsius). Plus Emacs is so important to me (it's my window manager even) that I wanted direct control over package versions and builds.

[akirak]

Borg's workflow made it to contribute to packages and switch to forks, so my design was heavily influenced by Borg's workflow (thanks, @tarsius).

Interesting. Actually, Borg was one of my initial motivations for working on my project. I read several reddit threads on Emacs Lisp package security and thought it would be a good idea to build a collective repository with Nix. Unfortunately, I wasn't sure when the submodules support of Nix flakes was going to be shipped, so I ended up building the non-submodule solution.

I've seen you around a few other repos beyond here (nixpkgs perhaps)... small world!

Thanks, indeed, I see the same people in the community quite often. I am not a heavy contributor of Nixpkgs, so you may have seen me on reddit.

[tarsius]

I reinvented the wheel with building Emacs packages in Nix because I'm overly opinionated and missed how easy Borg's workflow made it to contribute to packages and switch to forks, so my design was heavily influenced by Borg's workflow (thanks, @tarsius). Plus Emacs is so important to me (it's my window manager even) that I wanted direct control over package versions and builds.

I've started using Guix but for now I keep using Borg for Emacs packages. I do however plan to experiment with Guix and Elisp packaging eventually. I am vaguely aware of the emacs-overlay for Nix, but if you have any useful links that I could use for inspiration, that would be great!

[akirak]

@tarsius emacs-overlay provides a fresh set of Emacs Lisp packages from Nixpkgs, which are originally defined here:
https://github.com/NixOS/nixpkgs/tree/master/pkgs/applications/editors/emacs/elisp-packages

Besides the standard one, there are some non-standard solutions for Emacs Lisp package management with Nix:

• https://github.com/nix-community/nix-straight.el
• https://github.com/emacs-twist/twist.nix (my project)
• https://github.com/positron-solutions/yakspkgs

@positron-solutions works on some other projects related to Nix + Emacs. He seems to be focused on documenting requirements extensively but reuses existing solutions as much as possible.

On the contrary, I am interested in prototyping a solution that precisely meet my needs. I don't intend to draw public attention more than I can handle (because it's not a funded project at present). I also don't want to depend on third-party frameworks, except for Nix libraries that I consider dependable.

[jian-lin]

Recently, I consider treating native compilation errors of Emacs lisp packages more strictly in NixOS, which is of benefit to quality of Emacs lisp packages in general.

Currently, there are errors about missing taxy-magit-section when compiling taxy examples. It is a bit tricky to handle because making taxy depend on taxy-magit-section generates a circular dependency.

I propose two workarounds:

• adding -*- no-byte-compile: t -*- cookie to example files
• or moving example files to taxy-magit-section and mentioning examples in taxy

[alphapapa]

@jian-lin I'd probably suggest just excluding the example libraries from the package definition for now. They don't need to be included in the installed package files.

[jian-lin]

I'd probably suggest just excluding the example libraries from the package definition for now.

Thanks, I will do that in NixOS.

BTW, NixOS has this issue because we follow the package definition of GNU ELPA. .elpaignore or :ignored-files can be used to exclude files in GNU ELPA.

[alphapapa]

I'd probably suggest just excluding the example libraries from the package definition for now.

Thanks, I will do that in NixOS.

Good, thanks.

BTW, NixOS has this issue because we follow the package definition of GNU ELPA. .elpaignore or :ignored-files can be used to exclude files in GNU ELPA.

Yes, I may do that in the future.