Use Cookiecutter for the package-template (APE 12)

[Cadair]

No description provided.

[adrn]

This week is a little crazy because of travel so I'll have to do a more thorough review in a few days, but my gut reaction is that I'm +1 to the idea. I don't know much about cookiecutter but it sounds like it allows for a much nicer solution than copy-pasting directions from the package template setup.

[adrn]

Two "such as" clauses in the same sentence

[Cadair]

reworded

[astrofrog]

repeated 'package template' - maybe reword. Somewhere in abstract you need to spell out more clearly that creating new projects will be much easier for people (ease use doesn't quite cover it). Maybe have a separate sentence for the fact it's easier for people to use, and to 'subclass'. You said it better in the detailed bit. Can you try and improve it here too?

[astrofrog]

use a link instead of double backquotes?

[astrofrog]

Another alternative is to write our own setup script. Might be worth mentioning at least.

[Cadair]

done

[dkirkby]

I used my own cookiecutter template for new python projects until I discovered and switched to the astropy affiliated template because it was much more comprehensive in its support for RTD, travis, etc, and "just works". This proposal sounds like it would significantly simply the initial setup (although I found it useful to go through the steps manually, at least the first time, so I knew what was under the hood).

How do bug fixes and maintenance updates (e.g., version bumps) to the template propagate to affiliated packages in this proposal? I guess this part of the proposal is relevant but I wasn't sure how the automatically generated template repo would solve the problem:

As implemented in Cadair/package-template users who have followed the instructions where they use the package template as a git remote will no longer be able to pull updates from the package_template git repository. This could be mitigated by automatically generating a rendered version of the package template in another git repository or in a different branch.

[Cadair]

Thanks for the feedback!

The idea is that for people who want to continue a git-remote based update workflow (where they start a project with the git history of the template) could pull form the automatically generated repo.

My thoughts on people who start with cookiecutter would be that you could render any updates to a new folder and then diff the updated template to your project and see any changes and pull them in as needed.

[adrn]

@Cadair Hmm, that sounds quite a bit more tedious than the current workflow though...

[Cadair]

@adrn really because isn't that what you are doing in a mature project at the moment? Some files have changed in upstream and you have to manually merge them?

[adrn]

@Cadair Sorry, I meant the 2nd case above -- rending updates to a new folder and diffing.

[Cadair]

Yeah I did as well, although I went a little to "git" in my terminology. If you are not tracking a git remote, you have the rendered template in a package_template folder on your disk and your project, how are you merging it and managing updates at the moment?

[bsipocz]

What about adding a working solution to astropy/package-template#197 to this APE? I agree with the others that having an easy way to update an existing package that uses the template is just as important (actually in my view it is more important) as having an easy way to start up something new.

[dkirkby]

I suspect most users don't care about inheriting the template git history (and it has burned me a few times when I forgot to delete the template repo's tags). The ideal update workflow for a new user is something like:

% ./update_template
Your template is up to date.
% ./update_template
Your template needs updating but you have modified files that must be
committed / stashed / reset first.
% ./update_template
...
The following template files have been updated:
  setup.py, travis.yml
Use "git diff" to review the changes then "git add" and "git commit" to accept them.
%

[Cadair]

While I agree that having a nice update mechanism is important, I am not sure I want to conflate that with this PR. (Ignoring the git remote case) I can't see how this make things worse and because cookiecutter will remember the settings you used to render the template the first time it is possible to get your customised version of the template for diffing, which could be scripted out later.

[dkirkby]

I agree with @bsipocz that straightforward updates are at least as important as a more convenient initial install. The main benefit of the template is that it allows developers to adopt the best practices of the astropy community (RTD, travis, etc) without necessarily understanding all of the details of how they are implemented (which was true for me when I started using the template). However, a lower barrier to adopting best practices also requires a simple and robust mechanism for keeping the relevant files up to date.

[mhvk]

Another agreement with the importance of updating: I have one project for which I use the astropy template, and I found setting it up via the current instructions not particularly onerous, but I find updating it painful (mostly because it needs to be done so infrequently that I have no memory any more about how it is to be done; indeed, I have no clue whether I'm currently up-to-date).

That said, I do like the idea of providing a more automated way of getting a useful environment.

[astrofrog]

One idea I've had for a while would be that repositories which would like to see updates could include a configuration file (e.g. .astropy_template.yml) which could indicate which files should be updated automatically, and a bot could open a PR to do this. But that's a separate issue from the initial creation.

[hamogu]

Agreements that updating is important. I currently have 4 or 5 packages that use the template, most of them are for my own personal use (although two or three might be advertised to a wider community in the future and will hopefully grow into affiliated packages).

Suggestion: Poll the astropy template users list to find how many people use git for updating vs. a more manual approach.

[mhvk]

@hamogu - thanks for mentioning the template users list -- I didn't even know it existed... (for others like me: https://groups.google.com/forum/#!forum/astropy-affiliated-maintainers)

[cdeil]

@Cadair - Thanks!

I'm +1 to introduce cookiecutter for this.

Along the lines of what several others have stated, I think it would be important to think through (or even exercise through) how updating existing packages will work. That's been the painful and time wasted part for me.

Currently you have this which is pretty vague and thin IMO:

As implemented in Cadair/package-template users who have followed the instructions where they use the package template as a git remote will no longer be able to pull updates from the package_template git repository. This could be mitigated by automatically generating a rendered version of the package template in another git repository or in a different branch.

I currently have the following thoughts / questions, which I think would be useful to address in the APE (not just the GH discussion here):

• Should Change the package-template to use Cookiecutter package-template#202 be merged? Or would it better to leave the existing repo as-is and start a new one. In the discussion you say "maybe there will be a rendered version in a second repo" ... that needs to be decided so that the dicussion can be more concrete and move on, no?
• If we merge Change the package-template to use Cookiecutter package-template#202, then the "git update" method will not work any more for anyone (existing or future repos). Too much changes and it's practically a new repo. Personally I think "git update" never really worked well, I tried it a few times and there were always nasty git merge conflicts, it made the affiliated project history and contributor lists messy ... so good riddance to "git update"! But make this more clear in the APE, please.
• Should this change be taken as an opportunity to re-think what's in the astropy-helpers and package-template files and which files in astropy-helpers and package-template must be version-coupled? I mean, is it really necessary to have so many boilerplate files in the package-template. Can't some of them be removed completely (e.g. ez_setup, can't we just require setuptools and pip which has been the standard for years) and some others moved to astropy-helpers? A lot of the complexity comes from each affiliated package having two somewhat linked, somewhat independent connected other sets of files: astropy-helpers and package-template.
• I think a little tooling (i.e. a script) to help with updating a package to new astropy-helpers or package-template versions would be awesome. It's probably tricky to maintain that script, so that it works for older and newer projects. Maybe a section in the APE discussing if it's possible would be helpful?

[bsipocz]

I think a little tooling (i.e. a script) to help with updating a package to new astropy-helpers or package-template versions would be awesome. It's probably tricky to maintain that script, so that it works for older and newer projects. Maybe a section in the APE discussing if it's possible would be helpful?

nit-picking, but the astropy-helpers update in affiliated packages (and other packages that opt in) is already working more or less like that through a script (thanks to @astrofrog). Whenever a new release is made, the person releasing the helpers can run the script that generates automated PRs to the packages as part of the release process. Obviously this could be upgraded to do by a bot or whatever, but the current way is already much better than it used to be, and puts basically no burden on the package maintainers.

With the rest I agree.

[astrofrog]

And I think we can easily adapt the astropy-helpers update script to also update e.g. ez_setup.py, _astropy_init.py, setup.py, and so on.

[bsipocz]

Not setup.py though, as many packages had it customised.

[astrofrog]

Maybe mention it will share the history up to the point where cookiecutter was added. Also maybe just a new branch in the repo rather than a separate repo?

[Cadair]

could do, not sure which one is better, I suppose a new branch does not require remote change on the local git side.

[bsipocz]

I would be +:100: for having a new branch rather than a new repo for the current template (unless the cookiecutter version goes into the new repo)

[astrofrog]

maybe add 'but this would likely require a lot of effort'

[Cadair]

explicit is better than implicit I suppose :p

[eteq]

@Cadair - I like this idea, but have a couple small bits that should be stated expressly in the APE:

1. While initially worried about the fact that this breaks the auto-update, the suggestion of having a secondary branch to support that is a good one. But one more thing that should be added: this auto-generation should be automatic e.g. via a travis deploy script that runs on package-template and automatically pushes a new rendered template to the special branch. I can help with setting that up, as I've done this before in other contexts. But it should be specified in the APE that this automatic approach will be done, because it won't be kept up to date if it isn't automatic.
2. I see in your trial implementation that there's a set of instructions build into the template repo. Right now the instructions for the template are in the astropy repo (http://docs.astropy.org/en/stable/development/affiliated-packages.html) - is your intent here to change that so that the authoritative instructions are now in the template repo again? If so, that should be specified in the APE, if not, you may want to note that the instructions will still remain in the Astropy docs.

(Also have some thoughts on implementation details, but they don't affect the APE so I will put them in astropy/package-template#202)

[Cadair]

@eteq I did mention "automatically" in the text of the APE, I have added Travis to be explicit.

As for the documentation, the move to cookiecutter means that it is possible to have the canonical documentation for using the template to be in the template repo. I defer to group opinion on if this is something we want to do. Personally I like it, you would only need one PR to update the docs and the template, easy to version the docs with the template etc.

[Cadair]

@eteq I addressed both your comments.

[astrofrog]

Looks good to me. To make things smoother, I propose that in package-template the master branch remains the one where the rendered version gets put, and we put the cookiecutter template in a cookiecutter branch, and make that the default repo branch.

[bsipocz]

I very much like the branching suggestion by @astrofrog.

[Cadair]

@astrofrog it being the default repo branch means it is the one that's displayed by default by GitHub right? If so I am fine with it.

[astrofrog]

@Cadair - yes, that's correct

[astrofrog]

@Cadair - can you update the APE accordingly? (the bit about making the new branch for the rendered version)

[eteq]

@Cadair - alright, I'm satisfied with this - thanks!

On the documentation, after a bit further investigation, I think the reason we made it a separate doc in the first place was because it enabled the fancy javascript magic of filling in the package name using a form. But now that's not needed due to cookiecutter, so indeed it does make sense to move it back into the package-template.

[Cadair]

@astrofrog @eteq I pushed the re-word.

[astrofrog]

@Cadair - very minor before I approve and merge, we should keep the -c cookiecutter since we're going to merge this into a cookiecutter branch. So can you update it here and in the first command of the code block below?

[Cadair]

I have just tested this and cookiecutter uses the default git branch rather than "master".

[Cadair]

Wooop 🎆

[embray]

Great to see this finally happen!

[cdeil]

I have a package I'd like to set up in the next days using the package template.

@Cadair - Would you recommend doing it via cookiecutter version you have made now?
(I'm a bit familiar with the process, and didn't like the old scheme with the many manual changes much, so if at all possible I'd like to use the new scheme and wouldn't mind if there's still some issues or rough edges.)