Nunjucks 2.0

[jlongster]

Hey all,

I know this project has been a little quiet lately, even after #281 where we discussed some things to do to keep it active. Bringing on more contributors has certainly helped, but there's a few things in there I failed to do yet, like merging docs into this repo.

I don't have as much time on the side as I used to since switching to the Firefox devtools team. However, this is one project that I especially want to keep alive, if anything to learn how manage a large successful open source project. This is easily the largest one I manage.

I'm going to start a new effort and call it "nunjucks 2.0". It's 2.0 only because it sounds bigger, and will keep me motivated to push it though. Also, if we want to do anything small backwards incompatible changes, we can do it. One potential idea is to turn autoescaping on by default.

So please post here any ideas you'd like to see in 2.0 or things you think we should change. 2.0 isn't meant to be a huge change, so large features could still be delayed to a later release.

[carljm]

I think #267 is a very strong candidate for something that should be done in a version where small back-compat breaks are allowed.

[popomore]

Need this #228

[mkoryak]

I would like to see a custom tag API that is easier to use, or at least with more examples and docs.

[garygreen]

I'd like to see:

• Slim build should not include WebLoader #196 - Slim build should not include WebLoader
• Consider using Browserify for building #238 - Consider using Browserify for building (I think we need to move away from the custom bash compiling script, bit of a nightmare to use on windows), maybe in combination with Gulp/Grunt.
• Separate watch from nunjucks / 100% CPU Issues #289 - Separate watch from nunjucks - I think this kind of functionality shouldn't be a part of core and can be deprecated/removed in the next major version
• Update benchmark suite using benchmark.js #285 - Update benchmark suite, focus more on performance and make it be a real selling point for the templating engine
• Block filter #254 - Block filter
• Conditional extends should work, no? #39 - Conditional extends
• +1 to easier custom tag API. Creating custom tags is far more difficult than it should be imo.

[zephraph]

I'd be interested in seeing more code level documentation. The parser module stands out on this point. One other thing that would be nice is to be able to add filters to tags other than variables.

I.E.

{% block unsafe | escape %}
User content goes here
{% endblock %}

or

{% include "content.md" | markdown %}

This essentially gives added usage to filters which are easier to implement than custom tags.

Granted, this may be useless with a filter block... which I think could be implemented with the tag API relatively simply.

[oyyd]

#285 - Update benchmark suite, focus more on performance and make it be a real selling point for the templating engine

Agree. Good performance and benchmark would attract more developers.

[usmonster]

This was already mentioned all the way at the top, but here's the reference issue for turning on autoescaping by default: #138.

[metasansana]

Would be nice to be able to run async operations within a macro and have the calling template wait until it is finished.

[imjoshholloway]

+1 on the easier to reason custom tags API. Spent most of today trying to implement a custom tag and haven't had much luck.

That said, it would be pretty cool to split ALL tags out to be extensions in their own right - In theory it should make everything far simpler and easier to debug etc

[popomore]

Many bugs that already tagged in issues should be fixed.

[popomore]

Should we create a milestone?

[airtonix]

@popomore yes definitely.

[jlongster]

I'm tidying up all of the issues and making sure they all valid. Additionally I've fixed a few of them and prepping for releasing 2.0. When 2.0 is released I want this project to be in good shape for contributors (deployment process is automated, docs are integrated here (done), etc).

We've decided to match watch default to off (see issue #369) and autoescape default to on in 2.0. In addition I'm going to extract the watcher and express support somehow to it's more modular. Lastly I'm going to fix a few of the issues mentioned above, like implementing a filter block.

#267 is another one we need to decide on. I'm worried about breaking existing templates.

The above sounds like a lot of work but I think most of them are actually pretty small bugs/enhancement, and I'd like to push out 2.0 next week or the week after.

[oyyd]

@jlongster Great!

[jlongster]

These are the issues I plan to fix before releasing 2.0: #39 #254 #196 #238 #267 in addition to some of the smaller bugs.

Also figure out some sort of upgrade path for #304. I'm thinking of possible forcing the user to pass a second argument for one version (so that we can show a warning message that the behavior is changing) and then we can eventually change the semantics of that filter.

[carljm]

If you're planning to accept #267, I can write the PR for that one.

And I think your proposal for #304 makes sense.

[jlongster]

@carljm I'm not sure about #267. I suppose I would be fine with it, I just hope it doesn't break existing templates. I suppose that nobody should be relying on [] being false though, as checking an array as boolean isn't the best anyway. Make a PR.

[carljm]

I think it's unlikely to break many existing templates, because checking an array as a boolean is only really useful if empty arrays are falsy. It's certainly possible that someone is using an {% if arr %} to see if an array var is defined at all, and that would change behavior if the var were set to an empty array.

[jlongster]

@carljm I agree, after giving it some thought I think this is a good change. We shouldn't tie ourselves down too much to how things currently work, and there's an easy upgrade path, so it's fine.

[jlongster]

There are a ton of changes on master: watch is off by default (haven't turned autoescaping on by default yet), a new replace filter, a throwOnUndefined option which will error if an undefined/null value is displayed, a few scoping issues fixed between set & import, webpack builds for the browser, and bunch of other changes.

While I'm going to fix a few more bugs for 2.0, it would be awesome if you all can start testing master. I will feel a lot more comfortable releasing it if it's been tested on existing codebases

[jlongster]

Help needed: please test master! especially test the browser builds and precompiled templates; I've changed how those loaders are created.

I'm making lots of noise because I'm not pushing 2.0 out until I hear back from everybody that everything is not broken. :)

[carljm]

I tried upgrading one of my projects, and am getting TypeError: nunjucks.require is not a function in nunjucks-compat.js. So apparently nunjucks-compat.js will need updating before we can update; haven't looked into what changes exactly are needed yet.

[jlongster]

nunjucks-compat.js isn't officially supported, but we should include that in this repo. The browser files are built with webpack now so there is not an external require, but I'm sure we can make that work.

[carljm]

Yep, I know nunjucks-compat.js isn't part of Nunjucks or supported; just noting that it was affected by a backwards-incompatible change (dunno if there's any chance of that change affecting anyone else, but it probably needs documenting in the release notes) and that I need to figure that out before I can do any further testing of 2.0.

[grimurd]

Would love to see a parent tag like swig has, currently there is a extension for it (see here) but i think this tag should be a part of nunjucks itself. It's a very useful tag.

[jlongster]

@Sk1zY if you look at that tag, all is does is transform into {{ super() }}. That's what you use to get the parent block. He must have done that because he was transitioning from swig.

[jlongster]

I forgot about #267. @carljm do you have time to look at it?

[jlongster]

I'm thinking about #267 more and it's going to need to inject checks wherever we need to check for a falsy value, which is going to be quite messy. Still a little unsure if it's feasible or not

[carljm]

Yeah, I think #267 is probably doable but will be quite invasive. It had kind of dropped off my radar; I could take a look today, but not sure I can have a PR ready by Monday. (And if/when I have a PR ready, I'm not sure if you'll want it.) So I think we should forget about it for 2.0, and just document it as a known Jinja incompatibility.

[jlongster]

Sounds good. There's always 3.0 in the (distant?) future if people are that passionate about the change.

[carljm]

FWIW, I tried latest master again on my current project, and everything looks good here, including the built-in installJinjaCompat. Release away!

(Though it sure was confusing for a few minutes when I tried to run the site with latest runtime but hadn't recompiled templates yet - things broke in very interesting ways!)

Thanks @jlongster for all your work on getting the 2.0 release ready to ship! Much appreciated.

[keelii]

a. I'd like there is a setMacro API for some builtin macro ^!^
b. include tag add some params will be better

{% include '../nunjucks.html' foo bar %}

c. some self-close tag maybe looks nice

{% remote "http://nunjucks.com" params %}

[boutell]

Please see #465 and #466. There seems to be a significant bug affecting the ability to define a macro in an included file.

[eddywashere]

I think nunjucks 2.0 would be a great way to bring over people who were using swig or at least attract more people by taking some pointers from swig's developer ux. This includes fixing a couple of problems I ran into while working with nunjucks or documenting these common examples

• registering helpers in projects that use consolidate.js are not as straight forward as doing it for swig was (I gave up)
• date filter is not included, (include any other common filters) (see above)
• include js logic tags: || and && (not essential, just weird to not have it)

Or, at the very least I would love to see something in the docs about switching over from swig.

Thoughts?

[ghost]

@eddywashere Regarding helpers - do you want to consolidate.js to let you register custom Nunjucks extensions and filters? If so, consolidate.js is going to have to be patched to allow it because the way they are initializing Nunjucks right now seems incomplete.

It also looks like consolidate.js only renders strings and not file paths, so Nunjucks would have to be patched to make path information available to renderString (if you want relative paths to work with extend and include). Alternatively, if you can get a file path, pass it to render as seen here. If neither are possible, then all paths used with the extend or include tags will have to be relative to the current working directory or the site root.

You can see how Nunjucks lets you register custom filters and extensions to the Environment (env) here in gulp-nunjucks-api. Compare that to how the Nunjucks support is implemented in consolidate.js. (I myself actually had to fork gulp-nunjucks into gulp-nunjucks-api in order to get a working custom environment.)

Once you are able to load custom filters, you can make your own date filter or copy it from swig - although I do agree it would be nice to have something built-in for this.

So that leaves the logic tags. (You said tags, but I think you mean operators.) I think there might be a conflict of interest here philosophically, since one of the major tenants of jinja style templating is to reduce logic in templates. Worst case, you can always implement them as filters called or and and or make your own tag extension that handles it, once consolidate.js is patched to let you access more of the Nunjucks API.

[eddywashere]

I guess it's logic expressions according to the docs, http://mozilla.github.io/nunjucks/templating.html#logic - and and or are already there. If that's there, it makes sense to also have familiar js expressions && and ||.

RE: helpers/filters was just documentation about accessing the env from consolidate.js. I just gave it another go and found out how to access it properly.

var dateFilter = require('nunjucks-date-filter');
var cons = require('consolidate');
var env;

// grab nunjucks
cons.requires.nunjucks = require('nunjucks');
// grab env
env = cons.requires.nunjucks.configure('./templates', {watch: false});

env.addFilter('date', dateFilter);

That was more work than it should have been. Just needs to be documented to save others the trouble.

[ghost]

Thanks for posting your solution to accessing the configuration.

[ghost]

BTW, I agree about the && and || situation after reviewing it. I don't use any logic expressions in my templates, so I didn't even know they existed! I am putting most logic in filters or in "code-behind" functions or what I call "locals files" in gulp-nunjucks-api.

[eddywashere]

Last item on the wish list. Allow ! when evaluating a variable. It's incredibly weird to do {% if not isVisible %} vs what is done in js {% if !isVisible %}. Some other engines even have unless.

[ArmorDarks]

Guys, Nunjucks is standalone templating language, not JavaScript. Really,

If you want JavaScript — you should consider React or something similar to it.

[eddywashere]

@ArmorDarks that is not a helpful suggestion. This project certainly is JavaScript and is only inspired by jinja. The changes I'm asking for allow backwards compatibility for people coming from swig and people already familiar with the Javascript language. It does not take away from the core concepts of the project. It only improves the experience of using it for the community.

[ghost]

@ArmorDarks He's not asking for full Javascript logic support, just for compatibility with swig, which is also inspired by jinja.

[ricordisamoa]

JavaScript-like logic operators were rejected as #344.

[ghost]

Revisiting #344 would be a good idea in my opinion. @jlongster stated that making moves towards Nunjucks becoming a big, successful and active open source project is especially important for this release. Turning away a small and easily implemented features such as this (even as an optional mode) because of questionable ideological reasons seems counterintuitive to that end.

Another thing is that you're going to start seeing more swig users here since paularmstrong/swig#628. If you want those users to stick around, you might as well make a mode that is compatible with what they want.

[ricordisamoa]

So a swigCompat option that swaps Python-like operators with JavaScript ones might make sense?

[boutell]

The meta-layers that allow you to implement your own blocks, etc. in
nunjucks are pretty robust. Is it possible to implement swig-compat as a
freestanding module for those who want it?

On Sat, Jun 27, 2015 at 12:38 PM, ricordisamoa [email protected]
wrote:

So a swigCompat option that swaps Python-like operators with JavaScript
ones might make sense?

—
Reply to this email directly or view it on GitHub
#347 (comment).

*THOMAS BOUTELL, *DEV & OPS
P'UNK AVENUE | (215) 755-1330 | punkave.com

[eddywashere]

A good migration strategy would be to support both Python/JavaScript syntax with the swigCompat option. This would allow people to just switch out swig for nunjucks in package.json

[devoidfury]

@waynebloss @boutell for the javascript-like operators, we would need to change the lexer as the code that handles | and ! branches off from the part that reads symbols, see https://github.com/mozilla/nunjucks/blob/master/src/lexer.js#L152 -- probably not doable with a freestanding module/custom blocks

[boutell]

Mm. And I suppose that level of flexibility in the extension API could
affect performance... well, no, it's just a one-time parse after all...

On Sat, Jun 27, 2015 at 1:57 PM, Thomas Hunkapiller <
[email protected]> wrote:

@waynebloss https://github.com/waynebloss @boutell
https://github.com/boutell for the javascript-like operatiors, we'll
need to change the lexer as the code that handles | branches off from the
part that reads symbols, see
https://github.com/mozilla/nunjucks/blob/master/src/lexer.js#L172 --
probably not doable with a freestanding module/custom blocks

—
Reply to this email directly or view it on GitHub
#347 (comment).

*THOMAS BOUTELL, *DEV & OPS
P'UNK AVENUE | (215) 755-1330 | punkave.com

[carljm]

@jlongster Any news on a 2.0 release?

(FWIW I just filed PRs #480 and #481, both fixes which might be good to get into 2.0, since it hasn't been released yet. Assuming they look good to you.)

[jlongster]

I've been traveling a lot this summer, so sorry :( I'd really like to get 2.0 out the door this week. And then I'm going to filter through the new issues and tag them.

I hadn't seen the swig maintainer step back. Perhaps we could smooth the transition a bit. But as I stated in my previous comments, I also haven't really had much time to maintain it and am looking for more people help triage issues and such. I may have to step out completely as well. It's in you all's interested to make decisions anyway since you all are the one using it; I don't have anymore projects using this.

@carljm I'll look at those issues, but if they aren't obviously small changes we can just merge them in 2.1. Seams like there isn't any intent to break anything backwards compatible.

[carljm]

Those fixes are neither small nor fully working yet, so don't worry about them for 2.0.

[jlongster]

I just pushed out 2.0. Please file issues if anything breaks, and file new issues for anything discussed in here that was not finalized.

[boutell]

We're not ready for this jump yet, because of #466, but it's part of our
plan to make Apostrophe 0.6 compatible with 2.0.

On Thu, Aug 27, 2015 at 8:13 PM, James Long [email protected]
wrote:

Closed #347 #347.

—
Reply to this email directly or view it on GitHub
#347 (comment).

*THOMAS BOUTELL, *DEV & OPS
P'UNK AVENUE | (215) 755-1330 | punkave.com

[atian25]

+1 to easier custom tag API. Creating custom tags is far more difficult than it should be imo.