res.redirect unpredictable sometimes with Unicode and even ANSI/ASCII

[Martii]

We get on occasion with § the escaped version (e.g. only %A7 a.k.a ASCII 245) during a redirect() (shouldn't be but encodeURI/encodeURIComponent gives %C2%A7)... I've spent about 3 hours trying different things... unfortunately it's not an easy bug to find. I've poured over our code in a commit that I'm attempting to get pushed to our organization when it's complete... but I'm having to ensure at the very least encodeURI is applied to the path... currently I am defaulting to encodeURIComponent if needed for express routing... even though it's already applied to the String already in between the slashes when needed... same as GitHub does all the time with encodeURIComponent. Some redirects work like a charm depending on their content... others don't.

Anyhow... hope some of this info helps track this down a little better... the work around so far is to double encodeURI or encodeURIComponent on each individual p/a/t/h/to which I know will eventually fail down the line.

Providing a reduced test case will be nearly impossible I think... but I am able to reproduce this with node@4.3.x and express@4.13.4 in our project.

Here's a few stack traces:

Stringy is exactly /scripts/Marti/RFC_2606§3_-_Hello,_World! before redirect() and passed as aRes.redirect(Stringy) with defaulting to 200 status code... tried forcing .status(200).redirect(Stringy) too... no luck.

URIError: Failed to decode param 'RFC_2606%A73_-_Hello,_World!'
    at decodeURIComponent (native)
    at decode_param (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/layer.js:167:12)
    at Layer.match (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/layer.js:143:15)
    at matchLayer (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:557:18)
    at next (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:216:15)
    at ~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/less-middleware/lib/middleware.js:315:14
    at Layer.handle [as handle_request] (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:312:13)
    at ~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:330:12)
    at next (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:271:10)
    at ~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/app.js:255:3
    at Layer.handle [as handle_request] (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:312:13)
    at ~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:330:12)
    at next (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:271:10)
    at express_minify_middleware (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express-minify/index.js:185:5)
    at Layer.handle [as handle_request] (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:312:13)
    at ~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/router/index.js:330:12)

As it can be seen here the "Stringy" with this one is /users/テスト

GET /users/%E3%83%86%E3%82%B9%E3%83%88 200 468.579 ms - -
_http_outgoing.js:351
      throw new TypeError('The header content contains invalid characters');
      ^

TypeError: The header content contains invalid characters
    at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:351:13)
    at ServerResponse.header (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/response.js:718:10)
    at ServerResponse.location (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/response.js:835:8)
    at ServerResponse.redirect (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/express/lib/response.js:874:8)
    at Query.<anonymous> (~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/controllers/admin.js:504:10)
    at ~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/mongoose/node_modules/kareem/index.js:177:19
    at ~/repo/git/OpenUserJS.org/martii/OpenUserJS.org/node_modules/mongoose/node_modules/kareem/index.js:109:16
    at nextTickCallbackWith0Args (node.js:415:9)
    at process._tickDomainCallback (node.js:385:13)

Tested UserAgents:

• Mozilla/5.0 (X11; Linux x86_64; rv:42.0) Gecko/20100101 SeaMonkey/2.39
• Mozilla/5.0 (X11; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0
• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36
• Probably more when it's this many browsers

Anchor tags with these exact same values work 100%... just not redirect() in express's res Object... so it's a bit baffling.

Thanks for you patience and assistance with the provided intel. :)

[Martii]

... which I know will eventually fail down the line.

And there's a failure with it... http://localhost:8080/scripts/Marti/30%2525_volume/issues/Final_test ... should have been http://localhost:8080/scripts/Marti/30%25_volume/issues/Final_test... so the work-around definitely isn't working as I predicted with double encoding... which means we're stuck until this is addressed.

Guess we'll attempt to match encodeURIComponenting all folders like GH does until a fix comes by.

[Martii]

Bahh... Location and Content-Location headers require all urls to be at least URI encoded as per specs... e.g. all the browsers are adding in an escape for the section symbol which may be an issue with them... however express is a pass through on this.

Apologies for the noise and closing.

[dougwilson]

Hi @Martii, sorry we weren't able to get to you before you figured this out. I'm glad you were able to determine the issue! Always feel free to let us know if you have questions, and even drop by our support Gitter room, https://gitter.im/strongloop/express