-
Notifications
You must be signed in to change notification settings - Fork 365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug on on referrer header when value contains §ion
like www.asdf.com?a=1§ion=2
#503
Comments
@federico-piazza Seems like a bug to me since you have both Do you suppose you could attach a simple 2 or 3 line code fragment that will allow us to reproduce this (assuming these config settings) so I don't have to chase down exactly exactly which Encoder method you are calling. That would help us a lot. Thanks. |
hey @kwwall not sure really what snippet you need from me, so I think this is what you look for:
The rest is configuration con ESAPI.properties
|
Yeah; okay, I didn't realize that you were using ESAPI's
could you identify the method in |
Hope this help. The class is
The stack I could find was this
The
That method:
The
Above method has This
And that
So, here you can find the issue with the mixed encoding. |
@federico-piazza Excellent! Exactly what I was looking for. Thanks. |
Trying to help out with limited connectivity. Wrote up another method for EncoderTest that should capture the bug. Problem is that I'm not getting the IntrusionException when I run it. I may be overlooking something though. I've included below,, maybe you'll see what I'm missing.
|
Still catching up from Vacation. The long and the short of it is that the validation on the referer header needs to use the new @jeremiahjstacey , to get your test to work as expected you have to pick a URI GET parameter that maps to an HTML entity, like |
Actually I should have made a convenience method called |
I'm not convinced that '§ion=active' should have been matched as the named character reference for '§' and thus changed into '§ion=active' in the first place. My rudimentary (mis)understanding from what I vaguely recall from @planetlevel about this choice of not requiring trailing semicolons as part of the named character reference matches was that it had something to do with certain browsers out coming of Redmond, WA from a corporation whose thinking was that they know better than everyone else and in an attempt to discern the developers intent apparently allowed their browsers to match named character references that did not have trailing ';' that was part of the the W3C spec. (IMO, an abuse of Postel's Law if there ever was one.) So I think the thinking for ESAPI at the time was since all these broken browsers out there allow it (and at the time, they had the lion's share of the browser market), so then ESAPI should match named character references too if they have a missing trailing ';' in order to provide a "safe haven" of sorts. I really hate to carry something like that forward, but the only away around it and to allow backward compatibility with earlier ESAPI 2.x releases would be to set up a new ESAPI property that give developers the option to ignore the trailing ';' on named character references (which would be the default for backwards compatibility) or to require it and it not present then something like '§ion=active' would not be changed into '§ion=active'. (Because everything in ESAPI is pretty much a singleton and we presently don't have a way to reparse the ESAPI.properties file after it is initially read, you pretty much would still have to have it one way or another rather than allowing a developer to choose on a case-by-case basis, but it would still probably be better than it is.) Of Java supported default parameter values to methods like C++ does, we could easily make it backward compatible using that, but Java doesn't support that and AFAIK, has no plans to. We probably could kludge something that would work by adding a new method all the way down (in this case, maybe all the way down to StringValidationRule), but I see that as really clumsy and simply not worth it. Of course, if we can figure out somehow that what we are looking at is a URI, maybe we could require a strict adherence to the W3C spec as far as named character references go since I can't think of any time they legitimately be used within a URI. (That's certainly not the way you would URL encode them.) Thoughts? |
Very correct. I struggled with that and almost converted the HTMLEntityCodec to using semicolon-based entities but backed off.
It would be pretty simple really: I almost removed them once. We mark the HTMLEntityCodec class as Deprecated for the next release and then create a new Codec that matches the spec, change esapi.props to point to the new codec. Would be a great first issue. |
Our job is not to implement specs. Our job is to protect applications. Browsers, interpreters, and real world parsers do crazy things. They do not stick to specs. In this case, we have to canonicalize data that doesn’t meet the spec because that’s what browsers do.
I haven’t tested browsers recently, but here is a simple test you can run on Chrome right now.
<html>
<head>
</head>
<body>
<div>test <</div>
</body>
</html>
[cid:[email protected]]
…--Jeff
From: Matt Seil <[email protected]>
Reply-To: ESAPI/esapi-java-legacy <[email protected]>
Date: Tuesday, July 9, 2019 at 10:24 AM
To: ESAPI/esapi-java-legacy <[email protected]>
Cc: Jeff Williams <[email protected]>, Mention <[email protected]>
Subject: Re: [ESAPI/esapi-java-legacy] Bug on on referrer header when value contains `§ion` like `www.asdf.com?a=1§ion=2` (#503)
My rudimentary (mis)understanding from what I vaguely recall from @planetlevel<https://github.com/planetlevel> about this choice of not requiring trailing semicolons as part of the named character reference matches was that it had something to do with certain browsers out coming of Redmond, WA from a corporation whose thinking was that they know better than everyone else and in an attempt to discern the developers intent apparently allowed their browsers to match named character references that did not have trailing ';' that was part of the the W3C spec. (IMO, an abuse of Postel's Law if there ever was one.)
Very correct. I struggled with that and almost converted the HTMLEntityCodec to using semicolon-based entities but backed off.
Of course, if we can figure out somehow that what we are looking at is a URI, maybe we could require a strict adherence to the W3C spec as far as named character references go since I can't think of any time they legitimately be used within a URI. (That's certainly not the way you would URL encode them.)
It would be pretty simple really: I almost removed them once. We mark the HTMLEntityCodec class as @deprecated<https://github.com/deprecated> for the next release and then create a new Codec that matches the spec, change esapi.props to point to the new codec. Would be a great first issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#503?email_source=notifications&email_token=AAUUFTDX5ZBDJAS2AB5DPALP6SNSZA5CNFSM4H4AWSNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZQNXOI#issuecomment-509664185>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUUFTGAZPIIZHN66NQD7JLP6SNSZANCNFSM4H4AWSNA>.
|
Could have sworn I tested this before with a negative result, but I just tried it in chrome, firefox, and IE11 and they all render. |
Edge too. |
@planetlevel - I hear ya, and I don't disagree. But my point is that what we presently have is breaking things because ESAPI is overly aggressively canonicalizing everything. That would be fine if we could give developers control when it becomes problematic, but in this case we can't. (Well, rather, more accurately, we haven't.) In fact, until recently (see closed GitHub issue #476), even where we had promised that it could be disabled, we were not able to do so because of a bug. For reasons already mentioned, I don't think we can simply add some new property that disables canonicalization because that you couldn't enable it as-needed. But given that this problem ultimately happens in a JavaEE filter, I'm not sure how we can control it...maybe an init parameter for the filter. But I think it makes more sense to just not to do this for URIs. (And if we wanted it permanently disabled for everything, we already have the |
Again, the root cause of this issue isn't aggressive canonicalization. It's just that whoever developed the original canonicalize method didn't consider that URIs are a special case. We have the ability to safely and correctly canonicalize URI's now. I don't know how I missed this case, but I did and it's an easy fix. |
Nice
—Jeff
Jeff Williams
410-707-1487
…________________________________
From: Matt Seil <[email protected]>
Sent: Wednesday, July 10, 2019 1:46:41 PM
To: ESAPI/esapi-java-legacy
Cc: Jeff Williams; Mention
Subject: Re: [ESAPI/esapi-java-legacy] Bug on on referrer header when value contains `§ion` like `www.asdf.com?a=1§ion=2` (#503)
Again, the root cause of this issue isn't aggressive canonicalization. It's just that whoever developed the original canonicalize method didn't consider that URIs are a special case. We have the ability to safely and correctly canonicalize URI's now. I don't know how I missed this case, but I did and it's an easy fix.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#503?email_source=notifications&email_token=AAUUFTH75O6RMSFHAM3BZELP6YVBDA5CNFSM4H4AWSNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZUMJBI#issuecomment-510182533>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUUFTAOD64WK72HMARHTADP6YVBDANCNFSM4H4AWSNA>.
|
I know that one main advantages of canonicalize is the detection of mixed and double encoding cases, but it's original intent was to provide a canonicalized string that would be considered safe to run against our validation regexes. Maybe in the future we should consider separating the intrusion detection component and then migrate to defaulting ESAPI to the Encoder project. Jim's encoder however is encode only not decode so users of ESAPI that are using the decode methods will need a path forward--or we add decoding functionality to the Encoder project. Or we say "Oh well!" |
It wouldn’t be hard to parse the querystring, canonicalize the pieces, and rebuild the querystring. I think that would avoid problems.
—Jeff
Jeff Williams
410-707-1487
…________________________________
From: Matt Seil <[email protected]>
Sent: Wednesday, July 10, 2019 1:53:06 PM
To: ESAPI/esapi-java-legacy
Cc: Jeff Williams; Mention
Subject: Re: [ESAPI/esapi-java-legacy] Bug on on referrer header when value contains `§ion` like `www.asdf.com?a=1§ion=2` (#503)
I know that one main advantages of canonicalize is the detection of mixed and double encoding cases, but it's original intent was to provide a canonicalized string that would be considered safe to run against our validation regexes. Maybe in the future we should consider separating the intrusion detection component and then migrate to defaulting ESAPI to the Encoder project. Jim's encoder however is encode only not decode so users of ESAPI that are using the decode methods will need a path forward--or we add decoding functionality to the Encoder project. Or we say "Oh well!"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#503?email_source=notifications&email_token=AAUUFTGC2LOYFQI2QFIWL4LP6YVZFA5CNFSM4H4AWSNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZUMZXY#issuecomment-510184671>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUUFTAH7HZINR6G4WQWYVTP6YVZFANCNFSM4H4AWSNA>.
|
@federico-piazza can you provide a few more layers of your stack call? I'm interested in the path that leads to the addHeader method. For URI's we're going to have to bifurcate before that call. And I'm having a hard time reproducing @jeremiahjstacey 's unit test as well. Could you provide us with a unit test that reproduces this issue? |
So the solution is that in headers where a URI is expected, we need to ensure we use the proper canonicalize method. I just spent a couple hours testing down that path, and the @federico-piazza I think all I have to do is create some convenience methods, my hunch is that you're calling .addHeader and I just need to add an .addReferer method so the URI validation completes as expected. |
@xeno6696, excellent, so you figured it out the issue, looks like it was simple than expected. I'm not able right now to provide the info you asked me in the previous message since we are releasing to prod and we expect many annoying things. Do you know when this fix is going to be available in a next version? |
I'm writing an "addReferer" as we speak, and @kwwall and I want to try and push a release out before Sept 10. |
@xeno6696 fantastic! Thanks a lot. Do I have to modify my code or is this an internal fix transparent to me? |
@federico-piazza you'll have to use the new 'SecurityResponseWrapper.addReferer()' method that I'm adding. |
Hi @kwwall, @federico-piazza and @xeno6696 . |
The fix is present now. Just depends on whether your company allows you the flexibility to do the right thing! |
We are using ESAPI 2.2.0.0 version and found a bug when using
referrer
header.If we have below config:
And when we use a header like
Referer
http://127.0.0.1:3000/campaigns?goal=all§ion=active&sort-by=-id&status=Draft%2CLaunched
, this generates errors like this:By debugging the source code the encoding validation has this snippet on DefaultEncoder class:
As you can see, our referrer header value is decoded from this:
To this
Therefore this counts as one encoding applied, then it continues and in the next decode it goes with:
As you can see the
§ion
is decoded as§ion
and then the%2C
is decoded as,
, therefore the encoding counter is 2... which is treated as a mixed encoding and generates above error log.I think this is a bug since the url is valid to have
§ion=...
.What do you think?
The text was updated successfully, but these errors were encountered: