Inconsistent/unnecessary escaping of characters in inputs #9859

nikku · 2022-07-21T13:48:02Z

Describe the bug

As a user I want to pass special chars in strings (FEEL ref) in a way that I can consume them in a job worker.

I've defined the following input mappings:

name	value
dynamic_input	`= "Hello\n\nYOU!"`
static_input	`Hello\n\nYOU!`
static_input_explicit_newlines	`HELLO<newline><newline>YOU!`

Executing my process I receive the following input variable values in operate (but also in my process worker):

name	value
dynamic_input	`"Hello\\n\\nYOU!"`
static_input	`"Hello\\\\n\\\\nYOU!"`
static_input_explicit_newlines	`"Hello\\n\\nYOU!"`

As you can see inputs are escaped differently depending on whether I use \n encoded newlines or actual newline characters. Escaping happening in the first place is fine I believe as this is a JSON string and \\n is the standard way to encode newlines there.

To Reproduce

Deploy + run test process
See double escaping happening in string inputs

Expected behavior

We do the unescaping magic ourselves so that in user land no one needs to worry:

I define an input mapping foo = "a\nb"
In my job worker I see foo = "a\nb"

As a more user un-friendly alternative we could consider: String inputs are escaped like expression inputs (no double escaping), we clearly document how escaped strings shall be worked with (and how un-escaping has to be done in user land). This is closer to the existing behavior.

Log/Stacktrace

none.

Environment:

OS: Linux
Zeebe Version: 8.0.2
Configuration: nothing special

Support:
SUPPORT-15887
SUPPORT-17621

Tasks

Give feedback

Verify correct behaviour of input mappings after bump to FEEL 1.17.3 zeebe#15445

area/test kind/bug version:8.4.0 version:8.4.0-alpha2
Special characters in static inputs should not be escaped zeebe#15447

kind/bug version:8.4.0
Verify unicode characters are not escaped in input mappings zeebe#15446

area/test component/zeebe impact/high kind/task
Highlight the new behavior change in the release notes
Inform the support team
Update the input mappings docs.
Options

The text was updated successfully, but these errors were encountered:

nikku · 2022-07-21T13:53:07Z

To add to this I see zeebe escaping unicode characters (i.e. 🤩) despite JSON having full unicode support.

Hello\n\nYOU! 🤩

turns into

"Hello\\n\\nYOU! \uD83E\uDD29"

"Hello\\n\\nYOU! 🤩" is a valid JSON string.

nikku · 2022-08-15T15:39:51Z

Is there a chance this can be fixed at some point? Or do we regard this as a nasty-bug-turned-public-api?

menski · 2022-08-19T08:18:50Z

@felix-mueller this is something which mostly effects connectors but it is very unexpected behavior. We would like to understand if you have any input on priority for this.

felix-mueller · 2022-08-19T08:48:31Z

It should be fixed at some point. Since we still need to root-cause and better understand the issue it is probably not S. We won't priotize it for the next few iterations (first we need to finish instance modification, DMN evaluation and delete definitions).

menski · 2022-08-19T08:50:13Z

Thanks Felix, I put the priority on "Later" for now

darox · 2022-09-12T08:03:53Z

I don't know whether I'm hitting the same issue. I'm using the Go client to ingest JSON formatted data into Zeebe. Jobs are failing due to escaped double quotes. If I set vars manually via simple-monitor everything works as expected.

igpetrov · 2022-11-14T12:14:18Z

Hi team, is there any ETA to fix this? It is affecting currently 3 connectors.

felix-mueller · 2022-11-14T13:27:45Z

@igpetrov currently we have planned to look into it after instance modification, DMN evaluation and Delete Definitions. these topics are scheduled to be included in our April minor release, so if we don't change priorities we probably will look at it in a few months.

from connectors perspective can we live with this for a few months?

igpetrov · 2022-11-14T14:08:54Z

Hi @felix-mueller, appreciate your update. Absolutely. Just wanted to understand the ETA. Thank you!

tmetzke · 2022-11-17T05:53:39Z

For reference, in Connectors, we currently do the following to work around this with data we receive as variables after Zeebe has processed the user input:

StringEscapeUtils.unescapeJson(jsonStringVariable)

This uses StringEscapeUtils from org.apache.commons:commons-text.

Currently there is a known issue where modeler adds an escape character to special characters. Example: "Hello, World 🤩" will end up as "Hello, World \uD83E\uDD29". This commit forces unescaping. Functionality es expected to be removed after related issues are fixed. Closes #15. Refs: - camunda/camunda#9859 - camunda/camunda-modeler#2031

nikku · 2023-01-23T21:01:55Z

@felix-mueller The longer we keep such behavior in the core, the harder it will be for people to unbuild their hacks (trying to work with zeebe data). My 2️⃣ 🪙.

nikku · 2023-02-13T10:03:18Z

A similar issue is reported by a customer through SUPPORT-15887.

nikku · 2023-02-13T10:17:48Z

I've updated the issue with a clear expected behavior, reflecting https://jira.camunda.com/browse/SUPPORT-15887, but also #9859 (comment) have reported.

menski · 2023-02-17T09:38:27Z

Discussed in the planning meeting on 2023-02-17. Right now we don't have capacity to work on this, and deprioritize it as the customer has a workaround available.

korthout · 2023-02-22T09:13:49Z

When we work on this, we need to make sure not to break user space when backporting it to patches. We can do this by adding a configuration setting that allows switching between the current (broken) and new (fixed) behaviors. The default should stay the same in patch releases, but in a minor release we can change the default. This should be clearly communicated in the release notes (and blog) so users understand the change.

abbasadel · 2023-07-25T12:37:20Z

This issue was mentioned again in SUPPORT-17621 and blocking connectors team's here

korthout · 2023-07-26T08:05:23Z

When we resolve this issue, please verify whether the fix also resolves:

EvaluateDecision request returns double quoted string #13551

mschoe · 2023-11-07T13:00:54Z

I run into this issue when using the GH outbound connector and create a new issue with a multi line description field. As this is a quite common use case I'm wondering when we are planning to provide a fix. @felix-mueller is this still scheduled for the 8.5 release in April 2024?

christian-konrad · 2023-11-09T17:40:50Z

Would https://github.com/camunda/product-hub/issues/1040 help here or is it just a quick fix on string behavior?

saig0 · 2023-11-10T05:08:46Z

@nicpuppa started to work on this issue by fixing the escaping in the FEEL engine here. We plan to deliver a fix in one of the next patch releases. 🚀

@christian-konrad the FEEL templating is a bigger topic. Let's fix the issue with the escaping first. 👍

christian-konrad · 2023-11-10T08:04:27Z

Perfect @saig0 ! Thanks

MaxTru · 2023-11-22T09:57:13Z

Hi @abbasadel ,

according to the previous assessment this should have been fixed with this version bump. Can you verify from Zeebe perspective please and update this issue? Thanks

megglos · 2023-11-29T12:18:24Z

We need to clarify whether this change is breaking, especially in regards to present workaround in other components such as https://github.com/camunda/connector-slack/issues/48 &
https://github.com/camunda/connector-google-drive/issues/20
While for the REST connector no workaround seems to be in place and there a patch by zeebe would resolve the issue.

If it's breaking we may prefer to only release it in the next minor or at least coordinate with the affected components to remove the workaround applied for a patch. But then we have the risk that we have a dependency between patch levels of connectors and zeebe.

@abbasadel will sync with @saig0 on this tomorrow and ping back.

abbasadel · 2023-11-30T15:45:22Z

Hi everyone, this is Zeebe's assessment:

The recent bug fix on the FEEL engine is not backward compatible from Zeebe's point of view. However, we believe, based on @saig0’s feedback and our tests, the impact on our users is low since the recommended workaround we provided was to use unescapeJson API from Apache Commons which handles breaking change.
As the previous behavior was wrong, mentioning the new behavior change in the release notes and as an announcement would be needed
This FEEL fix should resolve bug, to be validated by Connectors/Operate teams.
The support team should also be aware of this FEEL behavior change once release

MaxTru · 2023-11-30T20:03:25Z

Hi everyone, this is Zeebe's assessment:

The recent bug fix on the FEEL engine is not backward compatible from Zeebe's point of view. However, we believe, based on @saig0’s feedback and our tests, the impact on our users is low since the recommended workaround we provided was to use unescapeJson API from Apache Commons which handles breaking change.

As the previous behavior was wrong, mentioning the new behavior change in the release notes and as an announcement would be needed

This FEEL fix should resolve bug, to be validated by Connectors/Operate teams.

The support team should also be aware of this FEEL behavior change once release

Thanks @abbasadel . Will you and the ZPA team please ensure that all these things happen? If so, can we please track this via a breakdown and DRI assignments? Why is the issue not in in progress and has no assignee?

abbasadel · 2023-12-01T08:41:49Z

@MaxTru, we will take care of this issue

nicpuppa · 2023-12-01T11:21:15Z

Hi everyone, as follow up I created 3 separate issues to reduce the scope and make easier to follow the progress:

Verify correct behaviour of input mappings after bump to FEEL 1.17.3 -> this is a testing purpose issue, to verify that the new FEEL version has the correct behaviour
Special characters in static inputs should not be escaped -> as mentioned also by @saig0 in this comment, characters in static input are still escaped. Static input are handled directly by Zeebe
Verify unicode characters are not escaped in input mappings -> after resolving the feel issue related to unicode, on Zeebe we should verify that the behaviour is as expected (unicode characters should not be escaped)

~~In accordance with @abbasadel and @saig0 I will close this issue~~ This issue is still open only for tracking purpose, I mistakenly closed the issue

abbasadel · 2023-12-01T12:29:55Z

Thanks, @nicpuppa, for the updates.

I added the sub-issues into the main issue description for clarity and tracking purpose

abbasadel · 2023-12-08T12:47:13Z

Hi everyone,

The dynamic_input unnecessary escaping bug is currently resolved in Zeebe 8.3.4 release
The bug fix is Verify correct behaviour of input mappings after bump to FEEL 1.17.3 #15445 resolves the connectors team bug. Thanks @saig0 and @nicpuppa
The corrected behavior change is highlighted in the release notes. Thanks @saig0 for the wording.
The support team is informed about the corrected behavior change
Next action: Update input mappings docs as suggested by @akeller

akeller · 2023-12-08T22:27:06Z

@abbasadel will this be included with the alpha release next week? Since this is a Zeebe task, I'm not sure what the expectation is. When you open your docs PR, please follow our PR template so we can review ASAP if you want it reviewed, merged, and published on Tuesday.

abbasadel · 2023-12-12T07:16:42Z

Thanks @akeller. The docs PR is already merged

15696: Dump feel-engine 1.17.4 r=saig0 a=saig0 ## Description Dump the FEEL engine from 1.17.3 to [1.17.4](https://github.com/camunda/feel-scala/releases/tag/1.17.4). The new version includes a bug fix that double quotes were escaped in the resulting string value. Read more about the issue [here](camunda/feel-scala#778). ## Related issues related to #9859 related to #13551 Co-authored-by: Philipp Ossler <[email protected]>

npepinpe · 2024-06-03T11:44:50Z

It seems this can be closed as done? @nicpuppa ?

nicpuppa · 2024-09-24T08:14:56Z

It seems this can be closed as done? @nicpuppa ?

Sorry, I missed the comment @npepinpe 🙇 . There is still a task to be completed, that is blocked by this FEEL issue

nikku added the kind/bug Categorizes an issue or PR as a bug label Jul 21, 2022

nikku changed the title ~~Inconsistent escaping of characters in inputs~~ Inconsistent/unnecessary escaping of characters in inputs Jul 21, 2022

igpetrov mentioned this issue Nov 10, 2022

fix: changes for emoji camunda/connector-sns#25

Merged

igpetrov mentioned this issue Dec 23, 2022

UIPath error: 'queueItemParameters must not be null' camunda/connectors#127

Closed

ChrisKujawa added the component/engine label Jan 3, 2023

markfarkas-camunda mentioned this issue Jan 24, 2023

REST connector sends different data format than it should camunda/connectors#239

Closed

nikku added the support Marks an issue as related to a customer support request label Feb 15, 2023

korthout assigned menski Feb 17, 2023

menski removed their assignment Feb 17, 2023

korthout mentioned this issue Jul 25, 2023

EvaluateDecision request returns double quoted string #13551

Open

abbasadel added impact/medium Marks issues relieving an occasional pain or fulfilling a reasonable desire and removed impact/medium Marks issues relieving an occasional pain or fulfilling a reasonable desire labels Jul 26, 2023

abbasadel added the impact/high Marks issues relieving a recurring pain or fulfilling an immediate need label Aug 11, 2023

saig0 mentioned this issue Aug 28, 2023

Newline characters should not be escaped camunda/feel-scala#701

Closed

sbuettner mentioned this issue Nov 13, 2023

GitHub Create Issue Comment Connector: gracefully handle numbers for issueNumber camunda/connectors#1378

Open

abbasadel self-assigned this Dec 1, 2023

abbasadel assigned nicpuppa Dec 1, 2023

nicpuppa mentioned this issue Dec 1, 2023

Special characters in static inputs should not be escaped #15447

Closed

nicpuppa closed this as completed Dec 1, 2023

nicpuppa reopened this Dec 1, 2023

saig0 mentioned this issue Dec 21, 2023

Dump feel-engine 1.17.4 #15696

Merged

14 tasks

deepthidevaki added the version:8.4.0 Label that represents issues released on verions 8.4.0 label Jan 4, 2024

aleksander-dytko added the kind/epic Categorizes an issue as an umbrella issue (e.g. OKR) which references other, smaller issues label Jan 5, 2024

romansmirnov added the component/zeebe Related to the Zeebe component/team label Mar 5, 2024

johnBgood mentioned this issue Jun 21, 2024

RabbitMQ Connector does unnecessary message unsescape action. camunda/connectors#2730

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inconsistent/unnecessary escaping of characters in inputs #9859

Inconsistent/unnecessary escaping of characters in inputs #9859

nikku commented Jul 21, 2022 •

edited by abbasadel

Loading

Tasks

nikku commented Jul 21, 2022 •

edited

Loading

nikku commented Aug 15, 2022

menski commented Aug 19, 2022

felix-mueller commented Aug 19, 2022

menski commented Aug 19, 2022

darox commented Sep 12, 2022

igpetrov commented Nov 14, 2022

felix-mueller commented Nov 14, 2022

igpetrov commented Nov 14, 2022

tmetzke commented Nov 17, 2022 •

edited

Loading

nikku commented Jan 23, 2023

nikku commented Feb 13, 2023

nikku commented Feb 13, 2023

menski commented Feb 17, 2023

korthout commented Feb 22, 2023

abbasadel commented Jul 25, 2023 •

edited

Loading

korthout commented Jul 26, 2023

mschoe commented Nov 7, 2023

christian-konrad commented Nov 9, 2023

saig0 commented Nov 10, 2023

christian-konrad commented Nov 10, 2023

MaxTru commented Nov 22, 2023

megglos commented Nov 29, 2023 •

edited

Loading

abbasadel commented Nov 30, 2023

MaxTru commented Nov 30, 2023 •

edited

Loading

abbasadel commented Dec 1, 2023

nicpuppa commented Dec 1, 2023 •

edited

Loading

abbasadel commented Dec 1, 2023

abbasadel commented Dec 8, 2023 •

edited

Loading

akeller commented Dec 8, 2023

abbasadel commented Dec 12, 2023

npepinpe commented Jun 3, 2024

nicpuppa commented Sep 24, 2024

Inconsistent/unnecessary escaping of characters in inputs #9859

Inconsistent/unnecessary escaping of characters in inputs #9859

Comments

nikku commented Jul 21, 2022 • edited by abbasadel Loading

Tasks

nikku commented Jul 21, 2022 • edited Loading

nikku commented Aug 15, 2022

menski commented Aug 19, 2022

felix-mueller commented Aug 19, 2022

menski commented Aug 19, 2022

darox commented Sep 12, 2022

igpetrov commented Nov 14, 2022

felix-mueller commented Nov 14, 2022

igpetrov commented Nov 14, 2022

tmetzke commented Nov 17, 2022 • edited Loading

nikku commented Jan 23, 2023

nikku commented Feb 13, 2023

nikku commented Feb 13, 2023

menski commented Feb 17, 2023

korthout commented Feb 22, 2023

abbasadel commented Jul 25, 2023 • edited Loading

korthout commented Jul 26, 2023

mschoe commented Nov 7, 2023

christian-konrad commented Nov 9, 2023

saig0 commented Nov 10, 2023

christian-konrad commented Nov 10, 2023

MaxTru commented Nov 22, 2023

megglos commented Nov 29, 2023 • edited Loading

abbasadel commented Nov 30, 2023

MaxTru commented Nov 30, 2023 • edited Loading

abbasadel commented Dec 1, 2023

nicpuppa commented Dec 1, 2023 • edited Loading

abbasadel commented Dec 1, 2023

abbasadel commented Dec 8, 2023 • edited Loading

akeller commented Dec 8, 2023

abbasadel commented Dec 12, 2023

npepinpe commented Jun 3, 2024

nicpuppa commented Sep 24, 2024

nikku commented Jul 21, 2022 •

edited by abbasadel

Loading

nikku commented Jul 21, 2022 •

edited

Loading

tmetzke commented Nov 17, 2022 •

edited

Loading

abbasadel commented Jul 25, 2023 •

edited

Loading

megglos commented Nov 29, 2023 •

edited

Loading

MaxTru commented Nov 30, 2023 •

edited

Loading

nicpuppa commented Dec 1, 2023 •

edited

Loading

abbasadel commented Dec 8, 2023 •

edited

Loading