JSON Api refuse to decode null value

[timdesi]

Describe the bug
I am trying to use JSON API to decode json messages with json_obj_parse function. Everything was working fine until same of keys came with null values. json_obj_parse function always return with error code -22 (EINVAL).

I have checked null at json.org and jsonlint.com, looks like that null is valid json value.

Try to parse the simplest json possible, but with same result.

{ "key" : null }

Try to look at zephyr json test, there is comment actually describing this situation.

static void test_json_invalid_null(void)
{
	struct encoding_test encoded[] = {
		/* Parser will recognize 'null', but refuse to decode it */
		{ "{\"some_string\":null }", -EINVAL},
		/* Null spelled wrong */
		{ "{\"some_string\":nutella }", -EINVAL},
	};

	parse_harness(encoded, ARRAY_SIZE(encoded));
}

To Reproduce

1. Try to parse json below with json_obj_parse function from JSON Api.

{
    "key" : null
}

struct test_json_null {
    const char* key;
    
};

struct json_obj_descr descr3[] = {
    JSON_OBJ_DESCR_PRIM(struct test_json_null, key, JSON_TOK_STRING),
    
};

Expected behavior
I expect JSON API to decode null values.

Impact
annoyance, showstopper

Logs and console output

starting test - test_parsing_message_with_null_file
D: Start parsing commands
D: ParseBuffer : 19 {    "key" : null}
D: json parse ret : -22
D: End parsing settings
Error parsing file [-22]

    Assertion failed at ../src/test_settings.c:181: test_parsing_message_with_null_file: parsing_message_with_null_file() == TC_PASS is false

FAIL - test_parsing_message_with_null_file
===================================================================
Test suite test_json_tests failed.
===================================================================
PROJECT EXECUTION FAILED

Environment (please complete the following information):

• OS: Centos 7
• Toolchain : Zephyr v2.3.0

Additional context
Add any other context about the problem here.

[pabigot]

Yep, it doesn't work, and I don't think it ever will.

The intent and use case behind the Zephyr JSON module isn't documented, but it appears to be designed to process nested non-empty structures. There's no provision for indicating that an object can be skipped. This is clear from the basic API:

struct mystruct val;

int rc = json_obj_parse("null", strlen("null"), descr, ARRAY_SIZE(descr), &val);

A successful return value indicates that the JSON text provided what's necessary to set all fields of the struct mystruct passed in via val. There's no return value that indicates the structure was absent.

If you need to process JSON that has more free-form content you'll need a different JSON support library. I've had very good experience in the past with jsmn, but it does require a buffer large enough to hold a record of all tokens provided. If that isn't usable I would consult the Google.

I'm converting this to an enhancement as it isn't something that can be easily fixed.

[timdesi]

thx. @pabigot jsmn library works us expected.

Dear @MaureenHelm thx. a lot and feel free to close this issue. Could you consider to add jsmn us alternative to current JSON Api?

[nashif]

not sure why it was closed and something that can't be easily fixed does not makes it an enhancement either. Also, it seems with have a proposal for a fix.

[pabigot]

It was an enhancement because the json API wasn't designed to support missing content. If the input specified doesn't cover the entire structure, there's a potential security issue because whatever was left in that buffer could be misinterpreted.

Looks like #28905 requires that the target object be pre-initialized with values that can be interpreted as "this isn't here". That may not be possible in general, especially when existing code relies on presence of all described data.

So I don't think this is a bug, it's a design limitation that is not trivial to eliminate.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.