SAX interface unexpectedly gets locale-altered float representation.

[jzakrzewski]

Description

I didn't expect (neither from experience, nor from the documentation) that the float string representation would be altered according to the locale when feeded into the SAX parser.
I understand, why it has to be altered internally, but I'd expect the original string in the callback.

Reproduction steps

LC_NUMERIC=de_DE.UTF-8

{ "aFloat": 42.0 }

bool ParserImpl::number_float( number_float_t val, const string_t &s )
{
    std::cout << s << '\n';
}

Expected vs. actual results

Expected: 42.0
Actual: 42,0

Minimal code example

#include <nlohmann/json.hpp>
#include <iostream>
#include <string_view>
#include <clocale>

using Json = nlohmann::json;

struct ParserImpl : public nlohmann::json_sax<Json>
{
    bool null() override { return true; };
    bool boolean( bool val ) override { return true; };
    bool number_integer( number_integer_t val ) override { return true; };
    bool number_unsigned( number_unsigned_t val ) override { return true; };
    bool number_float( number_float_t val, const string_t &s ) override {
        std::cout << s << '\n';
        return true; 
    };
    bool string( string_t &val ) override { return true; };
    bool binary( binary_t &val ) override { return true; };
    bool start_object( std::size_t elements ) override { return true; };
    bool key( string_t &val ) override { return true; };
    bool end_object() override { return true; };
    bool start_array( std::size_t elements ) override { return true; };
    bool end_array() override { return true; };
    bool parse_error( std::size_t position, const std::string &last_token,
                      const nlohmann::detail::exception &ex ) override { return true; };
};

int main() {
    if(!setlocale(LC_NUMERIC, "de_DE.utf8"))
        return -1;
    std::string_view data { R"json({ "aFloat": 42.0 })json" };
    ParserImpl sax {};
    Json::sax_parse( data, &sax );
    return 0;
}

g++-12 -o jsontest -I /path/to/nlohmann_json-src/single_include/ --std=c++20 jsontest.cpp

Error messages

No response

Compiler and operating system

Linux/gcc-12, Windows/MSVC19.35

Library version

3.11.2

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[nlohmann]

Can you please share a complete example?

[jzakrzewski]

Done.
Note that you need the "de_DE.utf8" locale in your system. Alternatively use any other locale with something else than a "." as decimal separator.

[nlohmann]

Thanks!

[nlohmann]

The error is that we replace . in the input by the decimal separator from the locale to be able to use std::strtod under the hood to make sure we properly parse floating-point numbers when a different locale is set. We forgot, however, that the string we pass to the SAX number_float function still contains the replaced separator.

To fix this, I need a way to efficiently replace this to . once we're done with std::strtod.

[nlohmann]

It took me a while to adjust the CI to run the tests with different locales, but I think the fix is good to go.