token pasting ## fails with nested concatenation macros

[SpareSimian]

Continued from issue #159

Error message:
testsuite\token_concat.cpp:2: syntax error: failed to expand 'wxSETUPH_PATH', Invalid ## usage when expanding 'wxCONCAT'.

Code to reproduce:

#define wxCONCAT(text1, text2)       text1 ## text2
#define wxCONCAT3(x1, x2, x3)       wxCONCAT(wxCONCAT(x1, x2), x3)
#define wxSETUPH_PATH               wxCONCAT3(vc142, _x64, _lib)
extern int wxSETUPH_PATH;

[ghost]

This doesn't appear to be valid input. Where is it from?

The formal arguments to ## are replaced with their actual arguments before pasting, but they are not further expanded until after pasting. Thus the expansion of wxSETUPH_PATH should proceed as:

wxCONCAT3(vc142, _x64, _lib)
wxCONCAT(wxCONCAT(vc142, _x64), _lib)
wxCONCAT(vc143, _x64)  ## _lib

The result of ) pasted with _lib isn't a valid token, and so the result is the much-maligned "undefined behavior". Erroring out is a perfectly acceptable action here.

See ANSI C89 6.8.3.3 or C99 6.10.3.3 et seq.

[SpareSimian]

The macros are defined here:

https://github.com/wxWidgets/wxWidgets/blob/master/include/wx/cpp.h

They're used to insert the "#pragma comment" to automatically pull in the correct library (based on compiler version and options) here:

https://github.com/wxWidgets/wxWidgets/blob/master/include/msvc/wx/setup.h

[SpareSimian]

Boost does the same thing here:

https://github.com/boostorg/config/blob/develop/include/boost/config/auto_link.hpp

[ghost]

Thanks. The original wxCONCAT from https://github.com/wxWidgets/wxWidgets/blob/master/include/wx/cpp.h reads:

#define wxCONCAT_HELPER(text, line) text ## line
#define wxCONCAT(x1, x2) wxCONCAT_HELPER(x1, x2)

Your test case abbreviates this:

#define wxCONCAT(text1, text2)       text1 ## text2

and that's the source of the problem. The indirection through wxCONCAT_HELPER seems useless at first glance, but it's there to satisfy the substitution rules discussed above.

Put the helper back in and simplecpp will generate the desired output:

extern int vc142_x64_lib ;

[SpareSimian]

Interesting. My test case was an attempt to simplify the wxWidgets case that was failing for me, and I simplified it too much.

[mgood7123]

in godbolt i get

<source>:16:61: error: pasting ")" and "_lib" does not give a valid preprocessing token
   16 | #define wxCONCAT3(x1, x2, x3)       wxCONCAT(wxCONCAT(x1, x2), x3)
      |                                                             ^
<source>:15:38: note: in definition of macro 'wxCONCAT'
   15 | #define wxCONCAT(text1, text2)       text1 ## text2
      |                                      ^~~~~
<source>:17:37: note: in expansion of macro 'wxCONCAT3'
   17 | #define wxSETUPH_PATH               wxCONCAT3(vc142, _x64, _lib)
      |                                     ^~~~~~~~~
<source>:18:12: note: in expansion of macro 'wxSETUPH_PATH'
   18 | extern int wxSETUPH_PATH;
      |            ^~~~~~~~~~~~~
Compiler returned: 1

could simplecpp's error message be modified to pasting ")" and "_lib" does not give a valid preprocessing token for invalid ## usage ?

maybe input:38: syntax error: failed to expand 'wxSETUPH_PATH', Pasting ")" and "_lib" does not give a valid preprocessing token when expanding 'wxCONCAT'.

[danmar]

@mgood7123 yes certainly that would be much better.

[ydamigos]

Hi all,

I have a similar issue. Below is the test code to reproduce it. cppcheck_test.c:

#define A_B_C                           0x1
#define A_ADDRESS                       0x00001000U
#define A                               ((uint32_t *) A_ADDRESS)
#define CONCAT(x, y, z)                 x ## _ ## y ## _ ## z
#define TEST_MACRO                      CONCAT(A, B, C)

int main(void)
{
        if (TEST_MACRO) {
                return 1;
        }
        return 0;
}

simplecpp returns the following error:

~$ simplecpp cppcheck_test.c 

cppcheck_test.c:4: syntax error: failed to expand 'TEST_MACRO', Invalid ## usage when expanding 'CONCAT': Unexpected token ')'

According to comment #225 (comment) I would expect A to be expanded after pasting.

GCC preprocessor returns what I expect:

~$ gcc -E cppcheck_test.c 

# 0 "cppcheck_test.c"
# 0 "<built-in>"
# 0 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 0 "<command-line>" 2
# 1 "cppcheck_test.c"

int main(void)
{
        if (0x1) {
                return 1;
        }
        return 0;
}

Which preprocessor follows the standard?

[mgood7123]

Obviously GCC/G++/Clang/Clang++ ALL follow the standards

SimpleCPP does not yet follow the standards and certainly CANNOT replace the cpp executable invoked during compilation to pre-process the input

[mgood7123]

Specifically, it's preprocessor is not on par with cpp's preprocessor

[datadiode]

Hi all,

I have a similar issue. Below is the test code to reproduce it. cppcheck_test.c:

#define A_B_C                           0x1
#define A_ADDRESS                       0x00001000U
#define A                               ((uint32_t *) A_ADDRESS)
#define CONCAT(x, y, z)                 x ## _ ## y ## _ ## z
#define TEST_MACRO                      CONCAT(A, B, C)

int main(void)
{
        if (TEST_MACRO) {
                return 1;
        }
        return 0;
}

simplecpp returns the following error:

~$ simplecpp cppcheck_test.c 

cppcheck_test.c:4: syntax error: failed to expand 'TEST_MACRO', Invalid ## usage when expanding 'CONCAT': Unexpected token ')'

According to comment #225 (comment) I would expect A to be expanded after pasting.

GCC preprocessor returns what I expect:

~$ gcc -E cppcheck_test.c 

# 0 "cppcheck_test.c"
# 0 "<built-in>"
# 0 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 0 "<command-line>" 2
# 1 "cppcheck_test.c"

int main(void)
{
        if (0x1) {
                return 1;
        }
        return 0;
}

Which preprocessor follows the standard?

This seems to be a regression from 8d5df36.