Can we export `compile_commands.json`? #3355

tan-wei · 2019-04-20T14:14:45Z

It is convenient for me to use cquery with compile_commands.json. It works quiet well for most of the c++ projects.

For CMake project, we can easily use CMAKE_EXPORT_COMPILE_COMMANDS option to export compile_commands.json
For Makefile project, compiledb can be used to generate a compile_commands.json

So can we export compile_commands.json when using SCons? Thank you very much!

The text was updated successfully, but these errors were encountered:

acmorrow · 2019-04-20T14:29:45Z

Hey -

Yes, this would be a great feature to have in mainline SCons. Meanwhile, In the MongoDB source code, we implemented our own site_tool to do it. It is (unlike most of the rest of that codebase) Apache 2 licensed:

Relevant snippets to demonstrate usage:

Installing the tool: https://github.com/mongodb/mongo/blob/master/SConstruct#L3628-L3630
Invoking the tool: https://github.com/mongodb/mongo/blob/master/SConstruct#L3768-L3769

The implementation could probably use some cleanup, but maybe it is close enough that we could work on getting it upstreamed someday?

tan-wei · 2019-04-21T04:58:36Z

Great!
Looking forward to using this feature in the future.

mwichmann · 2019-04-21T13:36:44Z

If you close this, how does it get taken care of? Instead maybe rebrand it as a feature request?

tan-wei · 2019-04-21T14:08:10Z

If you close this, how does it get taken care of? Instead maybe rebrand it as a feature request?

OK. I reopen this issue. After the feature is finished, then I close it. :-P
Thanks for your advice.

mwichmann · 2019-04-21T14:13:15Z

Bill doesn't favor bugs which did not have discussion on the mailing list, you might want to post a note there. Personally I have never heard of this compilation-database stuff, so it might be good to explain the benefits (and how they might fit into scons).

tan-wei · 2019-04-22T03:24:02Z

I'm sorry I haven't read the mailing list yet. Next time before I ask a question, I should search what I want in mailing list first.

For the background about compilation database, this page describes in detail. It comes from clang. So the cquery, which is based on clang, uses compile_commands.json to generate compilation database. It's a C/C++ language server, which can be used on many platforms and editors.
As this page describes, most of the build system can generate compile_commands.json, so it's convenience for editors which installed cquery if we can easily get the file.
For build system itself, it is not hard to export the file. If SCons can do this, I think it will benefit many people who use cquery.

bdbaddog · 2019-04-22T13:30:04Z

I'm sorry I haven't read the mailing list yet. Next time before I ask a question, I should search what I want in mailing list first.

For the background about compilation database, this page describes in detail. It comes from clang. So the cquery, which is based on clang, uses compile_commands.json to generate compilation database. It's a C/C++ language server, which can be used on many platforms and editors.
As this page describes, most of the build system can generate compile_commands.json, so it's convenience for editors which installed cquery if we can easily get the file.
For build system itself, it is not hard to export the file. If SCons can do this, I think it will benefit many people who use cquery.

Did you not see the request in the bug template that you bring your issue there first? (Or is that not working?)

tan-wei · 2019-04-22T13:58:53Z

I'm sorry I haven't read the mailing list yet. Next time before I ask a question, I should search what I want in mailing list first.
For the background about compilation database, this page describes in detail. It comes from clang. So the cquery, which is based on clang, uses compile_commands.json to generate compilation database. It's a C/C++ language server, which can be used on many platforms and editors.
As this page describes, most of the build system can generate compile_commands.json, so it's convenience for editors which installed cquery if we can easily get the file.
For build system itself, it is not hard to export the file. If SCons can do this, I think it will benefit many people who use cquery.

Did you not see the request in the bug template that you bring your issue there first? (Or is that not working?)

I'm sorry, but I didn't see the contents because I accidentally erased them when I entered the issue on my mobile phone. So how should we deal with this issue? Sorry again for the trouble.
Hope you have a nice day.

bdbaddog · 2019-04-22T15:14:23Z

I'll mark it an enhancement.
If you need it now, I'd suggest you try the code @acmorrow pointed to.

tan-wei · 2019-04-23T09:02:02Z

I'll mark it an enhancement.
If you need it now, I'd suggest you try the code @acmorrow pointed to.

Thanks very much. Hope you have a good day.

RevoluPowered · 2019-10-14T09:43:24Z

We are working on Godot and are interested in getting this supported.

We're speaking about scons at GodotCon and would really appreciate a discussion to implement this in the future.

Can you please let us know if someone at scons could assist us?

Thanks

mwichmann · 2019-10-14T13:30:26Z

If you drop a comment onto the scons-dev mailing list we can see where it goes. Meanwhile, can you make use of the "external" implementation referenced above, as a proof of concept if nothing else? Since you make the third project that has some interest in this it seems like there's an audience.

acmorrow · 2019-10-14T14:39:11Z

@mwichmann - We would be happy to see about upstreaming our implementation. It can surely be improved but it works for us pretty well and is reasonably fast.

@RevoluPowered - the version in the MongoDB library is Apache 2 licensed so you you may well be able to just use it: https://github.com/mongodb/mongo/blob/master/site_scons/site_tools/compilation_db.py

RevoluPowered · 2019-10-15T08:38:25Z

I would really appreciate that being upstreamed @acmorrow

if that could be arranged that would be great.

I'll pop a email on the scons mailinglist too when we are finished

RevoluPowered · 2019-11-16T11:29:27Z

@acmorrow we updated the script to handle xcode and objectivec files automatically.

bdbaddog · 2019-11-16T16:26:22Z

@RevoluPowered @acmorrow - can you work together on a pull request. Please don't forget to update CHANGES.txt, add tests, and update docs.

RevoluPowered · 2020-02-11T08:29:26Z

There is currently a limitation with this, which I have not solved, msvc compiler on our toolchain is not able to generate the compilation database. msvc builders seem to behave differently and not work with this custom tool.

not getting any errors as such just an empty compilation database.

Clang and gcc are working fine with little overhead too. (they even work fine on windows with this, seems literally just msvc which has this issue)

any ideas @acmorrow or @bdbaddog?

bdbaddog · 2020-02-11T16:40:32Z

@RevoluPowered - a little more information would be useful. Do you get zero output in compilation db? Can you provide a small reproducer for that?

acmorrow · 2020-02-11T16:50:16Z

@RevoluPowered - Yes, more details are definitely necessary. I just tried this out locally on a VS 2019 machine in the MongoDB tree and I definitely get a populated compile_commands.json.

ivankravets · 2020-02-11T16:55:21Z

@acmorrow thanks so much for your work. We also at @platformio are interested in this feature. Currently, we just copied the source file from mongo repo and modified it a little bit https://github.com/platformio/platformio-core/blob/develop/platformio/builder/tools/compilation_db.py

Is it possible to make it configurable without the original source code modification? For example, to have support for Assembly files, see https://github.com/platformio/platformio-core/blob/develop/platformio/builder/tools/compilation_db.py#L170

Thanks in advance!

RevoluPowered · 2020-02-12T20:33:20Z

@acmorrow thanks so much for your work. We also at @platformio are interested in this feature. Currently, we just copied the source file from mongo repo and modified it a little bit https://github.com/platformio/platformio-core/blob/develop/platformio/builder/tools/compilation_db.py

Is it possible to make it configurable without the original source code modification? For example, to have support for Assembly files, see https://github.com/platformio/platformio-core/blob/develop/platformio/builder/tools/compilation_db.py#L170

Thanks in advance!

I believe you possibly could do this with our version, we modified it so that objective-c files are added to the compilation database automatically on OSX, and when you do not have the tool available it will not be in the file. This change should also apply to assembly files. (I could be wrong, but it may work and worth testing).

You could try this and it may just work, it might not though.

The compilation database is useful for code browsing. The actual change is copied from SCons/scons#3355

lucmann · 2020-03-17T06:18:26Z

The implementation could probably use some cleanup, but maybe it is close enough that we could work on getting it upstreamed someday?

@acmorrow what do you mean by 'some cleanup'? I tried to apply the compilation_db.py to my Scons project just like that is done in Mongo project. But I got nothing but only empty bracket in compile_commands.json. I guess I missed something. I do the same with @vmiheer's commit db616e3 except that my source tree has a slightly difference. It's like
src/scripts/sconstruct
src/scripts/site_scons/site_tools/compilation_db.py

the scons command is like
scons -f scripts/sconstruct profile=x86_64 -j8

In addition, I added some prints to compilation_db.py for debugging so that I confirm that the compilation_db tool is triggered but no one entry is generated during the compliation.

RevoluPowered · 2020-03-22T15:49:55Z

I have got no time to add support for this to scons.

Can someone else create the PR?

pinetr2e · 2020-04-10T11:27:13Z

I wrote a pip package to generate compile_commands.json: https://github.com/pinetr2e/scons-compiledb

It might be useful until the SCons mainline supports it or for the old version of SCons.

bdbaddog · 2020-05-12T21:11:20Z

All. I've started work on creating a PR based on MongoDB's latest master branch as of today.
@ivankravets - can you post a diff between MongoDB's implementation and platformIO's?

ivankravets · 2020-05-12T21:18:13Z

Hi @bdbaddog, see https://www.diffchecker.com/s6z0cQJI

bdbaddog · 2020-05-12T21:22:31Z

@ivankravets so to summarize:

Add support for assembly files
resolve absolute path of compilers for (c,cxx, assembly)
change compilation db entries to use absolute path for files

Anything else?
Why use absolute paths for compilers/files?

ivankravets · 2020-05-12T21:26:26Z

@bdbaddog you are right!

Why use absolute paths for compilers/files?

These changes come from our community. People use PlatformIO's Compilation database compile_commands.json for IDEs and editors. They generate compile_commands.json and use it with Vim and Emacs. Absolute paths are required for Vim/Emacs plugins to provide code completion, navigation, and linting.

bdbaddog · 2020-05-12T21:28:57Z

@ivankravets - WRT assembly files, any reason to only define for static objects? (why exclude shared?)

ivankravets · 2020-05-12T21:32:10Z

@bdbaddog it is our specific case, we do not use $SHASCOM. So, you can implement both.

bdbaddog · 2020-05-12T21:34:09Z

@ivankravets - lastly. Your copy of the code is Apache Licensed. MongoDB recently relicensed the code to MIT to make it compatible with SCons.

Can you do that for your copy of compilation_db?
(Which hopefully once this is released (likely in 4.0.0) you'll be able to drop from your tree?)

ivankravets · 2020-05-12T21:36:14Z

I also found a more detailed answer for absolute paths => platformio/platformio-core@f81b0b2

Can you do that for your copy of compilation_db?

What should be done from our side? The main reason for that code that it is compatible with Python 2 & 3. We would be happy to switch to native SCons compilation_db.

P.S: We also plan to drop Python 2 next year. We plan to show warning to our customers soon.

bdbaddog · 2020-05-12T21:37:51Z

@ivankravets - I beleive if you change the license header in your copy of compilation_db.py to match what's in the current master on mongodb's repo:

https://github.com/mongodb/mongo/blob/master/site_scons/site_tools/compilation_db.py#L1

That should take care of the license issue.

ivankravets · 2020-05-12T21:48:37Z

Do you mean this platformio/platformio-core@099e3c7 ?

bdbaddog · 2020-05-12T21:49:46Z

@ivankravets - Yup. That should do it. Thanks!

bdbaddog · 2020-05-18T20:04:32Z

FYI - I'm working on a pull request for this now.

bdbaddog · 2020-05-18T23:09:26Z

@ivankravets - thus far I've added an env variable to enable/disable absolute paths for target/source. I'm not sure if doing the same for tools is safe as if the path has a space in it, that can cause issues with the current logic which spawns processes.

For tools (cc,cxx,as) it seems like if the user want absolute paths to them they should do that outside of the CompilationDatabase().. Thoughts?

ivankravets · 2020-05-19T09:21:52Z

@bdbaddog I think you are right. SCons is neutral to any toolchain and end-user or integrator is responsible for PATH or full resolving. We can do the next on our side when the user calls compiledb target:

Clone ENV
Resolve toolchains paths
Call SCons CompilationDatabase which will finally generate JSON.

bdbaddog · 2020-05-19T16:50:13Z

@ivankravets - o.k. in that case I'll push a PR today. Tests and docs not yet complete, but perhaps you can give the code a try?

ivankravets · 2020-05-22T13:32:31Z

@bdbaddog, sorry for the delay. I've finally got time to switch PlatformIO to your SCons dev-version. A default CompilationDatabase tool works well. There is a problem when I try to enable env["COMPILATIONDB_USE_ABSPATH"] = True. See screenshot:

I see generated JSON but a path to compiler is not absolute.

[
    {
        "command": "avr-g++ -o /private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/CDC.cpp.o -c -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -Os -Wall -ffunction-sections -fdata-sections -flto -mmcu=atmega328p -DPLATFORMIO=40304 -DARDUINO_AVR_UNO -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -I/Users/ikravets/.platformio/packages/framework-arduino-avr/cores/arduino -I/Users/ikravets/.platformio/packages/framework-arduino-avr/variants/standard /Users/ikravets/.platformio/packages/framework-arduino-avr/cores/arduino/CDC.cpp",
        "directory": "/Users/ikravets/Projects/PlatformIO/Development/core/examples/wiring-blink",
        "file": "/private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/CDC.cpp",
        "target": "/private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/CDC.cpp.o"
    },
    {
        "command": "avr-g++ -o /private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/HardwareSerial.cpp.o -c -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -Os -Wall -ffunction-sections -fdata-sections -flto -mmcu=atmega328p -DPLATFORMIO=40304 -DARDUINO_AVR_UNO -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -I/Users/ikravets/.platformio/packages/framework-arduino-avr/cores/arduino -I/Users/ikravets/.platformio/packages/framework-arduino-avr/variants/standard /Users/ikravets/.platformio/packages/framework-arduino-avr/cores/arduino/HardwareSerial.cpp",
        "directory": "/Users/ikravets/Projects/PlatformIO/Development/core/examples/wiring-blink",
        "file": "/private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/HardwareSerial.cpp",
        "target": "/private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/HardwareSerial.cpp.o"
    },
    {
        "command": "avr-g++ -o /private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/HardwareSerial0.cpp.o -c -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -Os -Wall -ffunction-sections -fdata-sections -flto -mmcu=atmega328p -DPLATFORMIO=40304 -DARDUINO_AVR_UNO -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -I/Users/ikravets/.platformio/packages/framework-arduino-avr/cores/arduino -I/Users/ikravets/.platformio/packages/framework-arduino-avr/variants/standard /Users/ikravets/.platformio/packages/framework-arduino-avr/cores/arduino/HardwareSerial0.cpp",
        "directory": "/Users/ikravets/Projects/PlatformIO/Development/core/examples/wiring-blink",
        "file": "/private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/HardwareSerial0.cpp",
        "target": "/private/tmp/pio-workspaces/wiring-blink-632516cf26/build/uno/FrameworkArduino/HardwareSerial0.cpp.o"
    }
]

avr-g++ is in SConss ENV PATH.

bdbaddog · 2020-05-22T15:06:41Z

@ivankravets - re tool abs path. As we discussed above. Exercise left for user. Actually would have been better to comment on the Pull request.. #3662

Add compilation db for use by clang tools and cquery (fix issue #3355)

tan-wei closed this as completed Apr 21, 2019

tan-wei reopened this Apr 21, 2019

bdbaddog added the enhancement label Apr 22, 2019

bdbaddog changed the title ~~[Question] Can we export compile_commands.json?~~ Can we export compile_commands.json? Apr 25, 2019

RevoluPowered mentioned this issue Oct 15, 2019

Compilation database support - clang, gcc, mingw godotengine/godot#32848

Merged

vmiheer added a commit to vmiheer/maestro that referenced this issue Feb 18, 2020

Create compilation database

db616e3

The compilation database is useful for code browsing. The actual change is copied from SCons/scons#3355

vmiheer mentioned this issue Feb 18, 2020

Enable cmake build maestro-project/maestro#5

Merged

bdbaddog self-assigned this May 11, 2020

bdbaddog mentioned this issue May 19, 2020

Add compilation db for use by clang tools and cquery (fix issue #3355) #3662

Merged

3 tasks

bdbaddog closed this as completed in #3662 Jun 8, 2020

bdbaddog added a commit that referenced this issue Jun 8, 2020

Merge pull request #3662 from bdbaddog/add_compilation_db

ce32ce7

Add compilation db for use by clang tools and cquery (fix issue #3355)

dogtopus mentioned this issue Mar 14, 2023

Improve vscode clangd experience flipperdevices/flipperzero-firmware#2431

Merged

3 tasks

Can we export compile_commands.json? #3355

Can we export compile_commands.json? #3355

Comments

tan-wei commented Apr 20, 2019 • edited by bdbaddog Loading

acmorrow commented Apr 20, 2019

tan-wei commented Apr 21, 2019

mwichmann commented Apr 21, 2019

tan-wei commented Apr 21, 2019

mwichmann commented Apr 21, 2019

tan-wei commented Apr 22, 2019 • edited Loading

bdbaddog commented Apr 22, 2019

tan-wei commented Apr 22, 2019

bdbaddog commented Apr 22, 2019

tan-wei commented Apr 23, 2019

RevoluPowered commented Oct 14, 2019

mwichmann commented Oct 14, 2019

acmorrow commented Oct 14, 2019

RevoluPowered commented Oct 15, 2019

RevoluPowered commented Nov 16, 2019 • edited Loading

bdbaddog commented Nov 16, 2019

RevoluPowered commented Feb 11, 2020 • edited Loading

bdbaddog commented Feb 11, 2020

acmorrow commented Feb 11, 2020

ivankravets commented Feb 11, 2020

RevoluPowered commented Feb 12, 2020

lucmann commented Mar 17, 2020

RevoluPowered commented Mar 22, 2020

pinetr2e commented Apr 10, 2020 • edited Loading

bdbaddog commented May 12, 2020

ivankravets commented May 12, 2020

bdbaddog commented May 12, 2020

ivankravets commented May 12, 2020

bdbaddog commented May 12, 2020

ivankravets commented May 12, 2020

bdbaddog commented May 12, 2020

ivankravets commented May 12, 2020 • edited Loading

bdbaddog commented May 12, 2020

ivankravets commented May 12, 2020

bdbaddog commented May 12, 2020

bdbaddog commented May 18, 2020

bdbaddog commented May 18, 2020

ivankravets commented May 19, 2020

bdbaddog commented May 19, 2020

ivankravets commented May 22, 2020

bdbaddog commented May 22, 2020

Can we export `compile_commands.json`? #3355

Can we export `compile_commands.json`? #3355

tan-wei commented Apr 20, 2019 •

edited by bdbaddog

Loading

tan-wei commented Apr 22, 2019 •

edited

Loading

RevoluPowered commented Nov 16, 2019 •

edited

Loading

RevoluPowered commented Feb 11, 2020 •

edited

Loading

pinetr2e commented Apr 10, 2020 •

edited

Loading

ivankravets commented May 12, 2020 •

edited

Loading