QGC not showing metadata of added param

[tuloski]

Hi.
I was able to add a parameter to EKF2, and I wrote the correct metadata, however when I connect in QGC I see the new parameter but I don't see any description of the param, nor the other metadata info (min, max, bool etc). Also it shows up as reboot_required but I didn't put that in the metadata.
What's wrong? I didn't find any docs on this.

PS: I also tried "make parameters_metadata" before building my code without success.

[mhkabir]

You need to upload your new .px4 file through QGC "custom firmware" option for it load the Metadata.

[tuloski]

I did it.
And the .xml of the new build is correct.

[mhkabir]

@dagar ideas?

[dagar]

This is a problem with QGC where the metadata update seems to get stuck (I haven't dug into the root cause). Try flashing through QGC then restarting. If that doesn't work dig into the QGC config directory and delete the cached metadata, then try again.

We should figure out how to fix this properly. @dogmaphobic @DonLakeFlyer

[tuloski]

The best way would be to include some basic metadata in MAVlink protocol. It can be a big overhead but it would make it GCS and firmware agnostic.

[dagar]

Yes we'll probably need to go there (optionally), but there's also a middle ground of QGC fetching and properly caching the metadata from a server as well.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing as stale.

[julianoes]

@dagar is this still an issue?

[mhkabir]

Still an issue for me the last time I tried. Seems to be QGC side though.

[tuloski]

Yeah, still an issue for me (last time I tried was a couple months ago).

[sundxfansky]

same problem :(

[NuclearManD]

I am having this problem as well. I have QGC v3.5.4 running on Ubuntu 18.04.3. I tried these steps:

1. Close QGroundControl
2. delete ~/.config/QGroundControl.org/ParamCache and ~/.config/QGroundControl.org/ParameterFactMetaData.xml
3. Clean build (make clean in Firmware/)
4. build firmware (make px4_fmu_v5 in Firmware/)
5. Disconnect PX4
6. Open QGC and go to Firmware
7. Plug in PX4, check advanced, select Firmware/build/px4_fmu-v5_default/px4_fmu-v5_default.px4
8. Check my custom parameters

I can find the parameters, and they work, but they do not have any of the metadata I specified. I also checked Firmware/build/px4_fmu-v5_default/parameters.xml to confirm my added metadata is there, and it is. I shouldn't have to delete the cache either - this should just work. I would really like to see this solved.

[tuloski]

@DonLakeFlyer ?

[DonLakeFlyer]

Parameter metadata is versioned. QGC will use the interval version of metadata compiled into the code if the cached version number is equal to the internal version number.

There has been discussion a while back about changing such that equal version would use cache. But I wasn't quite sure of the implications of doing that. It might be ok.

[DonLakeFlyer]

There has been discussion a while back about changing such that equal version would use cache. But I wasn't quite sure of the implications of doing that. It might be ok.

@dagar Do you see any reason why this shouldn't be changed? It seems ok to me.

[LowOrbitIonCannon]

any Updates on this? When i flash m Firmware with QGC it doesn't show the Metadata, and also doesn't show new options. You have to manually put in the Number to change parameters to an unknown value.

[Spir4L]

tions of doing that. It might be ok.

So how do I change the version in order to pick my new metadata?

[Spir4L]

tions of doing that. It might be ok.

So how do I change the version in order to pick my new metadata?

The file that needs to be updated is src/lib/parameters/px4params/xmlout.py and the minor version has to be increased of 1 , in order to overrride the actual being used.

[NuclearManD]

Where are the parameters actually cached? I tried updating ~/.config/QGroundControl.org/ParameterFactMetaData.xml but QGroundControl did not show the new metadata (which I verified was present in the file)

I closed QGC before execution, and started QGC after executing this. It does seem that these two files have the exact same format, a diff from before showed that the only difference was my missing parameter.

[NuclearManD]

Ok, I'm figuring out how the flashing works in QGC, and how the parameter metadata is extracted, saved, and cached.

It seems that the parameter metadata is pulled out of the JSON in a .px4 file, then it is cached.

FirmwareImage.cc line 278 onward

QString parameterFilename = QGCApplication::cachedParameterMetaDataFile();
        QFile parameterFile(QGCApplication::cachedParameterMetaDataFile());

        if (parameterFile.open(QIODevice::WriteOnly | QIODevice::Truncate)) {
            qint64 bytesWritten = parameterFile.write(decompressedBytes);
            if (bytesWritten != decompressedBytes.count()) {
                emit statusMessage(tr("Write failed for parameter meta data file, error: %1").arg(parameterFile.errorString()));
                parameterFile.close();
                QFile::remove(parameterFilename);
            } else {
                parameterFile.close();
            }
        } else {
            emit statusMessage(tr("Unable to open parameter meta data file %1 for writing, error: %2").arg(parameterFilename, parameterFile.errorString()));
        }

        // Cache this file with the system
        ParameterManager::cacheMetaDataFile(parameterFilename, firmwareType);

The file is always saved to ParameterFactMetaData.xml
QGCApplication.cc line 909 onward

QString QGCApplication::cachedParameterMetaDataFile(void)
{
    QSettings settings;
    QDir parameterDir = QFileInfo(settings.fileName()).dir();
    return parameterDir.filePath(QStringLiteral("ParameterFactMetaData.xml"));
}

It then goes to the cache, but it does not get saved, because the version is exactly the same.

ParameterManager.cc line 1445 onward

    bool cacheNewFile = false;
    if (cacheHit.isEmpty()) {
        // No cache hits, store the new file
        cacheNewFile = true;
    } else if (cacheMajorVersion == newMajorVersion) {
        // Direct hit on major version in cache:
        //      Cache new file if newer minor version
        //      Also delete older cache file
        if (newMinorVersion > cacheMinorVersion) {
            cacheNewFile = true;
            QFile::remove(cacheHit);
        }
    } else {
        // Indirect hit in cache, store new file
        cacheNewFile = true;
    }

But where is this cache? The XML files are located in the settings directory - the result of QSettings().fileName() - which is the same location of our ParameterFactMetaData.xml. But there's nothing here other than ParamCache directory (does not contain metadata), ParameterFactMetaData.xml and QGroundControl.ini.

My QGC must be loading parameter metadata from somewhere else then because ParameterFactMetadata.xml has metadata I do not see in QGC.

Wouldn't it be better to update the cache if the minor versions were identical? This way QGC would always have the last flashed metadata if the cache had something different... I'm pretty sure this would be a one byte addition to ParameterManager.cc.

Edit: I did a find to see if there were any QGC directories I was missing, didn't find anything. I checked the directories in ~/.cache/

[LorenzMeier]

Could you send a pull request for that or do you feel it is too hard to add it yourself? I always encourage to get your hands dirty because you can help drive the process towards better UX also going forward. So the learning curve is worth it.

[DonLakeFlyer]

Wouldn't it be better to update the cache if the minor versions were identical?

There are multiple issues in on this which I can't seem to find right now. In the, there was discussion on whether this was safe or not. It wasn't fully understood if there were unknown gotchas by doing this. I don't think so, but not 100% sure. The problem relates to having a cached versions of crappy/incorrect dev build metadata getting stuck there causing problems when you try to go back to using regular builds. I think i may require something a little more sophisticated to be safe like using a different cache space for dev builds. @LorenzMeier What do you think?

[LorenzMeier]

I would propose to cache the "last flashed" variant and use that by default if its set. Copying it always from the .px4 file (wether that file was local or downloaded) and putting it in there. It will always be correct, because if you had a bad version then you most likely re-flashed something else to fix it.

[DonLakeFlyer]

It will always be correct, because if you had a bad version then you most likely re-flashed something else to fix it.

Sort of. But not if you have multiple vehicles. Say you have a vehicle you test new stuff with and you also have a regular vehicle you do normal work with. You fart around with you test vehicle, put it to the side and then use you normal vehicle. The metadata can be out of whack. Not sure if that is just too obscure a case though. I guess firmware devs doing metadata changes will need to be careful and know what they are going to some extent.

[LorenzMeier]

All of these have a unique ID that can be used to store the cache with.

[LorenzMeier]

Here: https://github.com/mavlink/qgroundcontrol/blob/master/src/VehicleSetup/Bootloader.h#L138
and that's the same as uid2 in this message: https://mavlink.io/en/messages/common.html#AUTOPILOT_VERSION

[DonLakeFlyer]

But that's not really "unique". It's just a board type. That's closer to this: mavlink/qgroundcontrol#6906 but different from it. We can have that discussion as well.

But for right now a simpler solution would be nicer which is whether it's safe to overwrite the cache if the versions are the same. I think that is 90% an ok thing to do. Since we are at the start of a new daily cycle we should be able to gather any problems with it up until a new stable which should be a little while.

[DonLakeFlyer]

The other issue I was thinking about is here: mavlink/qgroundcontrol#4485 to keep track of it.

[LorenzMeier]

Sorry, QGC is lacking the implementation, but the bootloader provides PROTO_GET_CHIP, which will give you the UID: https://github.com/PX4/Bootloader/blob/master/bl.c#L881

[DonLakeFlyer]

I'd also rather wait to implement vehicle specific metadata caching to use the new COMPONENT_INFORMATION vehicle download thing in discussion though. Which would supercede flash based metadata. Id' hope that will get worked out in this new stable cycle. I'll get back to that and provide more concrete details.

[DonLakeFlyer]

Until then I think overriting cache on equal version is safe which is what's in the pull referenced above.

[tuloski]

What about modifying PARAM_VALUE(#22) in Mavlink 2 to add metadata at mavlink protocol level?

[junwoo091400]

Isn't this fixed by mavlink/qgroundcontrol#8670 @DonLakeFlyer?

[DonLakeFlyer]

Yes as well as the new parameter metadata coming through COMPONENT_INFORMATION now as well.