Support for AEC/CCSDS Compression (DRT 5.42) #449
Conversation
This commit modifies the top-level and source CMake configuration files to link to libaec to add support for AEC/CCSDS compression, as specified by GRIB2 DRT 5.42. NOTE: At this time, libaec v1.0.6 places cmake config files in /usr/lib64 on RHEL-based systems, so to build g2c, you need to add -DCMAKE_PREFIX_PATH=/usr/lib64 This commit references #424 [skip ci]
This commit allows for the ability to encode data into a GRIB2 message using AEC/CCSDS specified by DRS template 5.42. drstemplates.c now defines template 5.42 and its byte mapping. g2_addfield.c contains the proper logic to call aecpack.c New source file, aecpack.c performs the preparation of data for packing New source file, decenc_aec.c contains the encoding function, enc_aec Finally, grib2.h.in and grib2_int.h have been updated for AEC. This commit references #424
This commit references #424
This commit references #424
This commit builds out the capabilities of aecpack.c to handle data as float or double and establish g2c v2 API calls.. using jpcpack.c as a blueprint. This commit references #424
In the top-level CMake config, added aecunpack.c to sources for AEC
In utils/CMakeLists.txt, added ${AEC_LIBRARIES}
Added aec pack/unpack and encode/decode function definitions.
New source file, aecunpack.c contains functions for handling the unpacking of data. Floats and doubles are accommodated here. This source file mimmics jpcunpack.c Clean up of aecpack.c Adding dec_aec() to decenc_aec.c as well as documentation.
General clean up of aecpack.c. In the function, logic has been added to default values for the AEC/CCSDS parameters if they are set to zero in the input DRS template For nbits, added logic to round up the value to the next base 2 integer. For example, nbits set to 12 in the scaling part of the code will be rounded to 16.
Updated how nbits is handled from the template to the encoder. If nbits is set by the user, the value will be used and rounded to the next base 2 integer, otherwise, the value of nbits from scaling procedures will be used.
Repositioned code block that fills ctemp with ifld data to after the settings of the AEC compression parameters since nbits can be modified in that process.
Removed (char *) type setting for cpack in call to dec_aec(). This was not needed. Added check for return for call to dec_aec(). General cleanup of whitespace.
Also updated spack configuration for AEC.
Fixed typo in documentation of "data" to "date". [skip ci]
@edwardhartnett - When are you planning to release v1.8.0? |
Updated README.md to include information about LibAEC.
|
I'm planning on releasing 1.8.0 today or early next week... |
|
There were a couple of failed tests that I am working through. Does not seem major -- type mismatches. Going to push fixes soon. |
Redefined lcpack to g2int to match argument for aecpack().
|
This is happening on MacOS: /Users/runner/work/NCEPLIBS-g2c/NCEPLIBS-g2c/g2c/tests/tst_aec.c:9:10: fatal error: 'libaec.h' file not found Do you know how to install the AEC library on MacOS? |
|
Yes. I am working on the missing aec header for macOS now. I think it will be best to release early next week if the desire is to get AEC support in v1.8.0 |
Added logic to identify the root directory for libaec installed via Homebrew.
Using env var libaec_DIR now.
Also cleaned up a fprintf line in aecunpack.c that was tripping up another workflow.
Setting AEC_LIBRARIES and AEC_INCLUDE_DIR.
I realized that none of the 3rd party compression test include their respective library include files. This removes the need to have libaec.h.
|
Yay the macOS tests are passing with commit 16499e3 Now onto the Linux options tests |
Define lcpack as size_t when testing g2c_aecpackd()/g2c_aecunpackd().
Removing setting AEC_* CMake vars. This is not needed.
|
Next week is fine. I'm not ready yet... ;-) |
|
Also do you have any real-world examples of data files compressed with AEC? |
Added Findlibaec.cmake file to perform custom commands for finding libaec include and libraries files/paths. It seems that libaec on Linux does not provide cmake or pkg-config (.pc) files. Update other CMake config files to work with vars set from cmake/Findlibaec.cmake. These changes were tested on my Ubuntu 22.04 VM and build/test was successful.
Cleaning up some unsed variables in aecpack.c and aecunpack.c Added tests configs to Linux_options to round out the various test scenarios wrt Jasper, OpenJPEG, PNG, and AEC libraries.
Removed setting libaec_DIR env var -- no longer needed.
Added .DS_Store [skip ci]
|
What is happening with the CI here? How come nothing is running? |
|
Hmmm... I did put |
|
These 8 jobs do not have aec parameters, so something got messed up somewhere. Still looking... |
|
@edwardhartnett - I think somewhere in the GitHub CI, these 8 checks are "required", but these test do not exist anymore since I modified Linux_options and MacOS to include AEC parameters. In other words, these tests don't exist anymore, but they are required. |
|
OK, I forgot that you modified the tests. Stand by... |
|
🎉 Looks good! 🎉 |
|
@EricEngle-NOAA I'm going to merge this. Thanks very much for this contribution, it will benefit many other users as well. Could you please find some smallish real-world data file that uses this kind of packing, so we can have a test file on our FTP site? |
|
@edwardhartnett - It's been hard to find, but I can generate it. How many messages and what size limit should I consider? |
|
Some small file would be good, so just a couple of messages... |
Pull request for issue #424.
Changes in this pull request provide support for AEC/CCSDS compression in the g2c library as specified by DRT 5.42.
Tests have been created (tests/tst_aec.c) and are successful on my Fedora 38 VM (gcc 13.2)