G2C library support MET for large file size > 2GB #43
Comments
|
@BoiVuong-NOAA is this related to #28? |
|
No, It is not. The g2c will use int long for variables |
|
Do you have a pull request coming with the fix? |
|
No, I do not have a PR yet. I have it in the branch g2c_ver1.6.3 Do you want a PR now ? |
|
Yes, you should put that fix into develop. Generally, releases should be tagged off of the main branch. |
|
I will have a PR for the fix. I also have a unit_test too. Can you help me generate a cmake for unit_test ? |
|
Kyle, I created PR for g2c |
|
I don't see it |
|
Can you check again ? |
|
Does MET want to create or read a file larger than 2GB? Can we have a sample of a similar file that is less than 2GB? I can then construct a test... |
1 similar comment
|
Does MET want to create or read a file larger than 2GB? Can we have a sample of a similar file that is less than 2GB? I can then construct a test... |
|
MET wants to read a file larger than 2GB. I do not know where MET gets the file ? But, You can use the GFS master file which it is larger than 2 GB for testing. |
|
Is the GFS master file available on the web anywhere? |
|
I am not sure. You can check on NOMADS or You can request MET to create one for testing. |
|
Who at MET requested this change to g2c? |
|
You can contact Mallory. |
|
OK, hope you feel better! |
|
@malloryprow are you the one who requested this change to g2c? If so, can you provide me with a sample file? |
|
@edwardhartnett sample file at /gpfs/dell2/emc/verification/noscrub/emc.metplus/for_nco/met/met_test/case_5/fv3lam.t00z.namerica.f00.grib2 |
|
OK, is it really impossible to get me one of these files? ;-) @Hang-Lei-NOAA you tried to get the file off the dell2 and apparently could not becuase of a 2GB limit somewhere. Did you report that to the sysadmins? Seems like a serious problem that others will also encounter... @malloryprow is there any way to get such a file on hera where I can reach it? This is the last issue before a release for this library. If we get get me a file, and I can get it working, we can have a release by the end of the year... |
|
@edwardhartnett I sent an email on 12/9 the file is on Hera at /scratch1/NCEPDEV/stmp2/Mallory.Row/fv3lam.t00z.namerica.f00.grib2 |
|
OK, thanks, I'll take a look! |
|
Hi, Edward
I got it transferred to hera on 12/9. And send this to your email.
It is transferred into
/scratch1/NCEPDEV/global/Hang.Lei/fv3lam.t00z.namerica.f00.grib2
I reported it to helpdesk. They said that it was the network issue on 12/8.
But later I try through the hpss, the limitation is still there.
Hang
…On Tue, Dec 21, 2021 at 12:44 PM Edward Hartnett ***@***.***> wrote:
OK, thanks, I'll take a look!
—
Reply to this email directly, view it on GitHub
<#43 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKWSMFBJ4KTZQR6GY2MXMPTUSC4HTANCNFSM4XFFC2BA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
OK, I am circling around to this again. ;-) I have obtained my own very large grib2 file. I am pondering how to make it part of testing. Meanwhile, what g2c functions do I run on this file to demonstrate a >2GB problem? In fact, I don't think such a problem can exist in the C library, since we are using 64-bit ints for all arguments and internally for all ints. So why a 2 GB limit? When @kgerheiser looked into the 2GB problem, he found problems in the Fortran code, but not in the C code. So, any idea of a function in the C library that could look at a 2 GB file? |
|
I think that with #477 we will fully support > 2 GB files in g2c. |
Met failed to open file larger than > 2GB such as FV3 LAM file that has 4.5GB
The text was updated successfully, but these errors were encountered: