List of SCREAM log tasks

[AaronDonahue]

Currently the amount of information that is printed to one of the logs in a scream run is very minimal. In contrast, EAM prints quite a bit of information. This issue will detail a set of tasks that we may want to implement as we start running CIME based simulations.

• Write SCREAM output to the atm.log file rather than e3sm.log. Currently all SCREAM output uses just printf which seems to write to e3sm.log. Given that a lot of information is ATM specific, such as the set of HOMME namelist options used, we will want to create infrastructure to pass the ID of the atm.log file to SCREAM.
• Print a list of all Field Managed variables complete with short name, long name, units, dimensions
• Optional - also list if the FM variable is required, computed or both for a SCREAM run.
• Print the set of ATM processes used for this run, which order, what type of coupling.
• Print the timestep information. Such as the physics timestep, the start date, if there is any sub cycling.
• Print if this is an initial run or a restart run.
• Print the HOMME output to atm.log instead of e3sm.log. This is different then the task for scream in general because this task is focuses on passing the appropriate iulog to the homme code so it can handle output.
• Make an option to print the nstep and time information each step.
• Goes with the above, in EAM the nstep is accompanied by 3 basic values about global mass and energy. We could do something similar.
• Print the name of the initial condition file (or restart file) used.
• Print any output information, for example if output is requested, what fields are in it, what frequency of output, what type, and what the name of the file is.

[AaronDonahue]

I think that a lot of these tasks would be pretty easy to implement, provided we had a simple interface for printing to the atm.log file. I would propose creating a set of print_to_log routines that can be called anywhere in SCREAM that would accomplish just that.

[bartgol]

As I suggested in #1170, I think we should take advantage of the spdlog library support from EKAT that @pbosler put in a while ago. It has natural support for handling multi-rank output and log levels.

[AaronDonahue]

@bartgol, thats a great idea.

[PeterCaldwell]

I think @crterai told me that item 2 was accomplished by adding
Debug: Atmosphere DAG Verbosity Level: 5
to input.yaml???

[PeterCaldwell]

Item 7 and item 1 seem like the same thing (print all stuff to atm.log versus print homme stuff to atm log?). Also item 4 and 8 (both about printing info for each timestep)

[AaronDonahue]

I think @crterai told me that item 2 was accomplished by adding Debug: Atmosphere DAG Verbosity Level: 5 to input.yaml???

Kind of, but in a round about way. Doing that creates a .dot file which you can use to figure out what is in the field manager. But it isn't straightforward. I think we want something like we have in EAM where it lists them all as a list without the .dot syntax.

[bartgol]

Maybe I don't understand 3, but I think it's a bit complicated to write an "easy to read" dag to the atm log file...

[PeterCaldwell]

Regarding item 3: I like the idea of having to specify a verbosity flag before you start seeing stuff like the dag.

Luca - is your point that we should keep the current "write the dag to a .dot file" implementation as-is because plain text just isn't sufficient for describing the dag?

[bartgol]

I'm just saying that with tens of fields, and 5+ atm processes, the txt representation of the DAG is impossible to parse. One can only have a clue of what's going where by looking at the jpg/png/xyz picture you get processing the dot file.

Consider the dot file generated by our homme+physics test (I'm attaching it, in case you don't have it handy): even removing some of the xml syntax, it's hard to parse it to understand how the atm procs depend on each other...
scream_atm_dag.dot.txt

[PeterCaldwell]

Yeah, Chris T gave me a dot file recently and I found that Word (weirdly) parsed the text ok but it was still hard to read. Do you have an example of the jpg/png version of the dotfile handy which I could look at? It doesn't need to be recent - I just want to recall what these graphics look like.

[bartgol]

@PeterCaldwell Running dot -Tjpg -O scream_atm_dag.dot.txt produces the attached jpg file. It is still not super easy to visually parse (mostly b/c of the long field names lines, containing all the possible field info, and which can be made shorter by asking for a lower DAG verbosity), but definitely better than the text version.

[PeterCaldwell]

Oh, that's nice! Thanks. Should we be alarmed by the missing required fields?

[AaronDonahue]

@bartgol , sorry. I should have been more precise. In EAM during the init there is a list printed to the atm.log file of all of the variables that can be written to file. It's just a numbered list, it has no DAG properties. But it is useful if you want to create an output control file.

This idea came from talking to Chris T, who wanted a list of all the FM variables. The best I could give him was the DAG, but as the convo above points out it isn't very human readable.

[AaronDonahue]

@PeterCaldwell , I tried to edit the tasks to clarify the differences between 1 and 7, and 4 and 8.

[PeterCaldwell]

ok

[bartgol]

Oh, that's nice! Thanks. Should we be alarmed by the missing required fields?

No, though we need to improve it in v1.1. The missing fields are just fields that no other atm proc computes, and are in general just fields read from IC file and/or coming from surface coupling. We modified IC and SC logic quite a bit, in the last 6 months, but never really upgraded our DAG. We should find a way to add IC fields and SC fields in a way that they don't look as 'missing'. Another v1.1 task.

[AaronDonahue]

@bartgol , for reference, this is what I mean by copying what EAM has. Below is a snippet of a list that is printed to the atm.log.* file for every EAM run.

  ******* MASTER FIELD LIST *******
    1 NSTEP                            timestep            1 A  Model timestep
    2 PHIS                             m2/s2               1 I  Surface geopotential
    3 PS                               Pa                  1 A  Surface pressure
    4 T                                K                 128 A  Temperature
    5 U                                m/s               128 A  Zonal wind
    6 V                                m/s               128 A  Meridional wind
    7 Q                                kg/kg             128 A  Specific humidity
    8 TBP                              K                 128 A  Temperature (before physics)
    9 QBP                              kg/kg             128 A  Specific humidity  

[bartgol]

Is the list in EAM containing all fields that are in pbuf, pbuf+state, or other? Also, does it contain all fields or just those to be written to out/restart file?

[AaronDonahue]

In this list I believe EAM is just printing all fields which can be requested as output. So yes fields from the physics state and from PBUF. But not necessarily all fields from either of these sources. I think this list is populated using the IO routine addfld which registers a potential output. I propose that we make this list a bit more informative in SCREAM. We should list all variables in the field manager, and add a column for fields that will be added to the restart file.

[bartgol]

Well, we can print something like

   Name           Units            Grid/Layout         DataType           Dims           Goups
   
   horiz_winds     m/s        Phsics GLL:[COL,LEV]      double          [xyz,128]     state, restart
   ...

Not sure if all the fields/columns (overloaded words, I mean each entry of the row) are interesting, but it's info that we have, so we might as well...

[AaronDonahue]

Yes, agreed. We can set it up such that adding a column in the future - if desired - isn't too difficult. But this is the sort of information I think would be super useful to the user.