Offline driver initialisation refactor continued (cable_user%MetType) #498

SeanBryan51 · 2024-11-27T23:46:54Z

This change further consolidates the initialisation logic within the offline drivers (in particular logic related to cable_user%MetType) to remove code duplication and improve code clarity across each of the drivers (see #425, #440, #469 and #472).

As part of the refactoring changes, the cable_user%MetType = 'gpgs' namelist parameter is deprecated as it is equivalent to setting cable_user%MetType = 'gswp' and leaps = .TRUE..

Type of change

Code refactor
New or updated documentation

Checklist

The new content is accessible and located in the appropriate section.
I have checked that links are valid and point to the intended content.
I have checked my code/text and corrected any misspellings

📚 Documentation preview 📚: https://cable--498.org.readthedocs.build/en/498/

SeanBryan51 · 2024-11-28T00:51:03Z

I won't have time to move the remaining occurrences of CALL1 to a shared initialisation routine before going on leave next week but I have noted them below:

CABLE/src/offline/cable_serial.F90

Lines 367 to 444 in 7e6a8bb

    
           IF ( CALL1 ) THEN 
        
              IF (cable_user%POPLUC) THEN 
        
                 CALL LUC_EXPT_INIT (LUC_EXPT) 
        
              ENDIF 
        
              ! vh_js ! 
        
              CALL load_parameters( met, air, ssnow, veg,climate,bgc,           & 
        
                   soil, canopy, rough, rad, sum_flux,                   & 
        
                   bal, logn, vegparmnew, casabiome, casapool,           & 
        
                   casaflux, sum_casapool, sum_casaflux, & 
        
                   casamet, casabal, phen, POP, spinup,        & 
        
                   CEMSOIL, CTFRZ, LUC_EXPT, POPLUC ) 
        
              IF (check%ranges /= NO_CHECK) THEN 
        
                  WRITE (*, *) "Checking parameter ranges" 
        
                  CALL constant_check_range(soil, veg, 0, met) 
        
              END IF 
        
              IF ( CABLE_USER%POPLUC .AND. TRIM(CABLE_USER%POPLUC_RunType) .EQ. 'static') & 
        
                   CABLE_USER%POPLUC= .FALSE. 
        
              ! Open output file: 
        
              IF (.NOT.CASAONLY) THEN 
        
                 IF ( TRIM(filename%out) .EQ. '' ) THEN 
        
                    IF ( CABLE_USER%YEARSTART .GT. 0 ) THEN 
        
                       WRITE( dum, FMT="(I4,'_',I4)")CABLE_USER%YEARSTART, & 
        
                            CABLE_USER%YEAREND 
        
                       filename%out = TRIM(filename%path)//'/'//& 
        
                            TRIM(cable_user%RunIden)//'_'//& 
        
                            TRIM(dum)//'_cable_out.nc' 
        
                    ELSE 
        
                       filename%out = TRIM(filename%path)//'/'//& 
        
                            TRIM(cable_user%RunIden)//'_cable_out.nc' 
        
                    ENDIF 
        
                 ENDIF 
        
                 IF (RRRR.EQ.1) THEN 
        
                    CALL nullify_write() ! nullify pointers 
        
                    CALL open_output_file( dels, soil, veg, bgc, rough, met) 
        
                 ENDIF 
        
              ENDIF 
        
              ssnow%otss_0 = ssnow%tgg(:,1) 
        
              ssnow%otss = ssnow%tgg(:,1) 
        
              ssnow%tss = ssnow%tgg(:,1) 
        
              canopy%fes_cor = 0. 
        
              canopy%fhs_cor = 0. 
        
              met%ofsd = 0.1 
        
              CALL zero_sum_casa(sum_casapool, sum_casaflux) 
        
              count_sum_casa = 0 
        
              IF (cable_user%call_climate) CALL climate_init ( climate, mp, ktauday ) 
        
              IF (cable_user%call_climate .AND.(.NOT.cable_user%climate_fromzero)) & 
        
                   CALL READ_CLIMATE_RESTART_NC (climate, ktauday) 
        
              spinConv = .FALSE. ! initialise spinup convergence variable 
        
              IF (.NOT.spinup)  spinConv=.TRUE. 
        
              IF( icycle>0 .AND. spincasa) THEN 
        
                 PRINT *, 'EXT spincasacnp enabled with mloop= ', mloop 
        
                 CALL spincasacnp(dels,kstart,kend,mloop,veg,soil,casabiome,casapool, & 
        
                      casaflux,casamet,casabal,phen,POP,climate,LALLOC) 
        
                 SPINon = .FALSE. 
        
                 SPINconv = .FALSE. 
        
              ELSEIF ( casaonly .AND. (.NOT. spincasa) ) THEN !.AND. cable_user%popluc) THEN 
        
                 CALL CASAONLY_LUC(dels,kstart,kend,veg,soil,casabiome,casapool, & 
        
                      casaflux,casamet,casabal,phen,POP,climate,LALLOC, LUC_EXPT, POPLUC, & 
        
                      sum_casapool, sum_casaflux) 
        
                 SPINon = .FALSE. 
        
                 SPINconv = .FALSE. 
        
                 ktau = kend 
        
              ENDIF 
        
           ENDIF ! CALL 1

CABLE/src/offline/cable_serial.F90

Lines 483 to 487 in 7e6a8bb

    
           IF (( .NOT. CASAONLY ) .OR. (CASAONLY.AND.CALL1))  THEN 
        
              CALL PLUME_MIP_GET_MET(PLUME, MET, YYYY, ktau, kend, & 
        
                   (YYYY.EQ.CABLE_USER%YearEnd .AND. ktau.EQ.kend)) 
        
           ENDIF

CABLE/src/offline/cable_serial.F90

Lines 490 to 494 in 7e6a8bb

    
           IF (( .NOT. CASAONLY ).OR. (CASAONLY.AND.CALL1))  THEN 
        
              CALL CRU_GET_SUBDIURNAL_MET(CRU, met, & 
        
                   YYYY, ktau, kend, & 
        
                   YYYY.EQ.CABLE_USER%YearEnd) 
        
           ENDIF

CABLE/src/offline/cable_mpimaster.F90

Lines 384 to 420 in 7e6a8bb

    
           IF ( CALL1 ) THEN 
        
              IF (cable_user%POPLUC) THEN 
        
                 CALL LUC_EXPT_INIT (LUC_EXPT) 
        
              ENDIF 
        
              ! vh_js ! 
        
              CALL load_parameters( met, air, ssnow, veg,climate,bgc, & 
        
                   soil, canopy, rough, rad, sum_flux, & 
        
                   bal, logn, vegparmnew, casabiome, casapool, & 
        
                   casaflux, sum_casapool, sum_casaflux, & 
        
                   casamet, casabal, phen, POP, spinup, & 
        
                   CEMSOIL, CTFRZ, LUC_EXPT, POPLUC ) 
        
              IF (CABLE_USER%POPLUC .AND. TRIM(CABLE_USER%POPLUC_RunType) .EQ. 'static') & 
        
                   CABLE_USER%POPLUC= .FALSE. 
        
              ssnow%otss_0 = ssnow%tgg(:,1) 
        
              ssnow%otss = ssnow%tgg(:,1) 
        
              ssnow%tss = ssnow%tgg(:,1) 
        
              canopy%fes_cor = 0. 
        
              canopy%fhs_cor = 0. 
        
              met%ofsd = 0.1 
        
              IF (.NOT.spinup) spinConv=.TRUE. 
        
              ! MPI: above was standard serial code 
        
              ! now it's time to initialize the workers 
        
              ! MPI: bcast to workers so that they don't need to open the met 
        
              ! file themselves 
        
              CALL MPI_Bcast (dels, 1, MPI_REAL, 0, comm, ierr) 
        
           ENDIF

CABLE/src/offline/cable_mpimaster.F90

Lines 424 to 526 in 7e6a8bb

    
           IF ( CALL1 ) THEN 
        
              ! MPI: need to know extents before creating datatypes 
        
              CALL find_extents 
        
              ! MPI: calculate and broadcast landpoint decomposition to the workers 
        
              CALL master_decomp(comm, mland) 
        
              ! MPI: set up stuff for new irecv isend code that separates completion 
        
              ! from posting of requests 
        
              ! wnp is set in master_decomp above 
        
              ALLOCATE (inp_req(wnp)) 
        
              ALLOCATE (inp_stats(MPI_STATUS_SIZE, wnp)) 
        
              ALLOCATE (recv_req(wnp)) 
        
              ALLOCATE (recv_stats(MPI_STATUS_SIZE, wnp)) 
        
              CALL MPI_Comm_dup (comm, icomm, ierr) 
        
              CALL MPI_Comm_dup (comm, ocomm, ierr) 
        
              ! MPI: data set in load_parameter is now scattered out to the 
        
              ! workers 
        
              CALL master_cable_params(comm, met,air,ssnow,veg,bgc,soil,canopy,& 
        
                   &                         rough,rad,sum_flux,bal) 
        
              IF (cable_user%call_climate) THEN 
        
                 CALL master_climate_types(comm, climate, ktauday) 
        
              ENDIF 
        
              ! MPI: mvtype and mstype send out here instead of inside master_casa_params 
        
              !      so that old CABLE carbon module can use them. (BP May 2013) 
        
              CALL MPI_Bcast (mvtype, 1, MPI_INTEGER, 0, comm, ierr) 
        
              CALL MPI_Bcast (mstype, 1, MPI_INTEGER, 0, comm, ierr) 
        
              ! MPI: casa parameters scattered only if cnp module is active 
        
              IF (icycle>0) THEN 
        
                 ! MPI: 
        
                 CALL master_casa_params (comm,casabiome,casapool,casaflux,casamet,& 
        
                      &                        casabal,phen) 
        
                 IF ( CABLE_USER%CALL_POP ) CALL master_pop_types (comm,pop) 
        
              END IF 
        
              ! MPI: allocate read ahead buffers for input met and veg data 
        
              CALL alloc_cbm_var (imet, mp) 
        
              CALL alloc_cbm_var (iveg, mp) 
        
              ! MPI: create inp_t types to scatter input data to the workers 
        
              ! at the start of every timestep 
        
              !CALL master_intypes (comm,met,veg) 
        
              ! for read ahead use the new variables 
        
              CALL master_intypes (comm,imet,iveg) 
        
              ! MPI: create recv_t types to receive results from the workers 
        
              ! at the end of every timestep 
        
              CALL master_outtypes (comm,met,canopy,ssnow,rad,bal,air,soil,veg) 
        
              ! MPI: create type for receiving casa results 
        
              ! only if cnp module is active 
        
              IF (icycle>0) THEN 
        
                 CALL master_casa_types (comm, casapool, casaflux, & 
        
                      casamet, casabal, phen) 
        
                 IF ( CABLE_USER%CASA_DUMP_READ .OR. CABLE_USER%CASA_DUMP_WRITE ) & 
        
                      CALL master_casa_dump_types( comm, casamet, casaflux, phen ) 
        
                 WRITE(*,*) 'cable_mpimaster, POPLUC: ' ,  CABLE_USER%POPLUC 
        
                 IF ( CABLE_USER%POPLUC ) & 
        
                      CALL master_casa_LUC_types( comm, casapool, casabal) 
        
              END IF 
        
              ! MPI: create type to send restart data back to the master 
        
              ! only if restart file is to be created 
        
              IF(output%restart) THEN 
        
                 CALL master_restart_types (comm, canopy, air) 
        
              END IF 
        
              !  CALL zero_sum_casa(sum_casapool, sum_casaflux) 
        
              !  count_sum_casa = 0 
        
              ! CALL master_sumcasa_types(comm, sum_casapool, sum_casaflux) 
        
              IF( icycle>0 .AND. spincasa) THEN 
        
                 PRINT *, 'EXT spincasacnp enabled with mloop= ', mloop, dels, kstart, kend 
        
                 CALL master_spincasacnp(dels,kstart,kend,mloop,veg,soil,casabiome,casapool, & 
        
                      casaflux,casamet,casabal,phen,POP,climate,icomm, ocomm) 
        
                 SPINconv = .FALSE. 
        
                 CASAONLY                   = .TRUE. 
        
                 ktau_gl = 0 
        
                 ktau = 0 
        
              ELSEIF ( casaonly .AND. (.NOT. spincasa) .AND. cable_user%popluc) THEN 
        
                 CALL master_CASAONLY_LUC(dels,kstart,kend,veg,casabiome,casapool, & 
        
                      casaflux,casamet,casabal,phen,POP,climate,LUC_EXPT, POPLUC, & 
        
                      icomm, ocomm) 
        
                 SPINconv = .FALSE. 
        
                 ktau_gl = 0 
        
                 ktau = 0 
        
              ENDIF 
        
              ! MPI: mostly original serial code follows... 
        
           ENDIF ! CALL1

CABLE/src/offline/cable_mpiworker.F90

Lines 278 to 285 in 7e6a8bb

    
           IF ( CALL1 ) THEN 
        
              IF (.NOT.spinup) spinConv=.TRUE. 
        
              ! MPI: bcast to workers so that they don't need to open the met 
        
              ! file themselves 
        
              CALL MPI_Bcast (dels, 1, MPI_REAL, 0, comm, ierr) 
        
           ENDIF

CABLE/src/offline/cable_mpiworker.F90

Lines 291 to 419 in 7e6a8bb

    
           IF ( CALL1 ) THEN 
        
              ! MPI: need to know extents before creating datatypes 
        
              CALL find_extents 
        
              ! MPI: receive decomposition info from the master 
        
              CALL worker_decomp(comm) 
        
              ! MPI: in overlap version sends and receives occur on separate comms 
        
              CALL MPI_Comm_dup (comm, icomm, ierr) 
        
              CALL MPI_Comm_dup (comm, ocomm, ierr) 
        
              ! MPI: data set in load_parameter is now received from 
        
              ! the master 
        
              CALL worker_cable_params(comm, met,air,ssnow,veg,bgc,soil,canopy,& 
        
                   &                        rough,rad,sum_flux,bal) 
        
              !mrd561 debug 
        
              WRITE(logn,*) ' ssat_vec min',MINVAL(soil%ssat_vec),MINLOC(soil%ssat_vec) 
        
              WRITE(logn,*) ' sfc_vec min',MINVAL(soil%sfc_vec),MINLOC(soil%sfc_vec) 
        
              WRITE(logn,*) ' wb min',MINVAL(ssnow%wb),MINLOC(ssnow%wb) 
        
              CALL flush(logn) 
        
              IF (check%ranges /= NO_CHECK) THEN 
        
                WRITE (logn, *) "Checking parameter ranges" 
        
                CALL constant_check_range(soil, veg, 0, met) 
        
              END IF 
        
              IF (cable_user%call_climate) THEN 
        
                 CALL worker_climate_types(comm, climate, ktauday ) 
        
              ENDIF 
        
              ! MPI: mvtype and mstype send out here instead of inside worker_casa_params 
        
              !      so that old CABLE carbon module can use them. (BP May 2013) 
        
              CALL MPI_Bcast (mvtype, 1, MPI_INTEGER, 0, comm, ierr) 
        
              CALL MPI_Bcast (mstype, 1, MPI_INTEGER, 0, comm, ierr) 
        
              ! MPI: casa parameters received only if cnp module is active 
        
              IF (icycle>0) THEN 
        
                 CALL worker_casa_params (comm,casabiome,casapool,casaflux,casamet,& 
        
                      &                        casabal,phen) 
        
                 ! MPI: POP restart received only if pop module AND casa are active 
        
                 IF ( CABLE_USER%CALL_POP ) CALL worker_pop_types (comm,veg,pop) 
        
              END IF 
        
              ! MPI: create inp_t type to receive input data from the master 
        
              ! at the start of every timestep 
        
              CALL worker_intype (comm,met,veg) 
        
              ! MPI: casa parameters received only if cnp module is active 
        
              ! MPI: create send_t type to send the results to the master 
        
              ! at the end of every timestep 
        
              CALL worker_outtype (comm,met,canopy,ssnow,rad,bal,air,soil,veg) 
        
              ! MPI: casa parameters received only if cnp module is active 
        
              ! MPI: create type to send casa results back to the master 
        
              ! only if cnp module is active 
        
              IF (icycle>0) THEN 
        
                 CALL worker_casa_type (comm, casapool,casaflux, & 
        
                      casamet,casabal, phen) 
        
                 IF ( CABLE_USER%CASA_DUMP_READ .OR. CABLE_USER%CASA_DUMP_WRITE ) & 
        
                      CALL worker_casa_dump_types(comm, casamet, casaflux, phen) 
        
                 WRITE(logn,*) 'cable_mpiworker, POPLUC: ',  CABLE_USER%POPLUC 
        
                 WRITE(*,*) 'cable_mpiworker, POPLUC: ',  CABLE_USER%POPLUC 
        
                 CALL flush(logn) 
        
                 IF ( CABLE_USER%POPLUC ) & 
        
                      CALL worker_casa_LUC_types( comm, casapool, casabal) 
        
                 ! MPI: casa parameters received only if cnp module is active 
        
              END IF 
        
              ! MPI: create type to send restart data back to the master 
        
              ! only if restart file is to be created 
        
              IF(output%restart) THEN 
        
                 CALL worker_restart_type (comm, canopy, air) 
        
              END IF 
        
              ! Open output file: 
        
              ! MPI: only the master writes to the files 
        
              !CALL open_output_file( dels, soil, veg, bgc, rough ) 
        
              ssnow%otss_0   = ssnow%tgg(:,1) 
        
              ssnow%otss     = ssnow%tgg(:,1) 
        
              canopy%fes_cor = 0. 
        
              canopy%fhs_cor = 0. 
        
              met%ofsd = 0.1 
        
              ! CALL worker_sumcasa_types(comm, sum_casapool, sum_casaflux) 
        
              !count_sum_casa = 0 
        
              IF( icycle>0 .AND. spincasa) THEN 
        
                 WRITE(logn,*) 'EXT spincasacnp enabled with mloop= ', mloop 
        
                 CALL worker_spincasacnp(dels,kstart,kend,mloop,veg,soil,casabiome,casapool, & 
        
                      casaflux,casamet,casabal,phen,POP,climate,LALLOC, icomm, ocomm) 
        
                 SPINconv = .FALSE. 
        
                 CASAONLY                   = .TRUE. 
        
                 ktau_gl = 0 
        
                 ktau = 0 
        
              ELSEIF ( casaonly .AND. (.NOT. spincasa) .AND. cable_user%popluc) THEN 
        
                 CALL worker_CASAONLY_LUC(dels,kstart,kend,veg,soil,casabiome,casapool, & 
        
                      casaflux,casamet,casabal,phen,POP,climate,LALLOC, & 
        
                      icomm, ocomm) 
        
                 SPINconv = .FALSE. 
        
                 ktau_gl = 0 
        
                 ktau = 0 
        
              ENDIF 
        
           ELSE 
        
              IF (icycle.GT.0) THEN 
        
                 ! re-initalise annual flux sums 
        
                 casabal%FCgppyear =0.0 
        
                 casabal%FCrpyear  =0.0 
        
                 casabal%FCnppyear =0.0 
        
                 casabal%FCrsyear  =0.0 
        
                 casabal%FCneeyear =0.0 
        
              ENDIF 
        
           ENDIF !CALL1

src/offline/cable_driver_init.F90

ccarouge · 2024-11-28T02:39:13Z

@SeanBryan51 I know you are having issues testing all the changes but have you tried running benchab? The sites and the spatial runs should return identical results before and after.

ccarouge · 2024-11-28T02:43:24Z

For gpgs, please add it as deprecated on this page: https://cable.readthedocs.io/en/latest/user_guide/other_resources/obsolete_and_deprecated_features/obsolete_and_deprecated_features/ In replacement, just put the "gswp + leap" reason.

SeanBryan51 · 2024-11-28T03:34:13Z

Running it through benchcab now.

SeanBryan51 · 2024-11-28T03:44:14Z

It looks like I forgot to update the stubs for the serial build - but it also made me realise the build-ci currently builds only the MPI executable (due to the default value of the mpi variant in the CABLE spack package definition). Something to note for the future.

ccarouge

Seems ok but let's see what benchcab thinks of it.

This is done as setting `cable_user%MetType = 'gpgs'` via the namelist is equivalent to setting `cable_user%MetType = 'gswp'` and `leaps = .TRUE.`.

…ver_init

…ser%YearEnd to cable_driver_init_gswp Note prior to this change cable_user%YearStart and cable_user%YearEnd was only initialised for the master process. This change now initialises these variables over all processes so that the start and end year values are consistent across the drivers.

SeanBryan51 · 2024-11-28T04:47:07Z

Benchcab runs are bitwise reproducible with main:

PBS log file:

2024-11-28 15:40:15,127 - INFO - benchcab.benchcab.py:380 - Running comparison tasks...
2024-11-28 15:40:15,151 - INFO - benchcab.benchcab.py:381 - tasks: 168 (models: 2, sites: 42, science configurations: 4)
2024-11-28 15:43:03,606 - INFO - benchcab.benchcab.py:391 - 0 failed, 168 passed

Benchcab version: 4.2.0

config.yaml:

realisations:
  - repo:
      git:
        branch: main
  - repo:
      git:
        branch: 425-offline-driver-initialisation-refactor-part3

modules: [
  intel-compiler/2021.1.1,
  netcdf/4.7.4,
  openmpi/4.1.0
]

Spatial (MPI) runs all execute as expected (note outputs do not reproduce as per issue #463).

ccarouge

It's good for me.

Move allocation of GSWP_MID to cable_driver_init_gswp as it is gswp specific.

micaeljtoliveira

@SeanBryan51 I had another look and I have one suggestion and one question. Otherwise looks fine!

src/offline/cable_driver_init.F90

src/offline/cable_mpimaster.F90

SeanBryan51 requested review from micaeljtoliveira and ccarouge November 28, 2024 00:16

SeanBryan51 commented Nov 28, 2024

View reviewed changes

src/offline/cable_driver_init.F90 Outdated Show resolved Hide resolved

ccarouge marked this pull request as ready for review November 28, 2024 01:20

SeanBryan51 force-pushed the 425-offline-driver-initialisation-refactor-part3 branch from 7e6a8bb to 7a195f7 Compare November 28, 2024 03:53

ccarouge reviewed Nov 28, 2024

View reviewed changes

SeanBryan51 added 5 commits November 28, 2024 15:41

Deprecate cable_user%MetType = 'gpgs'

90f5885

This is done as setting `cable_user%MetType = 'gpgs'` via the namelist is equivalent to setting `cable_user%MetType = 'gswp'` and `leaps = .TRUE.`.

Cleanup assignment of local variables

70b47c7

Remove unnecessary broadcast of leaps namelist parameter

f91009e

Move initialisation related to cable_user%POPLUC_RunType to cable_dri…

8988b3f

…ver_init

SeanBryan51 force-pushed the 425-offline-driver-initialisation-refactor-part3 branch from 7a195f7 to 2fcf258 Compare November 28, 2024 04:41

ccarouge approved these changes Nov 28, 2024

View reviewed changes

SeanBryan51 added 4 commits November 28, 2024 16:21

Move open_met_file call (non-gswp) to cable_driver_init_default

21f53bc

Move allocation of GSWP_MID to cable_driver_init_gswp as it is gswp specific.

Move PLUME initialisation to cable_driver_init_plume

9b07aa0

Move CRU initialisation to cable_driver_init_cru

ca3a076

Move site initialisation to cable_driver_init_site

c78428c

SeanBryan51 force-pushed the 425-offline-driver-initialisation-refactor-part3 branch from 2fcf258 to c78428c Compare November 28, 2024 05:22

micaeljtoliveira reviewed Nov 28, 2024

View reviewed changes

src/offline/cable_driver_init.F90 Show resolved Hide resolved

src/offline/cable_mpimaster.F90 Show resolved Hide resolved

micaeljtoliveira approved these changes Nov 29, 2024

View reviewed changes

SeanBryan51 merged commit 8c970f2 into main Nov 29, 2024
7 checks passed

SeanBryan51 deleted the 425-offline-driver-initialisation-refactor-part3 branch November 29, 2024 01:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Offline driver initialisation refactor continued (cable_user%MetType) #498

Offline driver initialisation refactor continued (cable_user%MetType) #498

SeanBryan51 commented Nov 27, 2024 •

edited

Loading

SeanBryan51 commented Nov 28, 2024

ccarouge commented Nov 28, 2024

ccarouge commented Nov 28, 2024

SeanBryan51 commented Nov 28, 2024

SeanBryan51 commented Nov 28, 2024

ccarouge left a comment

SeanBryan51 commented Nov 28, 2024

ccarouge left a comment

micaeljtoliveira left a comment

Offline driver initialisation refactor continued (cable_user%MetType) #498

Offline driver initialisation refactor continued (cable_user%MetType) #498

Conversation

SeanBryan51 commented Nov 27, 2024 • edited Loading

Type of change

Checklist

SeanBryan51 commented Nov 28, 2024

ccarouge commented Nov 28, 2024

ccarouge commented Nov 28, 2024

SeanBryan51 commented Nov 28, 2024

SeanBryan51 commented Nov 28, 2024

ccarouge left a comment

Choose a reason for hiding this comment

SeanBryan51 commented Nov 28, 2024

ccarouge left a comment

Choose a reason for hiding this comment

micaeljtoliveira left a comment

Choose a reason for hiding this comment

SeanBryan51 commented Nov 27, 2024 •

edited

Loading