Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1469 Output LB statistics as JSON #1475

Merged
merged 62 commits into from
Jun 28, 2021
Merged

1469 Output LB statistics as JSON #1475

merged 62 commits into from
Jun 28, 2021

Conversation

lifflander
Copy link
Collaborator

@lifflander lifflander commented Jun 13, 2021

Fixes #1469

  • Include new json library
  • Include new compression library: brotli
  • Write C++ wrapper for encoding streaming data using compression library
  • Write output adaptor for the JSON library to stream output to compressed buffer
  • Write C++ wrapper for decoding? Do we need this? Probably for the stats restart reader
  • Adapt tests for new format
  • Provide options to users to output JSON uncompressed?

Should we completely remove the old stat output or leave it as an option that the user can select until we decide to remove it?

Sorry, something went wrong.

@lifflander
Copy link
Collaborator Author

@PhilMiller @JacobDomagala @jstrzebonski @ppebay @nlslatt

I would like feedback on this PR!

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (gcc-5, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (gcc-6, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (clang-5.0, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (gcc-10, ubuntu, openmpi, no LB)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (clang-3.9, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (gcc-8, ubuntu, mpich, address sanitizer)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (gcc-9, ubuntu, mpich, zoltan)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (intel 18.03, ubuntu, mpich)

Build for 1852358

Build failed for unknown reason. Check build logs


Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (clang-10, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (clang-10, alpine, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (clang-9, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (intel 19, ubuntu, mpich)

Build for 1852358

Build failed for unknown reason. Check build logs


Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (nvidia cuda 11.0, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (nvidia cuda 10.1, ubuntu, mpich)

Build for 1852358

Compilation - successful

Testing - passed

Build log

@github-actions
Copy link

github-actions bot commented Jun 13, 2021

PR tests (gcc-7, ubuntu, mpich, trace runtime, LB)

Build for 1852358

Compilation - successful

Testing - passed

Build log

std::unique_ptr<StatsData> sd
) {
std::deque<std::set<ElementIDType>> element_history;
for (PhaseType phase = 0; phase < sd->node_data_.size(); phase++) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be able to convert this loop into for-each? Maybe something like this:
for (auto const& nd : sd->node_data_) {

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that sd->node_data_ is an unordered_map but these need to be traversed in the correct order for element_history to be correct. So I can't use an iterator.


template <typename StreamLike>
bool Compressor::writeImpl(
StreamLike& s, uint8_t const* buffer, std::size_t const size, bool finish_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the point in the usage of finish_, but to be honest, I'm not a fan of changing the behaviour of the function with a boolean variable. But from the top of my head, I don't have any clever idea of refactoring this function. So I'm fine with leaving it be.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially had two different functions (one for finishing and one for incremental writes). The problem is that I have to duplicate basically this whole function so I combined them.

Copy link
Contributor

@jstrzebonski jstrzebonski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some minor notes and suggestions, but this looks really great!

Comment on lines +228 to +235
}
}
}
}
}
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes me think that extracting some helpers could help readability

Copy link
Contributor

@cz4rs cz4rs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Copy link
Collaborator

@nlslatt nlslatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with merging this after addressing the point that @cz4rs made about the stats restart reader.

@lifflander lifflander force-pushed the 1469-lb-stats-json branch from 0e952a9 to 1852358 Compare June 28, 2021 18:32
@lifflander lifflander merged commit a36acef into develop Jun 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Explore using JSON for LB stats files for a more consistent, readable format
6 participants