Skip to content

Troubleshooting CTF

Jump to bottom
Travis Christian edited this page Nov 7, 2023 · 6 revisions

If you encounter an unexpected error, test failure, or other issue with CTF, here are some steps to help identify the problem. See Reporting Issues below if you need to open a new issue. Please do not include non-open-source technical details on GitHub.

Best Practices

These tips will help you avoid or spot many commonly reported issues.

  • CTF is developed and tested on 64-bit Oracle Linux 8 (recommended), formerly CentOS 7, and is known to work well on Ubuntu and Mac OS. Other platforms including Windows, Docker, and 32-bit Linux are not officially supported.

  • The test scripts included in CTF are the best reference for writing your own tests. They are divided into 3 locations:

    • vv_tests are the official functional tests demonstrating individual CTF requirements
    • functional_tests are miscellaneous tests demonstrating various features in more complete use cases
    • example_tests are more specialized use cases such as particular configurations or intentionally failing tests

  • New users especially are encouraged to the use the CTF editor to create and run tests, as it automates much of the syntax.

  • CTF test scripts and CCDD data files use standard JSON syntax. If editing either of these by hand it is recommended to use a text editor with syntax highlighting to catch misplaced characters.

  • As a general rule in your test scripts, use a number when the value is a number and a string when it is not. In other words, do not wrap a raw number or other value in quotes unless it is to be interpreted as a string literal.

  • A wait of 0 (default) is almost always appropriate for telemetry and event checks. This will allow CTF to begin polling immediately after the previous instruction.

  • When using CTF on the command line, the optional colorlog Python package helps make the console output more readable. However it is not recommended for use with the CTF editor as it interferes with the editor's own formatting.

Gathering Information

First, enable debug logging in your config and run again:

[logging]

# What level of logging?
# ERROR : only show error logs - very minimal output
# INFO :  only show  info, warning, error, and critical logs
# DEBUG:  show all logs!
log_level = DEBUG

If the issue involves CCDD data types, also enable CCSDS debug logging:

[cfs]

# Log CCSDS Import Process (Logs all messages parsed from CCSDS Data Directory)
log_ccsds_imports = true

If the issue involves handling received messages, you may also want to enable telemetry debug logging:

[cfs]

# Log every telemetry packet received?
telemetry_debug = true

Check the logs from before the problem arises for anything unexpected, especially atypical ERROR or WARNING log statements or exceptions.

Common Issues

  • If CTF immediately exits due to a missing package (usually demjson) or other Python error, check that you have activated the CTF virtual environment (source ./activate_ctf_env.sh) or have installed the dependencies via Pip. It is recommended to reinstall the virtual environment (./setup_ctf_env.sh -c) after updating CTF or if you encounter such errors with the environment activated. If the virtual environment is active, you should see a modified prompt (ctf_env) ... on the console, and which python should return ~/ctf_env/bin/python.

  • If the issue is new, ie the same test used to work, ensure that you are using the latest release of CTF (indicated in the first line of CTF output), and check the release notes in the main ReadMe for any relevant changes. If you have recently updated CTF, run the upgrade script in ctf/tools against your test scripts and config files and note any changes that were made.

  • If the error happens while using the CTF Editor, try running CTF directly from the command line to see if anything changes.

  • If there is an error while parsing CCDD files (ccdd_export_reader or ccsds_interface) before the test starts, inspect the relevant files and compare their contents to the those provided in the sample workspace to ensure you are using the correct format and syntax.

  • If a verification instruction (such as CheckEvent) unexpectedly fails, check the logs to see if the packet was received, or if there is any error mentioning the MID. The line Receiving Packet for Data Type: ... will be logged the first time a packet for any given MID is successfully received; if this statement is not found no such packet was received. Also check instruction timing by comparing timestamps from the test and telemetry logs, as both the wait and the verify_timeout (or default ctf_verification_timeout), and evs_messages_clear_after_time in the case of EVS events, affect the time window for verification. Timestamps in the telemetry log are in seconds since the start of the test. Verification instructions often fail due to mismatched timing. Telemetry and event checks only analyze messages received within the corresponding instruction's time window.

  • If EnableCfsOutput fails (usually by timing out), verify that you are providing the right IP and port, and that cFS has started. On Linux, check for any persisting instances of cFS with ps -aux | grep core and kill them if necessary.

  • If SendCfsCommand fails, verify that you are providing the correct args corresponding to the definition in the CCDD JSON file, including structure, spelling, and data type.

  • If RegisterCfs fails or the first cFS instruction fails preceded by Failed to configure CFS target(s) from config file, there was a problem with the config file, usually in the cFS target section. Errors starting with Config value ... does not exist or is not the right type or Validation error - invalid field ... will indicate which values caused the failure. All config fields for a given CFS protocol are required even if not used, however the [cfs] section may be used to provide default values for other targets.

  • Errors starting with Error Applying Function usually mean that instruction parameters in data were invalid or missing, unless there are other errors logged immediately above. Check the instruction documentation in the corresponding plugin's Readme.

If these steps do not uncover the issue or something else is wrong, check the FAQ section near the end of the Software User's Guide.

Reporting Issues

If you still cannot resolve the issue or believe you have found a bug in CTF, please enable ALL debug logging as described above and send us your full test script, config file, output directory (Run_<datetime>), and if the issue involves a certain data type or MID, the associated CCDD JSON file(s). You may either open an issue on GitHub, or send the files to us directly depending on the content. Along with the logs, please also include the commit and/or release tag of CTF, your OS and Python setup, and if you are running with any additional plugins or other changes.

Please be mindful of sensitive data when attaching files. Do not include project-specific technical details or non-open-source code snippets on Github. If these are relevant send them to us directly.

Clone this wiki locally