💡 [REQUEST] - User defined success criteria

[l50]

Implementation PR

N/A

Reference Issues

N/A

Summary

Provide TTPForge users with the means to define "success criteria" for a TTP.

Basic Example

High-level example ideas:

• Check output for "string-match" to determine if TTP worked
• Check file created by TTP exists on disk, isn't empty, and includes some string

Drawbacks

None that I can think of

Unresolved questions

1. Is this something we should support?
2. How much of a lift would it be?
3. Should this work be prioritized and knocked out before DEF CON?

[l50]

@d3sch41n @CrimsonK1ng there are a couple of unresolved questions above that will provide a good starting point for this question/potential feature request.

[CrimsonK1ng]

What is the proposal for success criteria, what does success criteria mean to you? Do you have a better example, something that highlights the use case? What would that look like in your configuration?

Using string match for success, what strings are we checking? Is it just the current step output or another command output.

How are you intending to handle file lookups ?

What is the difference between checking for success in a step using a separate field vs building in the success check into the bash script for instance.

[l50]

What is the proposal for success criteria, what does success criteria mean to you?

• The first success criteria is determining that this is a good idea, collectively.

Do you have a better example, something that highlights the use case? What would that look like in your configuration?

More involved example:

---
name: Hello World
description: |
  Print hello world
steps:
  - name: hello
    file: ./ttps/privilege-escalation/credential-theft/hello-world/hello-world.sh
    cleanup:
      name: cleanup
      inline: |
        echo "cleaned up!"
    success_criteria: 
       - "Hello*"
       - "cleaned up!"

In this example, running this TTP will return as a successful operation IFF

"Hello*" and "cleaned up!" are in the TTP output.

Using string match for success, what strings are we checking? Is it just the current step output or another command output.

I was thinking all of the output from running a given TTP.

How are you intending to handle file lookups ?

Maybe I'm not understanding your question as these seem like too obvious of an answer:

// Check if file exists
os.Stat("filethatneedstoexist.txt")

func checkIfFileContainsString(filePath, searchString string) bool {
	file, err := os.Open(filePath)
	if err != nil {
		fmt.Printf("Error opening file: %v\n", err)
		return false
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		if strings.Contains(scanner.Text(), searchString) {
			return true
		}
	}

	if err := scanner.Err(); err != nil {
		fmt.Printf("Error reading file: %v\n", err)
		return false
	}

	return false
}

// Check if file contains "success yay!"
filePath := "example.txt"
searchString := "success yay!"

found := checkIfFileContainsString(filePath, searchString)
fmt.Printf("String %q found in file: %v\n", searchString, found)

What is the difference between checking for success in a step using a separate field vs building in the success check into the bash script for instance.

Checking for success for a TTP governs the state of the overall TTP vs. success in a step which can be used to determine if the next step should run, etc.

[CrimsonK1ng]

How would you represent that in yaml? Do you want the success criteria to have another value that uses a file keyword? How would you distinguish between a string and a file for success criteria?

Your example only highlights the simple case of string matches. What would a file success look like?

[l50]

Here's a more complete proposed example that also includes the file success criteria as well:

---
name: Hello World
description: |
  Print hello world
steps:
  - name: hello
    file: ./ttps/privilege-escalation/credential-theft/hello-world/hello-world.sh
    cleanup:
      name: cleanup
      inline: |
        echo "cleaned up!"
    success_criteria: 
      output_matches:
        - "Hello*"
        - "cleaned up!"
      file_exists:
        - "./output/file1.txt"
      file_contains:
        - path: "./output/file2.txt"
          pattern: "*sample content*"

As you can see, output_matches exists to check output, file_exists provides a mechanism to check if a file exists, and file_contains can be used to determine if a file matches input string or wildcard criteria.

Thoughts? Anything else missing @CrimsonK1ng ?

[d3sch41n]

I see the use case for this, as long as we don't make it too complicated and try to stretch too far to support weird
validation conditions. The alternative to this is that we tell users "write your validation into your TTP itself (in bash/python/golang)" - for bash especially, this won't be very viable bcs, as @l50 pointed out earlier, validation in bash looks like a bunch of pipes and grep awfulness. Even if we provide examples in our templates, it still is awful compared to this cleaner way.

Some thoughts that may be helpful:

• validation should run between step execution and cleanup
• this initial set of output_matches, file_exists, file_contains looks good - we probably don't want to add too many more conditions beyond those
• can architect validation + execution of these criteria blocks in a similar manner to how Alek architected TTP steps - sequentially attempt deserialization into instances of each criteria type, like how we validate steps

[d3sch41n]

Will be taking this on in the next sprint. I will add the requested success_criteria field with the following types of supported checks:

1. Check if the command exited with the appropriate status
2. Check if a file with a given path was created
3. Check if the command stdout matches a regular expression (inline: steps only, possibly add file: as well)
4. Check if a command stdout is JSON format and contains a given key (inline: steps only, possibly add file: as well)

Item (4) already exists via outputs: - I will move it under the success criteria umbrella and then add (3) under outputs: as well