Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asynchronous executors #913

Closed
wants to merge 18 commits into from
Closed

Asynchronous executors #913

wants to merge 18 commits into from

Conversation

addisoncrump
Copy link
Collaborator

This PR introduces asynchronous executors will task batching for use with asynchronous execution environments. This enables many new use cases, including (but not limited to):

  • distributed executors (fuzzer pipeline hosted on a single machine, executors distributed over multiple)
  • differential execution over diverse platforms (e.g., diff fuzzing a macOS target and a Windows target)
  • use of std fuzzer pipelines with no-std environments (e.g., low-level fuzzing with testcases submitted over a channel)

The process for performing asynchronous fuzzing prefers task batching. This is done using DeferredExecutionResult and expects each stage to offer a batching execution where possible, e.g. for a mutational stage:

  1. Determine how many mutations will be created (n)
  2. Allocate space for n DeferredExecutionResults
  3. For each mutation:
    1. Mutate the input
    2. Start the execution on the executor and stash the DeferredExecutionResult
  4. For each DeferredExecutionResult:
    1. Get the result
    2. Check the interestingness with the associated feedback

Because a blanket impl of AsyncExecutor exists for all Executors with observers, synchronous executors may be used in an asynchronous pipeline at the cost of an extra clone of the observers. Additionally, AsyncBridge may be used to use asynchronous executors in a synchronous pipeline, but may suffer execution speed due to being used without batching.

@domenukk
Copy link
Member

Cc @epi052 you may like this

@addisoncrump addisoncrump marked this pull request as ready for review November 25, 2022 15:55
@addisoncrump
Copy link
Collaborator Author

Well, the CI is broken, but I've tested the fuzzers locally and they do be worky 👍

@addisoncrump
Copy link
Collaborator Author

This is still missing a lot of perf items and such which are likely desirable for introspection. Working on that still, but it's in a semi-stable state.

@addisoncrump
Copy link
Collaborator Author

addisoncrump commented Nov 26, 2022

Stages that need async support (check is complete, cross-out is deemed impossible):

  • CalibrationStage
  • ClosureStage (by definition compatible)
  • ConcolicTracingStage
  • DumpToDiskStage (by definition compatible)
  • GeneralizationStage
  • PowerMutationalStage (batched executions, collect results later)
  • PushStageAdapter
  • SimpleConcolicMutationalStage
  • SkippableStage (by definition compatible)
  • StdMutationalStage (batched executions, collect results later)
  • StdTMinMutationalStage
  • SyncFromDiskStage
  • TracingStage
  • TuneableMutationalStage
  • ShadowTracingStage

Other things that need doing:

  • DifferentialExecutor
  • DifferentialObserver

@domenukk
Copy link
Member

Is this ready to merge?

@addisoncrump
Copy link
Collaborator Author

No, I need to complete the checklist above.

@domenukk domenukk marked this pull request as draft December 4, 2022 18:05
@addisoncrump addisoncrump mentioned this pull request Nov 7, 2023
9 tasks
@domenukk
Copy link
Member

Closing this, a lot has happened since. Feel free to reopen if it's still relevant.

@domenukk domenukk closed this Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants