feat: Developing Verifiers #1563
Conversation
| 3. Verifying outputs against expected results | ||
| """ | ||
|
|
||
| def __init__( |
There was a problem hiding this comment.
could we add batch process support and add timeout setting?
There was a problem hiding this comment.
@Wendong-Fan for adding batch processing, I would need to add a run_batch method to the BaseInterpreter class. Should I go ahead and make the changes?
There was a problem hiding this comment.
I think we can also leave the batch process logic within CodeVerifier, the batch processing logic is more related to verifier rather than code execution, WDYT?
There was a problem hiding this comment.
Using it in the CodeVerifier would mean using ThreadPoolExecutor while in interpreter it would be based on ProcessPoolExecutor.
I think we will be okay moving with ThreadPool for now. Let me know your thoughts on this.
| 3. Verifying outputs against expected results | ||
| """ | ||
|
|
||
| def __init__( |
There was a problem hiding this comment.
I think we can also leave the batch process logic within CodeVerifier, the batch processing logic is more related to verifier rather than code execution, WDYT?
Description
Adding verifiers module to camel
Motivation and Context
Why is this change required? What problem does it solve?
close #1551
Types of changes
What types of changes does your code introduce? Put an
xin all the boxes that apply:Implemented Tasks
Checklist
Go over all the following points, and put an
xin all the boxes that apply.If you are unsure about any of these, don't hesitate to ask. We are here to help!