Benchmarks

[gasse]

• JSON serializable Benchmark class with name, high_level_action_set and env_args_list

  from dataclasses import dataclass
  
  from dataclasses_json import DataClassJsonMixin
  
  
  @dataclass
  class Example(DataClassJsonMixin):
      a: int
      b: str
  
      def do_something(self):
          print(self.a, self.b)
    
  x = Example(0, "hello")
  x.do_something()
  
  x_json = x.to_json()
  
  y = Example.from_json(x_json)
  y.do_something()

• flexible, easy to crate your own benchmark

  Benchmark(
      name="miniwob_tiny_test",
      high_level_action_set=DEFAULT_HIGHLEVEL_ACTION_SETS["miniwob"],
      env_args_list=_make_env_args_list_from_repeat_tasks(
          task_list=["miniwob.click-dialog", "miniwob.click-checkboxes"],
          max_steps=5,
          n_repeats=2,
          seeds_rng=np.random.RandomState(42),
      ),
  )

• CSV files with benchmark task list and their metadata
• default benchmarks collection
  • miniwob_all
  • miniwob_webgum
  • miniwob_tiny_test
  • miniwob_train
  • miniwob_test
  • webarena
  • visualwebarena
  • workarena_l1
  • workarena_l1_sort
  • workarena_l2
  • workarena_l3

Missing:

• unit tests
• workarena l2 and l3 task metadata
• webarena metadata (categories)
• visualwebarena task list and metadata
• integration in AgentLab (different PR)

[gasse]

solves #170 #160

[gasse]

Trust-approved by @recursix