Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

异步执行器支持指定支持指定批量大小,将基于同一数据源的操作分为多个小任务 #195

Closed
Createsequence opened this issue Jan 28, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@Createsequence
Copy link
Collaborator

异步执行器支持指定支持指定批量大小,将基于同一数据源的操作分为多个小任务。

举个例子:

假设我们现在有 10 个待填充 Bean 对象,而每个 Bean 又各自通过 nsetedBean 嵌套一个 Bean 对象,即总共有 20 个 Bean。

现在,我们在 Bean 中分别基于 idkeycode 声明了三个装配操作,具体配置如下:

@Data
private static class Bean {

    @Assemble(container = "container1", props = @Mapping("name"))
    private Integer id;
    private String name;
    
    @Assemble(container = "container2", props = @Mapping("value"))
    private Integer key;
    private String value;
    
    @Assemble(container = "container2", props = @Mapping("val"))
    private Integer code;
    private String val;
    
    @Disassemble(type = Bean.class)
    private Bean nsetedBean;
}

那么,当我们填充这 10 个 Bean 时,实际上总共需要完成 10 * (1 + 1) * 3 共 60 组操作。

为了保证尽可能减少查库次数,因此默认情况下,执行器会将 60 次填充按对应的数据源容器打包成两个任务提交给线程池完成:

  • 查询 container1,然后完成全部基于 container1 的 40 组操作;
  • 查询 container2,然后完成基于 container1 的 20 组操作;

上述这个逻辑的问题在于,当需要填充的对象越来越多,且需要映射的字段也越来越多时,反射读写字段消耗的时间也会越来越多,甚至可能会超过查库或 RPC 调用所消耗的时间。

此时,为了提高效率,异步执行器需要可以指定批量大小,将每一组操作再拆分为更细力度的任务。比如,如果如果指定批量大小为 20,那么第一个任务就会被拆成两份,此时实际上提交到线程池中的任务就是三个:

  • 查询 container1,然后完成基于 container1 的 20 组操作;
  • 查询 container1,然后完成基于 container1 的 20 组操作;
  • 查询 container2,然后完成基于 container1 的 20 组操作;
@Createsequence Createsequence added the enhancement New feature or request label Jan 28, 2024
@Createsequence Createsequence added this to the release 2.5.0 milestone Jan 28, 2024
@Createsequence Createsequence self-assigned this Jan 28, 2024
Createsequence added a commit that referenced this issue Jan 28, 2024
…izes to divide operations based on the same data source into multiple small tasks (GitHub #195)
Createsequence added a commit that referenced this issue Jan 28, 2024
…izes to divide operations based on the same data source into multiple small tasks (GitHub #195)
Createsequence added a commit that referenced this issue Jan 28, 2024
…izes to divide operations based on the same data source into multiple small tasks (GitHub #195)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant