Skip to content

Commit

Permalink
feat: increase maximum message size
Browse files Browse the repository at this point in the history
A batch can contain up to 1024 rows. If the columns contain large
amounts of data (json blogs or free text), the total batch size can
exceed the max message size. By doubling this we should be able to
accommodate most use cases.

TODO: if we hear of people reaching this max, maybe we should make it
configurable.
  • Loading branch information
aiguofer committed Dec 13, 2024
1 parent f5e4eb2 commit 3a717c9
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions dbtsl/api/adbc/client/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ def _extra_db_kwargs(cls) -> Dict[str, str]:
return {
DatabaseOptions.WITH_COOKIE_MIDDLEWARE.value: "true",
f"{DatabaseOptions.RPC_CALL_HEADER_PREFIX.value}user-agent": env.PLATFORM.user_agent,
# double the default max msg size in case of queries with large batches
DatabaseOptions.WITH_MAX_MSG_SIZE: f"{1024 * 1024 * 32}",

Check failure on line 27 in dbtsl/api/adbc/client/base.py

View workflow job for this annotation

GitHub Actions / code-quality

Type "dict[str | DatabaseOptions, str]" is not assignable to return type "Dict[str, str]"   "Literal[DatabaseOptions.WITH_MAX_MSG_SIZE]" is not assignable to "str" (reportReturnType)
}

def __init__( # noqa: D107
Expand Down

0 comments on commit 3a717c9

Please sign in to comment.