Skip to content

Commit

Permalink
io: On importing data with Dask, use os.cpu_count() for npartitions
Browse files Browse the repository at this point in the history
... as a default value.
  • Loading branch information
amotl committed Nov 8, 2023
1 parent 5782ad8 commit 27ef4ad
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions cratedb_toolkit/util/database.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Copyright (c) 2023, Crate.io Inc.
# Distributed under the terms of the AGPLv3 license, see LICENSE.
import io
import os
import typing as t
from pathlib import Path

Expand Down Expand Up @@ -227,8 +228,7 @@ def import_csv_dask(
from crate.client.sqlalchemy.support import insert_bulk

# Set a few defaults.
# TODO: Use amount of CPU cores instead?
npartitions = npartitions or 4
npartitions = npartitions or os.cpu_count()

if progress:
from dask.diagnostics import ProgressBar
Expand Down

0 comments on commit 27ef4ad

Please sign in to comment.