Failed import when running deltalake==0.14.0

[avriiil]

To reproduce:

• create new mamba/conda env
• mamba install python==3.11 pip
• pip install deltalake==0.14
• pip install dask-deltatable
• run import dask_deltatable

ImportError: cannot import name '_write_new_deltalake' from 'deltalake.writer' (/Users/rpelgrim/miniforge3/envs/deltalake-0130/lib/python3.11/site-packages/deltalake/writer.py)

To solve:

• pip install deltalake==0.13

[mrocklin]

In [1]: from pipeline.reduce import *
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[1], line 1
----> 1 from pipeline.reduce import *

File ~/workspace/etl-tpch/pipeline/reduce.py:6
      4 import coiled
      5 import dask_expr as dd
----> 6 import dask_deltatable
      7 from dask.distributed import LocalCluster
      8 from prefect import flow, task

File ~/mambaforge/envs/etl-tpch/lib/python3.11/site-packages/dask_deltatable/__init__.py:9
      3 __all__ = [
      4     "read_deltalake",
      5     "to_deltalake",
      6 ]
      8 from .core import read_deltalake as read_deltalake
----> 9 from .write import to_deltalake as to_deltalake

File ~/mambaforge/envs/etl-tpch/lib/python3.11/site-packages/dask_deltatable/write.py:18
     16 from dask.highlevelgraph import HighLevelGraph
     17 from deltalake import DeltaTable
---> 18 from deltalake.writer import (
     19     MAX_SUPPORTED_WRITER_VERSION,
     20     PYARROW_MAJOR_VERSION,
     21     AddAction,
     22     DeltaJSONEncoder,
     23     DeltaProtocolError,
     24     DeltaStorageHandler,
     25     __enforce_append_only,
     26     _write_new_deltalake,
     27     get_file_stats_from_metadata,
     28     get_partitions_from_path,
     29     try_get_table_and_table_uri,
     30 )
     31 from toolz.itertoolz import pluck
     33 from ._schema import pyarrow_to_deltalake, validate_compatible

ImportError: cannot import name '_write_new_deltalake' from 'deltalake.writer' (/Users/mrocklin/mambaforge/envs/etl-tpch/lib/python3.11/site-packages/deltalake/writer.py)

[jrbourbeau]

Looks like that method got moved to

from deltalake._internal import write_new_deltalake

Though still a private method, which isn't ideal

[jrbourbeau]

Ah, I think we want this public method https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake (at least from a quick glance, it looks like we could use that instead)

[fjetter]

Ah, I think we want this public method

No. this would create a commit for every partition which is exactly what we don't want. Especially without a distributed lock in place this would likely cause data loss whenever transactions conflict