Can using Windmill to ETL/ELT data?

[nguyenthanhtien]

I'm looking to set up a new data pipeline using Windmill, and I'd like to get some feedback from the community. What are the key advantages and considerations when using Windmill for ETL/ELT workflows compared to other popular data engineering tools like Apache Airflow or Prefect? How does Windmill's architecture and feature set align with common data integration use cases? Any tips or best practices for designing robust and scalable Windmill-based data pipelines would be greatly appreciated.

[rubenfiszel]

Hi @nguyenthanhtien , I recommend taking a look at: https://www.windmill.dev/docs/core_concepts/data_pipelines

We're much faster than Airflow and Prefect. However, the only meaningful way to larger scale data pipeline is to use our S3 + parquet integration which is limited to 50MB uploads on the community edition. Windmill allow to use parquet (or any other blob format) files stored in s3 as input and outputs easily and then will even be able to cache the steps based on if the etags of those have changed or not.

[nguyenthanhtien]

Any limitation with the Community version? Still faster than the Airflow & Perfect tool?

[nguyenthanhtien]

@rubenfiszel Any limitation with the Community version? Still faster than the Airflow & Perfect tool?

[rubenfiszel]

There are some limitation but not with performance or number of workers. However, the native s3 integration is limited to 50MB