-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Website] Add Blog about Comet Donation #479
Changes from 7 commits
e737e84
1515c4e
8ae61a6
70ff88d
c512a68
e58fba0
b3d17ce
64bf139
919c934
8d7e76e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
--- | ||
layout: post | ||
title: "Announcing Apache Arrow DataFusion Comet" | ||
date: "2024-02-27 00:00:00" | ||
author: pmc | ||
categories: [release] | ||
--- | ||
<!-- | ||
{% comment %} | ||
Licensed to the Apache Software Foundation (ASF) under one or more | ||
contributor license agreements. See the NOTICE file distributed with | ||
this work for additional information regarding copyright ownership. | ||
The ASF licenses this file to you under the Apache License, Version 2.0 | ||
(the "License"); you may not use this file except in compliance with | ||
the License. You may obtain a copy of the License at | ||
|
||
http://www.apache.org/licenses/LICENSE-2.0 | ||
|
||
Unless required by applicable law or agreed to in writing, software | ||
distributed under the License is distributed on an "AS IS" BASIS, | ||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
See the License for the specific language governing permissions and | ||
limitations under the License. | ||
{% endcomment %} | ||
--> | ||
|
||
# Introduction | ||
The Apache Arrow PMC is pleased to announce the donation of the [Comet project], | ||
a native Spark SQL Accelerator built on [Apache Arrow DataFusion]. | ||
|
||
Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to | ||
accelerate Spark workloads. It is designed as a drop-in | ||
replacement for Spark's JVM based SQL execution engine and offers significant | ||
performance improvements for some workloads as shown below. | ||
|
||
```text | ||
┌─────────────────────────────────────────────────────────────────┐ | ||
│ │ | ||
│ ┌──────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ | ||
│ │ SQL │ │ Cluster │ │ DAG/Task │ ... │ Executor │ │ | ||
│ │ Planner │ │ Manager │ │ Scheduler │ │ │ │ | ||
│ └──────────┘ └────────────┘ └────────────┘ └────────────┘ │ | ||
│ │ │ | ||
└─────────────────────────────────────────────────────────────────┘ | ||
Spark (JVM Based) │ | ||
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ | ||
|
||
│ | ||
▼ | ||
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ | ||
Comet Execution ┃ ┃ | ||
Engine ┃ ┌─────────────────────────┐ ┃ | ||
(Native Code) ┃ │ Apache Arrow DataFusion │ ┃ | ||
┃ └─────────────────────────┘ ┃ | ||
┃ ┃ | ||
┃ ┌─────────────────────────┐ ┃ | ||
┃ │ Spark Compatible │ ┃ | ||
┃ │ Expressions/Operators │ ┃ | ||
┃ └─────────────────────────┘ ┃ | ||
┃ ┃ | ||
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ | ||
``` | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I felt we needed some sort of image, but this is pretty basic ASCII art... If anyone has other ideas that would be appreciated There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The image in comet's README is a good alternative, but we need to add the driver part. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That is a good image, but it is quite detailed What I was hoping was to have an image that visually depicts the relationship between Spark and Comet Something similar maybe to https://github.com/oap-project/gluten?tab=readme-ov-file#2-architecture I don't know enough about how spark is implemented internally to make such a picture now, though There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm thinking a figure like: Spark Frontend / Optimizer (JVM Based) -> Comet Execution Engine (Native Code)
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks good to me. Thanks @alamb |
||
**Figure 1**: With Comet, users interact with the same Spark ecosystem, tools | ||
and APIs such as Spark SQL. Queries still run through Spark's mature and feature | ||
rich query optimizer and planner. However, the execution is delegated to Comet, | ||
which is significantly faster and more resource efficient than the JVM based | ||
implementation. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I assume you might already be planning to add more details, but I'd love to hear about any Comet-specific optimizations that were implemented. Similarly, I'd be interested to hear about DataFusion-specific features that made it a particularly good fit for Comet's implementation (DataFusion's focus on extensibility comes to mind). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
||
[Rust]: https://www.rust-lang.org/ | ||
|
||
# Background | ||
|
||
Comet is one of a growing class of projects that aim to accelerate Spark using | ||
native columnar engines such as the proprietary [Databricks Photon Engine] and | ||
the open source [Gluten project] and [Spark RAPIDS]. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would it help to give some background as to why these initiatives have been popping up in the first place? I know the article alludes to columnar execution and lower resource usage, but (imo) it would be more approachable if it were framed from the perspective of the struggles Spark users face today, and then explaining how these initiatives address those problems. Relatedly, I vaguely remember hearing about an effort to bring columnar execution into mainline Spark (was it Project Tungsten?). I would be curious to hear about the differences between that effort and Comet's approach. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thank you @SChakravorti21 for this suggestion. I agree such background would help -- maybe some of the other reviewers (some of whom are spark committers I believe) can offer more specifics and how this relates to mainline spark. The Velox Paper basically says the JVM implementation of spark is slow so native columnar execution is better, but I don't recall it delving into any more detail |
||
|
||
Comet was originally implemented at Apple and the engineers who worked on the | ||
project are also significant contributors to Arrow and DataFusion. Bringing | ||
Comet into the Apache Software Foundation will accelerate its development and | ||
grow its community of contributors and users. | ||
|
||
[Comet project]: https://github.com/apache/arrow-datafusion-comet | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I learned that there is a similar effort to build a Spark-native accelerator using DataFusion: Blaze. Wondering if it deserves a mention as well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @richox or @yjshen, who worked on Blaze in the past, do you have any suggestions or thoughts on if/how we should mention Blaze in this post? The last time I checked the Blaze project hadn't had any updates, but it appears that in the last 6 months @richox has been working on it (BTW @richox contributed some excellent improvements to DataFusion as well such a a loser tree for the sorting implementation 🙏 ) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added a link in 919c934 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I heard the current Blaze team has proposed donating it to Apache in the past, but I am not sure what the status is. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is probably unrealistic, but perhaps the projects could converge over time There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1. It would be great if Blaze and Comet can join forces in some way! |
||
[Apache Arrow DataFusion]: https://arrow.apache.org/datafusion | ||
[Databricks Photon Engine]: https://www.databricks.com/product/photon | ||
[Gluten project]: https://incubator.apache.org/projects/gluten.html | ||
alamb marked this conversation as resolved.
Show resolved
Hide resolved
|
||
[Spark RAPIDS]: https://github.com/NVIDIA/spark-rapids | ||
|
||
# Get Involved | ||
Comet is still in the early stages of development and we would love to have you | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be good to double check / make sure this is the right way to ask people to get involved There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This looks good enough for now. Maybe we can consider opening the "Discussions" tab for people to ask questions & raise discussions, but that can be done later. |
||
join us and help shape the project. We are working on an initial release, and | ||
expect to post another update with more details at that time. | ||
|
||
Before then, here are some ways to get involved: | ||
|
||
* Learn more by visiting the [Comet project] page and reading the [mailing list | ||
discussion] about the initial donation. | ||
|
||
* Help us plan out the [roadmap] | ||
|
||
* Try out the project and provide feedback, file issues, and contribute code. | ||
|
||
[mailing list discussion]: https://lists.apache.org/thread/0q1rb11jtpopc7vt1ffdzro0omblsh0s | ||
[roadmap]: https://github.com/apache/arrow-datafusion-comet/issues/19 | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
instead of text, maybe we can create some diagrams using tools like Lucidchart? I can help to translate this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would be great if you can translate the diagram to an image. Maybe we could also (re)use such an image in the main comet readme.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does this look? Let me know if you have ideas to improve it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I swapped the blog to use the image and I think it is looking pretty good
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also I liked the image so much I propose putting it in the main comet repo: apache/datafusion-comet#148