Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Website] Add Blog about Comet Donation #479

Merged
merged 10 commits into from
Mar 6, 2024
106 changes: 106 additions & 0 deletions _posts/2024-02-27-comet-donation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
---
layout: post
title: "Announcing Apache Arrow DataFusion Comet"
date: "2024-02-27 00:00:00"
author: pmc
categories: [release]
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

# Introduction
The Apache Arrow PMC is pleased to announce the donation of the [Comet project],
a native Spark SQL Accelerator built on [Apache Arrow DataFusion].

Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to
accelerate Spark workloads. It is designed as a drop-in
replacement for Spark's JVM based SQL execution engine and offers significant
performance improvements for some workloads as shown below.

```text
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of text, maybe we can create some diagrams using tools like Lucidchart? I can help to translate this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be great if you can translate the diagram to an image. Maybe we could also (re)use such an image in the main comet readme.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Boson Vectorized Shuffle - Page 1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this look? Let me know if you have ideas to improve it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I swapped the blog to use the image and I think it is looking pretty good

Screenshot 2024-03-01 at 3 31 49 PM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I liked the image so much I propose putting it in the main comet repo: apache/datafusion-comet#148

┌─────────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ SQL │ │ Cluster │ │ DAG/Task │ ... │ Executor │ │
│ │ Planner │ │ Manager │ │ Scheduler │ │ │ │
│ └──────────┘ └────────────┘ └────────────┘ └────────────┘ │
│ │ │
└─────────────────────────────────────────────────────────────────┘
Spark (JVM Based) │
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
Comet Execution ┃ ┃
Engine ┃ ┌─────────────────────────┐ ┃
(Native Code) ┃ │ Apache Arrow DataFusion │ ┃
┃ └─────────────────────────┘ ┃
┃ ┃
┃ ┌─────────────────────────┐ ┃
┃ │ Spark Compatible │ ┃
┃ │ Expressions/Operators │ ┃
┃ └─────────────────────────┘ ┃
┃ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I felt we needed some sort of image, but this is pretty basic ASCII art... If anyone has other ideas that would be appreciated

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The image in comet's README is a good alternative, but we need to add the driver part.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good image, but it is quite detailed

What I was hoping was to have an image that visually depicts the relationship between Spark and Comet

Something similar maybe to https://github.com/oap-project/gluten?tab=readme-ov-file#2-architecture

image

I don't know enough about how spark is implemented internally to make such a picture now, though

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking a figure like:

Spark Frontend / Optimizer (JVM Based) -> Comet Execution Engine (Native Code)

  • -> Arrow DataFusion
  • -> Spark compatible expression/operator

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a shot at updating it in 70ff88d

What do you think?

Screenshot 2024-02-28 at 6 50 30 AM

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks @alamb

**Figure 1**: With Comet, users interact with the same Spark ecosystem, tools
and APIs such as Spark SQL. Queries still run through Spark's mature and feature
rich query optimizer and planner. However, the execution is delegated to Comet,
which is significantly faster and more resource efficient than the JVM based
implementation.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you might already be planning to add more details, but I'd love to hear about any Comet-specific optimizations that were implemented. Similarly, I'd be interested to hear about DataFusion-specific features that made it a particularly good fit for Comet's implementation (DataFusion's focus on extensibility comes to mind).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I defer to @sunchao / @viirya or one of the other comet authors for this. Maybe it would be a good topic to address in the initial release blog


[Rust]: https://www.rust-lang.org/

# Background

Comet is one of a growing class of projects that aim to accelerate Spark using
native columnar engines such as the proprietary [Databricks Photon Engine] and
the open source [Gluten project] and [Spark RAPIDS].

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it help to give some background as to why these initiatives have been popping up in the first place? I know the article alludes to columnar execution and lower resource usage, but (imo) it would be more approachable if it were framed from the perspective of the struggles Spark users face today, and then explaining how these initiatives address those problems.

Relatedly, I vaguely remember hearing about an effort to bring columnar execution into mainline Spark (was it Project Tungsten?). I would be curious to hear about the differences between that effort and Comet's approach.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @SChakravorti21 for this suggestion. I agree such background would help -- maybe some of the other reviewers (some of whom are spark committers I believe) can offer more specifics and how this relates to mainline spark.

The Velox Paper basically says the JVM implementation of spark is slow so native columnar execution is better, but I don't recall it delving into any more detail


Comet was originally implemented at Apple and the engineers who worked on the
project are also significant contributors to Arrow and DataFusion. Bringing
Comet into the Apache Software Foundation will accelerate its development and
grow its community of contributors and users.

[Comet project]: https://github.com/apache/arrow-datafusion-comet

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I learned that there is a similar effort to build a Spark-native accelerator using DataFusion: Blaze. Wondering if it deserves a mention as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@richox or @yjshen, who worked on Blaze in the past, do you have any suggestions or thoughts on if/how we should mention Blaze in this post?

The last time I checked the Blaze project hadn't had any updates, but it appears that in the last 6 months @richox has been working on it (BTW @richox contributed some excellent improvements to DataFusion as well such a a loser tree for the sorting implementation 🙏 )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link in 919c934

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I heard the current Blaze team has proposed donating it to Apache in the past, but I am not sure what the status is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is probably unrealistic, but perhaps the projects could converge over time

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. It would be great if Blaze and Comet can join forces in some way!

[Apache Arrow DataFusion]: https://arrow.apache.org/datafusion
[Databricks Photon Engine]: https://www.databricks.com/product/photon
[Gluten project]: https://incubator.apache.org/projects/gluten.html
alamb marked this conversation as resolved.
Show resolved Hide resolved
[Spark RAPIDS]: https://github.com/NVIDIA/spark-rapids

# Get Involved
Comet is still in the early stages of development and we would love to have you
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to double check / make sure this is the right way to ask people to get involved

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good enough for now. Maybe we can consider opening the "Discussions" tab for people to ask questions & raise discussions, but that can be done later.

join us and help shape the project. We are working on an initial release, and
expect to post another update with more details at that time.

Before then, here are some ways to get involved:

* Learn more by visiting the [Comet project] page and reading the [mailing list
discussion] about the initial donation.

* Help us plan out the [roadmap]

* Try out the project and provide feedback, file issues, and contribute code.

[mailing list discussion]: https://lists.apache.org/thread/0q1rb11jtpopc7vt1ffdzro0omblsh0s
[roadmap]: https://github.com/apache/arrow-datafusion-comet/issues/19