GraphFrames 0.8.3

Major Changes

• Support Spark 3.3 / Scala 2.12 , Spark 3.4 / Scala 2.12 and Scala 2.13, Spark 3.5 / Scala 2.12 and Scala 2.13
• Remove deprecated SQLContext from code

What's Changed

• Use official Spark 3.2 PyPI release in CI
• Upgrade to latest spark versions 3.3.0, 3.2.2 and 3.1.3
• Python: Replace SQLContext with SparkSession
• Update CI actions and dependencies
• Add support for scala 2.13 (
• Remove deprecated sqlContext in graphframe scala code
• Spark 3.4.1 and drop EOL Spark versions 3.0 and 3.1

New Contributors

• eejbyfeldt
• ttc0419

Full Changelog: v0.8.2...v0.8.3

GraphFrames 0.8.2

Major Changes

• Support Spark 3.1 / Scala 2.12 and Spark 3.2 / Scala 2.12.

What's Changed

• [ML-12428] Upgrade CI spark version to 3.0.1 by @WeichenXu123 in #372
• Extend to Spark 3.1 by @EnricoMi in #381
• Correct the order of the analogy by @johnnyleitrim in #377
• Replace bintray by @menglewis in #385
• Switch from Travis CI to GitHub Actions by @mengxr in #402
• Build against Spark 3.2 by @mengxr in #400

New Contributors

• @EnricoMi made their first contribution in #381
• @johnnyleitrim made their first contribution in #377
• @menglewis made their first contribution in #385

Full Changelog: v0.8.1...v0.8.2

GraphFrames 0.8.1

• Build against official Spark 3.0.0 release.
• Cross build: Spark 3.0/Scala 2.12, Spark 2.4/Scala 2.11, Spark 2.4/Scala 2.12 (Experimental).
• Fixed an error in shortest path (#367).
• Removed the use of deprecated methods (#363).

GraphFrames 0.8.0

• Support Spark 2.4 / Scala 2.11 and Spark 3.0 (Preview) / Scala 2.12.

GraphFrames 0.7.0

• Pregel API.
• Support Spark 2.4, Python 3.6, and removed support for Spark 2.2.
• Removed scala-logging-slf4j from dependencies.

GraphFrames 0.6.0

We encourage all users to upgrade to this release because of motif finding bug fixes and fixes to the release process for the Python API!

Bug fixes

• [Correctness] Connected Components had a bug which affected correctness in some cases. It has been fixed.
• All previous versions of GraphFrames could encounter failures with motif finding using negated edges. These were not correctness issues, but they would cause Exceptions on perfectly valid queries.
• The last release was missing Python files required for the Python API. This release fixes that issue.

New features

• Filter vertices or edges to select subgraphs

Support for Apache Spark 2.2 and 2.3

Thanks to the many contributors to this release!

GraphFrames 0.5.0

We strongly encourage all users to use this latest release because of the bug fix described below!

CRITICAL bug fix

• All previous versions of GraphFrames had a bug which can affect ConnectedComponents and other algorithms.
• This fixes a bug in indexing vertices with non-Integer IDs.
• It can affect all algorithms which are wrappers around GraphX, including ConnectedComponents, PageRank, and others.
• The bug surfaces when the input DataFrame is non-deterministic. E.g., running an algorithm on a DataFrame just loaded from disk should be fine in previous releases, but running that algorithm on a DataFrame produced using shuffling, unions, and other operators can cause incorrect results. This issue is fixed in this release.

New features

• Python API for aggregateMessages for building custom graph algorithms
• Scala API for parallel personalized PageRank, wrapping the GraphX implementation. This is only available when using GraphFrames with Spark 2.1+.

Support for Spark 1.6, 2.0, and 2.1

Contributors to this release

• Felix Cheung
• Tuomas Sivula
• Xiangrui Meng
• Joseph Bradley
• Bagrat Amirbekian
• Santiago Castro
• Philip Yang

GraphFrames 0.4.0

New release of GraphFrames for Apache Spark 2.0 and 2.1

• Minor fix for checkpointing issue in DataFrame-based connected components implementation (issue #160)

GraphFrames 0.3.0

• DataFrame-based connected components implementation
  • Users can fall back to GraphX implementation as needed
• removed support for Spark 1.4 and 1.5

GraphFrames: DataFrame-based Graphs for Apache Spark, release 0.2.0

New methods (in Scala, Java and Python APIs)

• cache, persist, unpersist
• triplets

Improvements

• Motif finding (find() method)
  • Result DataFrame now orders the columns in the same order specified by the motif.
  • more robust internals (intermediate column naming)
• Various documentation and example fixes
• Compatibility with Apache Spark 1.4, 1.5, 1.6, 2.0 (with Scala 2.10), as well as Apache Spark 2.0 with Scala 2.11

Contributors to this release

• Tim Hunter
• Felix Cheung
• Joseph Bradley
• Xiangrui Meng
• Shagun Sodhani
• Frederick Lefebvre
• Qingpeng "Q.P." Zhang
• Bill Chambers
• Reynold Xin