[SPARK-3418] Sparse Matrix support (CCS) and additional native BLAS operations added

[brkyvz]

Local SparseMatrix support added in Compressed Column Storage (CCS) format in addition to Level-2 and Level-3 BLAS operations such as dgemv and dgemm respectively.

BLAS doesn't support sparse matrix operations, therefore support for SparseMatrix-DenseMatrix multiplication and SparseMatrix-DenseVector implementations have been added. I will post performance comparisons in the comments momentarily.

[SparkQA]

QA tests have started for PR 2294 at commit 4362ff1.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2294 at commit 4362ff1.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparseMatrix(val numRows: Int,
  • case class AddJar(path: String) extends LeafNode with Command

[SparkQA]

QA tests have started for PR 2294 at commit 8dcb763.

• This patch merges cleanly.

[brkyvz]

The following tests were run on a Mac OS X 10.9.3
2.8 GHz Intel Core i7
8 GB 1600 MHz DDR3.

Hyper-threading was disabled and turbo boost was turned off. Effective numRows or Effective numCols stand for the effective number of rows or columns that are used in the computation. As the transpose multiplications are also used, that means that the number of rows of A, which has dimensions m x n may be m, but the effective number of rows are n.

[SparkQA]

QA tests have finished for PR 2294 at commit 8dcb763.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparseMatrix(val numRows: Int,

[SparkQA]

QA tests have started for PR 2294 at commit 41b2da3.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2294 at commit 41b2da3.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparseMatrix(val numRows: Int,

[SparkQA]

QA tests have started for PR 2294 at commit 56d7c85.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2294 at commit 56d7c85.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparseMatrix(

[SparkQA]

QA tests have started for PR 2294 at commit 848406c.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2294 at commit 848406c.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparseMatrix(

[mengxr]

test this please

[SparkQA]

QA tests have started for PR 2294 at commit 848406c.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2294 at commit 848406c.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparseMatrix(

[mengxr]

Change type to Boolean?

[SparkQA]

QA tests have finished for PR 2294 at commit d162684.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • sealed trait Matrix extends Serializable
  • class SparseMatrix(
  • sealed trait Vector extends Serializable
  • Seq( // Ignore new functions added to traitMatrix. They are sealed traits now.

[SparkQA]

QA tests have started for PR 2294 at commit 272feb9.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2294 at commit 272feb9.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • sealed trait Matrix extends Serializable
  • class SparseMatrix(
  • sealed trait Vector extends Serializable

[brkyvz]

@ScrapCodes, I have some mima incompatibility issues here. I've added a few methods to the trait Matrix in MLlib, and that causes a MissingMethodProblem. I sealed the trait, and added the exclude statements to the MimaExcludes, but the tests still fail.
dev/mima passes on my local machine.
Do you have any idea of what might be going on? The commits d162684 and
272feb9 were to address this issue.

Can you please take a look?

[ScrapCodes]

Can you rebase to the tip of master and add tests in version 1.2 section:
https://github.com/apache/spark/blob/master/project/MimaExcludes.scala#L37

instead of 1.1.

Prashant Sharma

On Fri, Sep 19, 2014 at 4:42 AM, Burak Yavuz [email protected]
wrote:

@ScrapCodes https://github.com/ScrapCodes, I have some mima
incompatibility issues here. I've added a few methods to the trait Matrix
in MLlib, and that causes a MissingMethodProblem. I sealed the trait, and
added the exclude statements to the MimaExcludes, but the tests still fail.
dev/mima passes on my local machine.
Do you have any idea of what might be going on? The commits d162684
d162684
and
272feb9
272feb9
were to address this issue.

Can you please take a look?

—
Reply to this email directly or view it on GitHub
#2294 (comment).

[SparkQA]

QA tests have started for PR 2294 at commit 88814ed.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2294 at commit 88814ed.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • sealed trait Matrix extends Serializable
  • class SparseMatrix(
  • sealed trait Vector extends Serializable

[brkyvz]

@ScrapCodes THANKS A LOT! That fixed it! I didn't realize I didn't update my local repo for such a long time.

[mengxr]

LGTM. I'm merging this into master. Thanks! (We might need to make slight changes to some methods before the 1.2 release, but let's not block the multi-model training PR for now.)