Code

Driver HA support using WriteAheadLog#14

Code

Closed

tdas wants to merge 12 commits intodriver-hatdas/spark:driver-hafrom driver-ha-workingtdas/spark:driver-ha-workingCopy head branch name to clipboard

+1,630-270

Commits

Commits on Oct 4, 2014

• Initial version of ReceivedDataManager

  tdas

  committed
  a33bf48

  Copy full SHA for a33bf48

• Merge branch 'driver-ha' of github.com:tdas/spark into driver-ha

  tdas

  committed
  51183f6

  Copy full SHA for 51183f6

• Fixed compilation errors.

  tdas

  committed
  a49fd1d

  Copy full SHA for a49fd1d

Commits on Oct 6, 2014

• Added checkpoint data of ReceiverInputDStream, improved WALManager, refactored WALSuite.

  tdas

  committed
  5eafb2e

  Copy full SHA for 5eafb2e

Commits on Oct 7, 2014

• Improved WALSuite, and added ReceivedBlockHandlerSuite.

  tdas

  committed
  b86af2d

  Copy full SHA for b86af2d

Commits on Oct 8, 2014

• Added ReceiverTrackerSuite

  tdas

  committed
  4d9c830

  Copy full SHA for 4d9c830

Commits on Oct 9, 2014

• Initial implementation of DriverFailureSuite unit test for testing with receivers + write ahead log.

  tdas

  committed
  9be5f22

  Copy full SHA for 9be5f22

• Merge remote-tracking branch 'origin/driver-ha' into driver-ha-working

  tdas

  committed
  7ac7325

  Copy full SHA for 7ac7325

• Updated receiver input stream to use HDFSBackedBlockRDD.

  tdas

  committed
  09cda57

  Copy full SHA for 09cda57

• Made WriteAheadLogWriter use local fileoutputstream for local file system.

  tdas

  committed
  e390c71

  Copy full SHA for e390c71

Commits on Oct 21, 2014

• Added ReceivedBlockTracker to track all block metadata related actions. Updated ReceivedBlockTrackerSuite, and DriverFailureSuite.

  tdas

  committed
  6b61287

  Copy full SHA for 6b61287

• Moved around packages.

  tdas

  committed
  acd9467

  Copy full SHA for acd9467