Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update spark-avro dependency #75

Closed
wants to merge 3 commits into from
Closed

Update spark-avro dependency #75

wants to merge 3 commits into from

Conversation

marmbrus
Copy link
Contributor

@marmbrus marmbrus commented Sep 8, 2015

No description provided.

@JoshRosen
Copy link
Contributor

java.lang.ClassCastException: org.apache.hadoop.mapreduce.Job cannot be cast to org.apache.hadoop.mapred.JobContext
    at org.apache.hadoop.mapred.OutputCommitter.setupJob(OutputCommitter.java:213)
    at org.apache.spark.sql.sources.BaseWriterContainer.driverSideSetup(commands.scala:351)
    at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.insert(commands.scala:157)
    at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.run(commands.scala:139)
    at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
    at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
    at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)
    at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:950)
    at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:950)

Looks like a symptom of a possible Hadoop version mismatch?

@JoshRosen
Copy link
Contributor

I'm thinking that this could actually be a classloading issue because otherwise the error message doesn't make a lot of sense to me. The error is org.apache.hadoop.mapreduce.Job cannot be cast to org.apache.hadoop.mapred.JobContext, but Job implements mapreduce.JobContext, which in turn extends mapred.JobContext.

@JoshRosen JoshRosen changed the title Update avro dependency Update spark-avro dependency Sep 8, 2015
@JoshRosen JoshRosen added this to the 0.5.0 milestone Sep 8, 2015
@codecov-io
Copy link

Current coverage is 94.90%

Merging #75 into master will decrease coverage by -0.05% as of 85e5560

@@            master    #75   diff @@
=====================================
  Files           11     11       
  Stmts          436    432     -4
  Branches       100    100       
  Methods          0      0       
=====================================
- Hit            414    410     -4
  Partial          0      0       
  Missed          22     22       

Review entire Coverage Diff as of 85e5560

Powered by Codecov. Updated on successful CI builds.

@JoshRosen
Copy link
Contributor

Spotted the problem and fixed things, so I'm going to merge this fix now. We can bump to 2.0.1 in a separate commit.

@JoshRosen JoshRosen closed this in a9198c3 Sep 8, 2015
@JoshRosen JoshRosen deleted the newAvro branch September 8, 2015 20:13
munk pushed a commit to ActionIQ-OSS/spark-redshift that referenced this pull request May 4, 2021
…ump-to-4.1.1

Bump to 4.1.1 to cross publish for scala 2.12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants