-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MINOR] Fix usages of orElse #10435
[MINOR] Fix usages of orElse #10435
Changes from 2 commits
072ac26
402d1eb
ca74f15
4f9744e
2684527
9674bfb
cc0c686
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -107,23 +107,19 @@ object HoodieSparkUtils extends SparkAdapterSupport with SparkVersionsSupport wi | |
// injecting [[SQLConf]], which by default isn't propagated by Spark to the executor(s). | ||
// [[SQLConf]] is required by [[AvroSerializer]] | ||
injectSQLConf(df.queryExecution.toRdd.mapPartitions { rows => | ||
if (rows.isEmpty) { | ||
Iterator.empty | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does removal of this provide any benefit? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, this will trigger the dag to see if it is in fact returning an empty set of rows. You can see this on the spark UI when running your jobs. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was looking at the wrong place. This is the spot that needed to be updated: https://github.com/apache/hudi/pull/10435/files#diff-21ccee08cebace9de801e104f356bf4333c017d5d5c0cac4e3de63c7861f3c13R100 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Got it. |
||
} else { | ||
val readerAvroSchema = new Schema.Parser().parse(readerAvroSchemaStr) | ||
val transform: GenericRecord => GenericRecord = | ||
if (sameSchema) identity | ||
else { | ||
HoodieAvroUtils.rewriteRecordDeep(_, readerAvroSchema) | ||
} | ||
val readerAvroSchema = new Schema.Parser().parse(readerAvroSchemaStr) | ||
val transform: GenericRecord => GenericRecord = | ||
if (sameSchema) identity | ||
else { | ||
HoodieAvroUtils.rewriteRecordDeep(_, readerAvroSchema) | ||
} | ||
|
||
// Since caller might request to get records in a different ("evolved") schema, we will be rewriting from | ||
// existing Writer's schema into Reader's (avro) schema | ||
val writerAvroSchema = new Schema.Parser().parse(writerAvroSchemaStr) | ||
val convert = AvroConversionUtils.createInternalRowToAvroConverter(writerSchema, writerAvroSchema, nullable = nullable) | ||
// Since caller might request to get records in a different ("evolved") schema, we will be rewriting from | ||
// existing Writer's schema into Reader's (avro) schema | ||
val writerAvroSchema = new Schema.Parser().parse(writerAvroSchemaStr) | ||
val convert = AvroConversionUtils.createInternalRowToAvroConverter(writerSchema, writerAvroSchema, nullable = nullable) | ||
|
||
rows.map { ir => transform(convert(ir)) } | ||
} | ||
rows.map { ir => transform(convert(ir)) } | ||
}, SQLConf.get) | ||
} | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, so
#orElseGet
is always preferrable than#orElse
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, you do not want to execute methods or create objects you will not use. Therefore you can use
orElse
when returning a constant but otherwise you should avoid it.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch!