log sizes of created smoosh files #3817
Conversation
| @@ -695,9 +695,12 @@ public void doRun() | |||
| for (File file : toMerge) { | |||
| indexes.add(HadoopDruidIndexerConfig.INDEX_IO.loadIndex(file)); | |||
| } | |||
|
|
|||
| log.info("starting merge of intermediate persisted segments..."); | |||
There was a problem hiding this comment.
this is only going to get printed once during batch ingestion and most of the time users are not running indexing with debug level. its not going to pollute the logs really.
There was a problem hiding this comment.
Ok, can the formatting be made more consistent with other logging messages. like simply
Starting merge of intermediate persisted segments
There was a problem hiding this comment.
Same request with other logging statements
| mergedBase = mergeQueryableIndex( | ||
| indexes, aggregators, new File(baseFlushFile, "merged"), progressIndicator | ||
| ); | ||
| log.info("finished merge of intermediate persisted segments..."); |
There was a problem hiding this comment.
same question, can this be debug level?
There was a problem hiding this comment.
same reason as above
|
|
||
| FileOutputStream outStream = new FileOutputStream(outFile); | ||
| this.channel = outStream.getChannel(); | ||
| closer.register(outStream); |
There was a problem hiding this comment.
can you move the closer to the same place the file output stream is created?
final FileOutputStream outStream = closer.register(new FileOutputStream(outFile));should work
| @@ -494,6 +498,7 @@ public boolean isOpen() | |||
| public void close() throws IOException | |||
| { | |||
| closer.close(); | |||
| FileSmoosher.LOG.info("Created smoosh file [%s] of size [%s] bytes.", outFile.getAbsolutePath(), outFile.length()); | |||
There was a problem hiding this comment.
Same question, can this be debug level?
There was a problem hiding this comment.
same reason.... mostly we look at logs afterwards where most jobs would've run without debug level logs.
himanshug
force-pushed
the
dump_segment_sizes
branch
from
e847be7 to
8dceb26
Compare
| @@ -695,9 +695,12 @@ public void doRun() | |||
| for (File file : toMerge) { | |||
| indexes.add(HadoopDruidIndexerConfig.INDEX_IO.loadIndex(file)); | |||
| } | |||
|
|
|||
| log.info("starting merge of intermediate persisted segments..."); | |||
There was a problem hiding this comment.
Ok, can the formatting be made more consistent with other logging messages. like simply
Starting merge of intermediate persisted segments
| @@ -695,9 +695,12 @@ public void doRun() | |||
| for (File file : toMerge) { | |||
| indexes.add(HadoopDruidIndexerConfig.INDEX_IO.loadIndex(file)); | |||
| } | |||
|
|
|||
| log.info("starting merge of intermediate persisted segments..."); | |||
There was a problem hiding this comment.
Same request with other logging statements
|
Minor logging format request, 👍 after that |
himanshug
force-pushed
the
dump_segment_sizes
branch
from
8dceb26 to
4b001dd
Compare
|
@drcrallen updated formatting. |
| mergedBase = mergeQueryableIndex( | ||
| indexes, aggregators, new File(baseFlushFile, "merged"), progressIndicator | ||
| ); | ||
| log.info("finished merge of intermediate persisted segments."); |
himanshug
force-pushed
the
dump_segment_sizes
branch
from
4b001dd to
98a86de
Compare
* log when merging of intermediate segments starts during batch ingestion * log sizes of created smoosh files
just adding logs to primarily understand impact of changing maxRowsInMemory setting during batch ingestion on size of intermediate segments created.