SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/hadoop-root/nm-local-dir/usercache/root/filecache/11/__spark_libs__5487253704463664943.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/cluster/opt/hadoop-2.6.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/05/30 02:18:22 INFO util.SignalUtils: Registered signal handler for TERM 17/05/30 02:18:22 INFO util.SignalUtils: Registered signal handler for HUP 17/05/30 02:18:22 INFO util.SignalUtils: Registered signal handler for INT 17/05/30 02:18:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/05/30 02:18:22 INFO yarn.ApplicationMaster: Preparing Local resources 17/05/30 02:18:23 INFO yarn.ApplicationMaster: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "master" port: -1 file: "/user/root/.sparkStaging/application_1496110651001_0002/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar" } size: 2508346 timestamp: 1496110698186 type: FILE visibility: PRIVATE, lenet_memory_solver.prototxt -> resource { scheme: "hdfs" host: "master" port: -1 file: "/user/root/.sparkStaging/application_1496110651001_0002/lenet_memory_solver.prototxt" } size: 720 timestamp: 1496110698220 type: FILE visibility: PRIVATE, __spark_libs__ -> resource { scheme: "hdfs" host: "master" port: -1 file: "/user/root/.sparkStaging/application_1496110651001_0002/__spark_libs__5487253704463664943.zip" } size: 189213980 timestamp: 1496110698056 type: ARCHIVE visibility: PRIVATE, lenet_memory_train_test.prototxt -> resource { scheme: "hdfs" host: "master" port: -1 file: "/user/root/.sparkStaging/application_1496110651001_0002/lenet_memory_train_test.prototxt" } size: 2569 timestamp: 1496110698268 type: FILE visibility: PRIVATE, __spark_conf__ -> resource { scheme: "hdfs" host: "master" port: -1 file: "/user/root/.sparkStaging/application_1496110651001_0002/__spark_conf__.zip" } size: 86951 timestamp: 1496110698313 type: ARCHIVE visibility: PRIVATE) 17/05/30 02:18:23 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1496110651001_0002_000001 17/05/30 02:18:23 INFO spark.SecurityManager: Changing view acls to: root 17/05/30 02:18:23 INFO spark.SecurityManager: Changing modify acls to: root 17/05/30 02:18:23 INFO spark.SecurityManager: Changing view acls groups to: 17/05/30 02:18:23 INFO spark.SecurityManager: Changing modify acls groups to: 17/05/30 02:18:23 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 17/05/30 02:18:23 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread 17/05/30 02:18:23 INFO yarn.ApplicationMaster: Waiting for spark context initialization 17/05/30 02:18:23 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 17/05/30 02:18:23 INFO spark.SparkContext: Running Spark version 2.0.0 17/05/30 02:18:23 INFO spark.SecurityManager: Changing view acls to: root 17/05/30 02:18:23 INFO spark.SecurityManager: Changing modify acls to: root 17/05/30 02:18:23 INFO spark.SecurityManager: Changing view acls groups to: 17/05/30 02:18:23 INFO spark.SecurityManager: Changing modify acls groups to: 17/05/30 02:18:23 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 17/05/30 02:18:23 INFO util.Utils: Successfully started service 'sparkDriver' on port 46722. 17/05/30 02:18:23 INFO spark.SparkEnv: Registering MapOutputTracker 17/05/30 02:18:23 INFO spark.SparkEnv: Registering BlockManagerMaster 17/05/30 02:18:23 INFO storage.DiskBlockManager: Created local directory at /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/blockmgr-76bf5a23-e13a-4fa0-a64f-1d35d1b6e9b7 17/05/30 02:18:23 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB 17/05/30 02:18:23 INFO spark.SparkEnv: Registering OutputCommitCoordinator 17/05/30 02:18:23 INFO util.log: Logging initialized @2227ms 17/05/30 02:18:23 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 17/05/30 02:18:23 INFO server.Server: jetty-9.2.z-SNAPSHOT 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1f15e41c{/jobs,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@27ee1748{/jobs/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3e4ae082{/jobs/job,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5fd2dc31{/jobs/job/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6f57f6d3{/stages,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@38d4f826{/stages/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@51f3514e{/stages/stage,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@18892c15{/stages/stage/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@48b8b0b9{/stages/pool,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@57b26440{/stages/pool/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1c8036d9{/storage,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4db858d8{/storage/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@400804ae{/storage/rdd,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@45cf88b{/storage/rdd/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@19772027{/environment,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1606d549{/environment/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@162b9e53{/executors,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@75d0b4bd{/executors/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5586c438{/executors/threadDump,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@758aef5c{/executors/threadDump/json,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3f993eb8{/static,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4f9c00aa{/,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2cd8ba65{/api,null,AVAILABLE} 17/05/30 02:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@e0a40ad{/stages/stage/kill,null,AVAILABLE} 17/05/30 02:18:23 INFO server.ServerConnector: Started ServerConnector@1131c988{HTTP/1.1}{0.0.0.0:32983} 17/05/30 02:18:23 INFO server.Server: Started @2350ms 17/05/30 02:18:23 INFO util.Utils: Successfully started service 'SparkUI' on port 32983. 17/05/30 02:18:23 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://slave2:32983 17/05/30 02:18:24 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler 17/05/30 02:18:24 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1496110651001_0002 and attemptId Some(appattempt_1496110651001_0002_000001) 17/05/30 02:18:24 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44524. 17/05/30 02:18:24 INFO netty.NettyBlockTransferService: Server created on slave2:44524 17/05/30 02:18:24 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, slave2, 44524) 17/05/30 02:18:24 INFO storage.BlockManagerMasterEndpoint: Registering block manager slave2:44524 with 366.3 MB RAM, BlockManagerId(driver, slave2, 44524) 17/05/30 02:18:24 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, slave2, 44524) 17/05/30 02:18:24 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@40d016ca{/metrics/json,null,AVAILABLE} 17/05/30 02:18:24 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@slave2:46722) 17/05/30 02:18:24 INFO client.RMProxy: Connecting to ResourceManager at /master:8030 17/05/30 02:18:24 INFO yarn.YarnRMClient: Registering the ApplicationMaster 17/05/30 02:18:24 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead 17/05/30 02:18:24 INFO yarn.YarnAllocator: Canceled 0 container requests (locality no longer needed) 17/05/30 02:18:24 INFO yarn.YarnAllocator: Submitted container request (host: Any, capability: ) 17/05/30 02:18:24 INFO yarn.YarnAllocator: Submitted container request (host: Any, capability: ) 17/05/30 02:18:24 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals 17/05/30 02:18:25 INFO impl.AMRMClientImpl: Received new token for : slave2:43525 17/05/30 02:18:25 INFO impl.AMRMClientImpl: Received new token for : slave1:42569 17/05/30 02:18:25 INFO yarn.YarnAllocator: Launching container container_1496110651001_0002_01_000002 for on host slave2 17/05/30 02:18:25 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@slave2:46722, executorHostname: slave2 17/05/30 02:18:25 INFO yarn.YarnAllocator: Launching container container_1496110651001_0002_01_000003 for on host slave1 17/05/30 02:18:25 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@slave2:46722, executorHostname: slave1 17/05/30 02:18:25 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. 17/05/30 02:18:25 INFO yarn.ExecutorRunnable: Starting Executor Container 17/05/30 02:18:25 INFO yarn.ExecutorRunnable: Starting Executor Container 17/05/30 02:18:25 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/05/30 02:18:25 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/05/30 02:18:25 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext 17/05/30 02:18:25 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext 17/05/30 02:18:25 INFO yarn.ExecutorRunnable: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}{{PWD}}/__spark_conf__{{PWD}}/__spark_libs__/*$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* SPARK_LOG_URL_STDERR -> http://slave2:8042/node/containerlogs/container_1496110651001_0002_01_000002/root/stderr?start=-4096 SPARK_YARN_STAGING_DIR -> hdfs://master/user/root/.sparkStaging/application_1496110651001_0002 SPARK_USER -> root SPARK_YARN_MODE -> true SPARK_HOME -> /home/cluster/opt/spark-2.0.0-bin-hadoop2.6 LD_LIBRARY_PATH -> /usr/local/lib/:/home/cluster/opt/CaffeOnSpark/caffe-public/distribute/lib:/home/cluster/opt/CaffeOnSpark/caffe-distri/distribute/lib:/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/ia32_lin/:/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin/ SPARK_LOG_URL_STDOUT -> http://slave2:8042/node/containerlogs/container_1496110651001_0002_01_000002/root/stdout?start=-4096 command: {{JAVA_HOME}}/bin/java -server -Xmx1024m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.ui.port=0' '-Dspark.driver.port=46722' -Dspark.yarn.app.container.log.dir= -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@slave2:46722 --executor-id 1 --hostname slave2 --cores 1 --app-id application_1496110651001_0002 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr =============================================================================== 17/05/30 02:18:25 INFO yarn.ExecutorRunnable: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}{{PWD}}/__spark_conf__{{PWD}}/__spark_libs__/*$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* SPARK_LOG_URL_STDERR -> http://slave1:8042/node/containerlogs/container_1496110651001_0002_01_000003/root/stderr?start=-4096 SPARK_YARN_STAGING_DIR -> hdfs://master/user/root/.sparkStaging/application_1496110651001_0002 SPARK_USER -> root SPARK_YARN_MODE -> true SPARK_HOME -> /home/cluster/opt/spark-2.0.0-bin-hadoop2.6 LD_LIBRARY_PATH -> /usr/local/lib/:/home/cluster/opt/CaffeOnSpark/caffe-public/distribute/lib:/home/cluster/opt/CaffeOnSpark/caffe-distri/distribute/lib:/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/ia32_lin/:/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin/ SPARK_LOG_URL_STDOUT -> http://slave1:8042/node/containerlogs/container_1496110651001_0002_01_000003/root/stdout?start=-4096 command: {{JAVA_HOME}}/bin/java -server -Xmx1024m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.ui.port=0' '-Dspark.driver.port=46722' -Dspark.yarn.app.container.log.dir= -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@slave2:46722 --executor-id 2 --hostname slave1 --cores 1 --app-id application_1496110651001_0002 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr =============================================================================== 17/05/30 02:18:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : slave1:42569 17/05/30 02:18:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : slave2:43525 17/05/30 02:18:27 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (slave2:48106) with ID 1 17/05/30 02:18:27 INFO storage.BlockManagerMasterEndpoint: Registering block manager slave2:42368 with 366.3 MB RAM, BlockManagerId(1, slave2, 42368) 17/05/30 02:18:29 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (slave1:46370) with ID 2 17/05/30 02:18:29 INFO storage.BlockManagerMasterEndpoint: Registering block manager slave1:37625 with 366.3 MB RAM, BlockManagerId(2, slave1, 37625) 17/05/30 02:18:29 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 1.0 17/05/30 02:18:29 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 17/05/30 02:18:30 INFO caffe.DataSource$: Source data layer:0 17/05/30 02:18:30 INFO caffe.LMDB: Batch size:32 17/05/30 02:18:31 INFO spark.SparkContext: Starting job: collect at CaffeOnSpark.scala:123 17/05/30 02:18:31 INFO scheduler.DAGScheduler: Got job 0 (collect at CaffeOnSpark.scala:123) with 2 output partitions 17/05/30 02:18:31 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (collect at CaffeOnSpark.scala:123) 17/05/30 02:18:31 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:18:31 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:18:31 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at map at CaffeOnSpark.scala:113), which has no missing parents 17/05/30 02:18:31 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.1 KB, free 366.3 MB) 17/05/30 02:18:31 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.6 KB, free 366.3 MB) 17/05/30 02:18:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slave2:44524 (size: 2.6 KB, free: 366.3 MB) 17/05/30 02:18:31 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1012 17/05/30 02:18:31 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at map at CaffeOnSpark.scala:113) 17/05/30 02:18:31 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 2 tasks 17/05/30 02:18:31 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, slave2, partition 0, PROCESS_LOCAL, 5232 bytes) 17/05/30 02:18:31 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, slave1, partition 1, PROCESS_LOCAL, 5232 bytes) 17/05/30 02:18:31 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 0 on executor id: 1 hostname: slave2. 17/05/30 02:18:31 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 1 on executor id: 2 hostname: slave1. 17/05/30 02:18:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slave1:37625 (size: 2.6 KB, free: 366.3 MB) 17/05/30 02:18:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slave2:42368 (size: 2.6 KB, free: 366.3 MB) 17/05/30 02:18:32 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 876 ms on slave1 (1/2) 17/05/30 02:18:32 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 907 ms on slave2 (2/2) 17/05/30 02:18:32 INFO cluster.YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 17/05/30 02:18:32 INFO scheduler.DAGScheduler: ResultStage 0 (collect at CaffeOnSpark.scala:123) finished in 0.916 s 17/05/30 02:18:32 INFO scheduler.DAGScheduler: Job 0 finished: collect at CaffeOnSpark.scala:123, took 1.177096 s 17/05/30 02:18:32 INFO caffe.CaffeOnSpark: rank = 0, address = ,slave2:55328, hostname = slave2 17/05/30 02:18:32 INFO caffe.CaffeOnSpark: rank = 1, address = slave1:48274,, hostname = slave1 17/05/30 02:18:32 INFO caffe.CaffeOnSpark: rank 0:slave2 17/05/30 02:18:32 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 520.0 B, free 366.3 MB) 17/05/30 02:18:32 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 156.0 B, free 366.3 MB) 17/05/30 02:18:32 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on slave2:44524 (size: 156.0 B, free: 366.3 MB) 17/05/30 02:18:32 INFO spark.SparkContext: Created broadcast 1 from broadcast at CaffeOnSpark.scala:142 17/05/30 02:18:32 INFO spark.SparkContext: Starting job: collect at CaffeOnSpark.scala:154 17/05/30 02:18:32 INFO scheduler.DAGScheduler: Got job 1 (collect at CaffeOnSpark.scala:154) with 2 output partitions 17/05/30 02:18:32 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (collect at CaffeOnSpark.scala:154) 17/05/30 02:18:32 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:18:32 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:18:32 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at map at CaffeOnSpark.scala:145), which has no missing parents 17/05/30 02:18:32 INFO memory.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 366.3 MB) 17/05/30 02:18:32 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1560.0 B, free 366.3 MB) 17/05/30 02:18:32 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on slave2:44524 (size: 1560.0 B, free: 366.3 MB) 17/05/30 02:18:32 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1012 17/05/30 02:18:32 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at map at CaffeOnSpark.scala:145) 17/05/30 02:18:32 INFO cluster.YarnClusterScheduler: Adding task set 1.0 with 2 tasks 17/05/30 02:18:32 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, slave2, partition 0, PROCESS_LOCAL, 5232 bytes) 17/05/30 02:18:32 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, slave1, partition 1, PROCESS_LOCAL, 5232 bytes) 17/05/30 02:18:32 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 2 on executor id: 1 hostname: slave2. 17/05/30 02:18:32 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 3 on executor id: 2 hostname: slave1. 17/05/30 02:18:32 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on slave2:42368 (size: 1560.0 B, free: 366.3 MB) 17/05/30 02:18:32 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on slave1:37625 (size: 1560.0 B, free: 366.3 MB) 17/05/30 02:18:32 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on slave1:37625 (size: 156.0 B, free: 366.3 MB) 17/05/30 02:18:32 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on slave2:42368 (size: 156.0 B, free: 366.3 MB) 17/05/30 02:18:33 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 1078 ms on slave1 (1/2) 17/05/30 02:18:33 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 1086 ms on slave2 (2/2) 17/05/30 02:18:33 INFO cluster.YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 17/05/30 02:18:33 INFO scheduler.DAGScheduler: ResultStage 1 (collect at CaffeOnSpark.scala:154) finished in 1.086 s 17/05/30 02:18:33 INFO scheduler.DAGScheduler: Job 1 finished: collect at CaffeOnSpark.scala:154, took 1.095393 s 17/05/30 02:18:33 INFO util.Utils: Fetching hdfs://master/projects/machine_learning/image_dataset/mnist_train_lmdb/data.mdb to /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/spark-af9b6dcf-5b64-42f5-b512-7c3e04466776/userFiles-75588463-5a16-4ad0-a5ec-b34b34c0c4a3/mnist_train_lmdb/fetchFileTemp4976145142600396272.tmp 17/05/30 02:18:33 INFO util.Utils: Fetching hdfs://master/projects/machine_learning/image_dataset/mnist_train_lmdb/lock.mdb to /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/spark-af9b6dcf-5b64-42f5-b512-7c3e04466776/userFiles-75588463-5a16-4ad0-a5ec-b34b34c0c4a3/mnist_train_lmdb/fetchFileTemp6780863289525981199.tmp 17/05/30 02:18:33 INFO spark.SparkContext: Added file hdfs:///projects/machine_learning/image_dataset/mnist_train_lmdb/ at hdfs:///projects/machine_learning/image_dataset/mnist_train_lmdb/ with timestamp 1496110713553 17/05/30 02:18:33 INFO caffe.LmdbRDD: local LMDB path:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/spark-af9b6dcf-5b64-42f5-b512-7c3e04466776/userFiles-75588463-5a16-4ad0-a5ec-b34b34c0c4a3/mnist_train_lmdb 17/05/30 02:18:33 INFO caffe.LmdbRDD: 2 LMDB RDD partitions 17/05/30 02:18:33 INFO spark.SparkContext: Starting job: min at CaffeOnSpark.scala:198 17/05/30 02:18:33 INFO scheduler.DAGScheduler: Got job 2 (min at CaffeOnSpark.scala:198) with 2 output partitions 17/05/30 02:18:33 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (min at CaffeOnSpark.scala:198) 17/05/30 02:18:33 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:18:33 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:18:33 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[6] at mapPartitions at CaffeOnSpark.scala:187), which has no missing parents 17/05/30 02:18:33 INFO memory.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.9 KB, free 366.3 MB) 17/05/30 02:18:33 INFO memory.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:18:33 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:18:33 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1012 17/05/30 02:18:33 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 2 (MapPartitionsRDD[6] at mapPartitions at CaffeOnSpark.scala:187) 17/05/30 02:18:33 INFO cluster.YarnClusterScheduler: Adding task set 2.0 with 2 tasks 17/05/30 02:18:33 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, slave2, partition 0, PROCESS_LOCAL, 5219 bytes) 17/05/30 02:18:33 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 5, slave1, partition 1, PROCESS_LOCAL, 5236 bytes) 17/05/30 02:18:33 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 4 on executor id: 1 hostname: slave2. 17/05/30 02:18:33 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 5 on executor id: 2 hostname: slave1. 17/05/30 02:18:34 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added rdd_1_0 on disk on slave2:42368 (size: 23.2 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added rdd_1_1 on disk on slave1:37625 (size: 23.2 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added rdd_6_1 in memory on slave1:37625 (size: 24.0 B, free: 366.3 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added rdd_6_0 in memory on slave2:42368 (size: 24.0 B, free: 366.3 MB) 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 1619 ms on slave2 (1/2) 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 5) in 1617 ms on slave1 (2/2) 17/05/30 02:18:35 INFO cluster.YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool 17/05/30 02:18:35 INFO scheduler.DAGScheduler: ResultStage 2 (min at CaffeOnSpark.scala:198) finished in 1.619 s 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Job 2 finished: min at CaffeOnSpark.scala:198, took 1.631385 s 17/05/30 02:18:35 INFO spark.SparkContext: Starting job: max at CaffeOnSpark.scala:199 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Got job 3 (max at CaffeOnSpark.scala:199) with 2 output partitions 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (max at CaffeOnSpark.scala:199) 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[6] at mapPartitions at CaffeOnSpark.scala:187), which has no missing parents 17/05/30 02:18:35 INFO memory.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.9 KB, free 366.3 MB) 17/05/30 02:18:35 INFO memory.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.5 KB, free 366.3 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on slave2:44524 (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:18:35 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1012 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 3 (MapPartitionsRDD[6] at mapPartitions at CaffeOnSpark.scala:187) 17/05/30 02:18:35 INFO cluster.YarnClusterScheduler: Adding task set 3.0 with 2 tasks 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 6, slave2, partition 0, PROCESS_LOCAL, 5219 bytes) 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 7, slave1, partition 1, PROCESS_LOCAL, 5236 bytes) 17/05/30 02:18:35 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 6 on executor id: 1 hostname: slave2. 17/05/30 02:18:35 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 7 on executor id: 2 hostname: slave1. 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on slave2:42368 (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on slave1:37625 (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 6) in 36 ms on slave2 (1/2) 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 7) in 34 ms on slave1 (2/2) 17/05/30 02:18:35 INFO cluster.YarnClusterScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool 17/05/30 02:18:35 INFO scheduler.DAGScheduler: ResultStage 3 (max at CaffeOnSpark.scala:199) finished in 0.037 s 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Job 3 finished: max at CaffeOnSpark.scala:199, took 0.053778 s 17/05/30 02:18:35 INFO caffe.CaffeOnSpark: Partition size: min=30000 max=30000 17/05/30 02:18:35 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Got job 4 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Final stage: ResultStage 4 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[7] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:18:35 INFO memory.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:18:35 INFO memory.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:18:35 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1012 17/05/30 02:18:35 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 4 (MapPartitionsRDD[7] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:18:35 INFO cluster.YarnClusterScheduler: Adding task set 4.0 with 2 tasks 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 8, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:18:35 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 9, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:18:35 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 8 on executor id: 1 hostname: slave2. 17/05/30 02:18:35 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 9 on executor id: 2 hostname: slave1. 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:18:35 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:20:03 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 87778 ms on slave1 (1/2) 17/05/30 02:20:03 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 87794 ms on slave2 (2/2) 17/05/30 02:20:03 INFO cluster.YarnClusterScheduler: Removed TaskSet 4.0, whose tasks have all completed, from pool 17/05/30 02:20:03 INFO scheduler.DAGScheduler: ResultStage 4 (reduce at CaffeOnSpark.scala:226) finished in 87.795 s 17/05/30 02:20:03 INFO scheduler.DAGScheduler: Job 4 finished: reduce at CaffeOnSpark.scala:226, took 87.805648 s 17/05/30 02:20:03 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:20:03 INFO scheduler.DAGScheduler: Got job 5 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:20:03 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:20:03 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:20:03 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:20:03 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[8] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:20:03 INFO memory.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:20:03 INFO memory.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:20:03 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:20:03 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1012 17/05/30 02:20:03 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 5 (MapPartitionsRDD[8] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:20:03 INFO cluster.YarnClusterScheduler: Adding task set 5.0 with 2 tasks 17/05/30 02:20:03 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 10, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:20:03 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 11, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:20:03 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 10 on executor id: 1 hostname: slave2. 17/05/30 02:20:03 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 11 on executor id: 2 hostname: slave1. 17/05/30 02:20:03 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:20:03 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:33 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 11) in 90394 ms on slave1 (1/2) 17/05/30 02:21:33 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 10) in 90477 ms on slave2 (2/2) 17/05/30 02:21:33 INFO cluster.YarnClusterScheduler: Removed TaskSet 5.0, whose tasks have all completed, from pool 17/05/30 02:21:33 INFO scheduler.DAGScheduler: ResultStage 5 (reduce at CaffeOnSpark.scala:226) finished in 90.486 s 17/05/30 02:21:33 INFO scheduler.DAGScheduler: Job 5 finished: reduce at CaffeOnSpark.scala:226, took 90.499345 s 17/05/30 02:21:33 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:21:33 INFO scheduler.DAGScheduler: Got job 6 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:21:33 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:21:33 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:21:33 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:21:33 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[9] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:21:33 INFO memory.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:21:34 INFO memory.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1012 17/05/30 02:21:34 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 6 (MapPartitionsRDD[9] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:21:34 INFO cluster.YarnClusterScheduler: Adding task set 6.0 with 2 tasks 17/05/30 02:21:34 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 12, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:21:34 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 13, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:21:34 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 12 on executor id: 1 hostname: slave2. 17/05/30 02:21:34 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 13 on executor id: 2 hostname: slave1. 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on slave2:44524 in memory (size: 2.6 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on slave2:42368 in memory (size: 2.6 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on slave1:37625 in memory (size: 2.6 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on slave2:44524 in memory (size: 156.0 B, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on slave2:42368 in memory (size: 156.0 B, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on slave1:37625 in memory (size: 156.0 B, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on slave2:44524 in memory (size: 1560.0 B, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on slave2:42368 in memory (size: 1560.0 B, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on slave1:37625 in memory (size: 1560.0 B, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManager: Removing RDD 6 17/05/30 02:21:34 INFO spark.ContextCleaner: Cleaned RDD 6 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on slave2:44524 in memory (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on slave1:37625 in memory (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on slave2:42368 in memory (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:21:34 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:23:06 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 12) in 92838 ms on slave2 (1/2) 17/05/30 02:23:06 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 13) in 92935 ms on slave1 (2/2) 17/05/30 02:23:06 INFO cluster.YarnClusterScheduler: Removed TaskSet 6.0, whose tasks have all completed, from pool 17/05/30 02:23:06 INFO scheduler.DAGScheduler: ResultStage 6 (reduce at CaffeOnSpark.scala:226) finished in 92.953 s 17/05/30 02:23:06 INFO scheduler.DAGScheduler: Job 6 finished: reduce at CaffeOnSpark.scala:226, took 93.044005 s 17/05/30 02:23:06 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:23:06 INFO scheduler.DAGScheduler: Got job 7 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:23:06 INFO scheduler.DAGScheduler: Final stage: ResultStage 7 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:23:06 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:23:06 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:23:06 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[10] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:23:06 INFO memory.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:23:06 INFO memory.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:23:07 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:23:07 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1012 17/05/30 02:23:07 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 7 (MapPartitionsRDD[10] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:23:07 INFO cluster.YarnClusterScheduler: Adding task set 7.0 with 2 tasks 17/05/30 02:23:07 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 14, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:23:07 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 15, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:23:07 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 14 on executor id: 2 hostname: slave1. 17/05/30 02:23:07 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 15 on executor id: 1 hostname: slave2. 17/05/30 02:23:07 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:23:07 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:24:38 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 15) in 91165 ms on slave2 (1/2) 17/05/30 02:24:38 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 14) in 91170 ms on slave1 (2/2) 17/05/30 02:24:38 INFO cluster.YarnClusterScheduler: Removed TaskSet 7.0, whose tasks have all completed, from pool 17/05/30 02:24:38 INFO scheduler.DAGScheduler: ResultStage 7 (reduce at CaffeOnSpark.scala:226) finished in 91.170 s 17/05/30 02:24:38 INFO scheduler.DAGScheduler: Job 7 finished: reduce at CaffeOnSpark.scala:226, took 91.182646 s 17/05/30 02:24:38 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:24:38 INFO scheduler.DAGScheduler: Got job 8 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:24:38 INFO scheduler.DAGScheduler: Final stage: ResultStage 8 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:24:38 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:24:38 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:24:38 INFO scheduler.DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[11] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:24:38 INFO memory.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:24:38 INFO memory.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:24:38 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:24:38 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1012 17/05/30 02:24:38 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 8 (MapPartitionsRDD[11] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:24:38 INFO cluster.YarnClusterScheduler: Adding task set 8.0 with 2 tasks 17/05/30 02:24:38 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 8.0 (TID 16, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:24:38 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 17, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:24:38 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 16 on executor id: 2 hostname: slave1. 17/05/30 02:24:38 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 17 on executor id: 1 hostname: slave2. 17/05/30 02:24:38 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:24:38 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:26:09 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 8.0 (TID 16) in 91430 ms on slave1 (1/2) 17/05/30 02:26:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 17) in 91467 ms on slave2 (2/2) 17/05/30 02:26:09 INFO cluster.YarnClusterScheduler: Removed TaskSet 8.0, whose tasks have all completed, from pool 17/05/30 02:26:09 INFO scheduler.DAGScheduler: ResultStage 8 (reduce at CaffeOnSpark.scala:226) finished in 91.470 s 17/05/30 02:26:09 INFO scheduler.DAGScheduler: Job 8 finished: reduce at CaffeOnSpark.scala:226, took 91.475651 s 17/05/30 02:26:09 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:26:09 INFO scheduler.DAGScheduler: Got job 9 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:26:09 INFO scheduler.DAGScheduler: Final stage: ResultStage 9 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:26:09 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:26:09 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:26:09 INFO scheduler.DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[12] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:26:09 INFO memory.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:26:09 INFO memory.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:26:09 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:26:09 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1012 17/05/30 02:26:09 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 9 (MapPartitionsRDD[12] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:26:09 INFO cluster.YarnClusterScheduler: Adding task set 9.0 with 2 tasks 17/05/30 02:26:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 18, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:26:09 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 9.0 (TID 19, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:26:09 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 18 on executor id: 1 hostname: slave2. 17/05/30 02:26:09 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 19 on executor id: 2 hostname: slave1. 17/05/30 02:26:09 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:26:09 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:27:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 18) in 98539 ms on slave2 (1/2) 17/05/30 02:27:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 9.0 (TID 19) in 98584 ms on slave1 (2/2) 17/05/30 02:27:48 INFO cluster.YarnClusterScheduler: Removed TaskSet 9.0, whose tasks have all completed, from pool 17/05/30 02:27:48 INFO scheduler.DAGScheduler: ResultStage 9 (reduce at CaffeOnSpark.scala:226) finished in 98.590 s 17/05/30 02:27:48 INFO scheduler.DAGScheduler: Job 9 finished: reduce at CaffeOnSpark.scala:226, took 98.599888 s 17/05/30 02:27:48 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:27:48 INFO scheduler.DAGScheduler: Got job 10 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:27:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 10 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:27:48 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:27:48 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:27:48 INFO scheduler.DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[13] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:27:48 INFO memory.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:27:48 INFO memory.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:27:48 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:27:48 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1012 17/05/30 02:27:48 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 10 (MapPartitionsRDD[13] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:27:48 INFO cluster.YarnClusterScheduler: Adding task set 10.0 with 2 tasks 17/05/30 02:27:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 10.0 (TID 20, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:27:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 10.0 (TID 21, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:27:48 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 20 on executor id: 1 hostname: slave2. 17/05/30 02:27:48 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 21 on executor id: 2 hostname: slave1. 17/05/30 02:27:48 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:27:48 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:29:22 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 10.0 (TID 20) in 93878 ms on slave2 (1/2) 17/05/30 02:29:22 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 10.0 (TID 21) in 93894 ms on slave1 (2/2) 17/05/30 02:29:22 INFO cluster.YarnClusterScheduler: Removed TaskSet 10.0, whose tasks have all completed, from pool 17/05/30 02:29:22 INFO scheduler.DAGScheduler: ResultStage 10 (reduce at CaffeOnSpark.scala:226) finished in 93.896 s 17/05/30 02:29:22 INFO scheduler.DAGScheduler: Job 10 finished: reduce at CaffeOnSpark.scala:226, took 93.911454 s 17/05/30 02:29:22 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:29:22 INFO scheduler.DAGScheduler: Got job 11 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:29:22 INFO scheduler.DAGScheduler: Final stage: ResultStage 11 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:29:22 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:29:22 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:29:22 INFO scheduler.DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[14] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:29:22 INFO memory.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:29:22 INFO memory.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:29:22 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:29:22 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1012 17/05/30 02:29:22 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 11 (MapPartitionsRDD[14] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:29:22 INFO cluster.YarnClusterScheduler: Adding task set 11.0 with 2 tasks 17/05/30 02:29:22 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 22, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:29:22 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 23, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:29:22 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 22 on executor id: 2 hostname: slave1. 17/05/30 02:29:22 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 23 on executor id: 1 hostname: slave2. 17/05/30 02:29:22 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:29:22 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:30:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 22) in 88050 ms on slave1 (1/2) 17/05/30 02:30:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 23) in 88061 ms on slave2 (2/2) 17/05/30 02:30:50 INFO cluster.YarnClusterScheduler: Removed TaskSet 11.0, whose tasks have all completed, from pool 17/05/30 02:30:50 INFO scheduler.DAGScheduler: ResultStage 11 (reduce at CaffeOnSpark.scala:226) finished in 88.072 s 17/05/30 02:30:50 INFO scheduler.DAGScheduler: Job 11 finished: reduce at CaffeOnSpark.scala:226, took 88.091617 s 17/05/30 02:30:50 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:30:50 INFO scheduler.DAGScheduler: Got job 12 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:30:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 12 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:30:50 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:30:50 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:30:50 INFO scheduler.DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[15] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:30:50 INFO memory.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:30:50 INFO memory.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:30:50 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:30:50 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1012 17/05/30 02:30:50 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 12 (MapPartitionsRDD[15] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:30:50 INFO cluster.YarnClusterScheduler: Adding task set 12.0 with 2 tasks 17/05/30 02:30:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 24, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:30:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 25, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:30:50 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 24 on executor id: 2 hostname: slave1. 17/05/30 02:30:50 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 25 on executor id: 1 hostname: slave2. 17/05/30 02:30:50 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:30:50 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:32:18 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 25) in 88579 ms on slave2 (1/2) 17/05/30 02:32:18 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 24) in 88601 ms on slave1 (2/2) 17/05/30 02:32:18 INFO cluster.YarnClusterScheduler: Removed TaskSet 12.0, whose tasks have all completed, from pool 17/05/30 02:32:18 INFO scheduler.DAGScheduler: ResultStage 12 (reduce at CaffeOnSpark.scala:226) finished in 88.609 s 17/05/30 02:32:18 INFO scheduler.DAGScheduler: Job 12 finished: reduce at CaffeOnSpark.scala:226, took 88.639875 s 17/05/30 02:32:18 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:32:18 INFO scheduler.DAGScheduler: Got job 13 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:32:18 INFO scheduler.DAGScheduler: Final stage: ResultStage 13 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:32:18 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:32:18 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:32:18 INFO scheduler.DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[16] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:32:18 INFO memory.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 3.7 KB, free 366.3 MB) 17/05/30 02:32:18 INFO memory.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.3 MB) 17/05/30 02:32:18 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:32:18 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1012 17/05/30 02:32:18 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 13 (MapPartitionsRDD[16] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:32:18 INFO cluster.YarnClusterScheduler: Adding task set 13.0 with 2 tasks 17/05/30 02:32:18 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 13.0 (TID 26, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:32:18 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 27, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:32:18 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 26 on executor id: 2 hostname: slave1. 17/05/30 02:32:18 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 27 on executor id: 1 hostname: slave2. 17/05/30 02:32:18 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:32:19 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:33:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 27) in 91206 ms on slave2 (1/2) 17/05/30 02:33:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 13.0 (TID 26) in 91234 ms on slave1 (2/2) 17/05/30 02:33:50 INFO cluster.YarnClusterScheduler: Removed TaskSet 13.0, whose tasks have all completed, from pool 17/05/30 02:33:50 INFO scheduler.DAGScheduler: ResultStage 13 (reduce at CaffeOnSpark.scala:226) finished in 91.239 s 17/05/30 02:33:50 INFO scheduler.DAGScheduler: Job 13 finished: reduce at CaffeOnSpark.scala:226, took 91.248705 s 17/05/30 02:33:50 INFO spark.SparkContext: Starting job: reduce at CaffeOnSpark.scala:226 17/05/30 02:33:50 INFO scheduler.DAGScheduler: Got job 14 (reduce at CaffeOnSpark.scala:226) with 2 output partitions 17/05/30 02:33:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 14 (reduce at CaffeOnSpark.scala:226) 17/05/30 02:33:50 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:33:50 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:33:50 INFO scheduler.DAGScheduler: Submitting ResultStage 14 (MapPartitionsRDD[17] at mapPartitions at CaffeOnSpark.scala:206), which has no missing parents 17/05/30 02:33:50 INFO memory.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 3.7 KB, free 366.2 MB) 17/05/30 02:33:50 INFO memory.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.4 KB, free 366.2 MB) 17/05/30 02:33:50 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on slave2:44524 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:33:50 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1012 17/05/30 02:33:50 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 14 (MapPartitionsRDD[17] at mapPartitions at CaffeOnSpark.scala:206) 17/05/30 02:33:50 INFO cluster.YarnClusterScheduler: Adding task set 14.0 with 2 tasks 17/05/30 02:33:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 14.0 (TID 28, slave2, partition 0, PROCESS_LOCAL, 5222 bytes) 17/05/30 02:33:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 14.0 (TID 29, slave1, partition 1, PROCESS_LOCAL, 5239 bytes) 17/05/30 02:33:50 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 28 on executor id: 1 hostname: slave2. 17/05/30 02:33:50 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 29 on executor id: 2 hostname: slave1. 17/05/30 02:33:50 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on slave1:37625 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:33:50 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on slave2:42368 (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 14.0 (TID 29) in 61680 ms on slave1 (1/2) 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 14.0 (TID 28) in 61954 ms on slave2 (2/2) 17/05/30 02:34:52 INFO cluster.YarnClusterScheduler: Removed TaskSet 14.0, whose tasks have all completed, from pool 17/05/30 02:34:52 INFO scheduler.DAGScheduler: ResultStage 14 (reduce at CaffeOnSpark.scala:226) finished in 61.954 s 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Job 14 finished: reduce at CaffeOnSpark.scala:226, took 61.969233 s 17/05/30 02:34:52 INFO spark.SparkContext: Starting job: collect at CaffeOnSpark.scala:386 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Got job 15 (collect at CaffeOnSpark.scala:386) with 2 output partitions 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 15 (collect at CaffeOnSpark.scala:386) 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[19] at map at CaffeOnSpark.scala:381), which has no missing parents 17/05/30 02:34:52 INFO memory.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 1952.0 B, free 366.2 MB) 17/05/30 02:34:52 INFO memory.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 1227.0 B, free 366.2 MB) 17/05/30 02:34:52 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on slave2:44524 (size: 1227.0 B, free: 366.3 MB) 17/05/30 02:34:52 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1012 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 15 (MapPartitionsRDD[19] at map at CaffeOnSpark.scala:381) 17/05/30 02:34:52 INFO cluster.YarnClusterScheduler: Adding task set 15.0 with 2 tasks 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 30, slave2, partition 0, PROCESS_LOCAL, 5308 bytes) 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 31, slave1, partition 1, PROCESS_LOCAL, 5308 bytes) 17/05/30 02:34:52 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 30 on executor id: 1 hostname: slave2. 17/05/30 02:34:52 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 31 on executor id: 2 hostname: slave1. 17/05/30 02:34:52 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on slave1:37625 (size: 1227.0 B, free: 366.3 MB) 17/05/30 02:34:52 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on slave2:42368 (size: 1227.0 B, free: 366.3 MB) 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 31) in 26 ms on slave1 (1/2) 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 30) in 34 ms on slave2 (2/2) 17/05/30 02:34:52 INFO cluster.YarnClusterScheduler: Removed TaskSet 15.0, whose tasks have all completed, from pool 17/05/30 02:34:52 INFO scheduler.DAGScheduler: ResultStage 15 (collect at CaffeOnSpark.scala:386) finished in 0.040 s 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Job 15 finished: collect at CaffeOnSpark.scala:386, took 0.045014 s 17/05/30 02:34:52 INFO caffe.DataSource$: Source data layer:1 17/05/30 02:34:52 INFO caffe.LMDB: Batch size:32 17/05/30 02:34:52 INFO spark.SparkContext: Starting job: count at CaffeOnSpark.scala:456 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Got job 16 (count at CaffeOnSpark.scala:456) with 2 output partitions 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 16 (count at CaffeOnSpark.scala:456) 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Missing parents: List() 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[23] at map at CaffeOnSpark.scala:451), which has no missing parents 17/05/30 02:34:52 INFO memory.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 3.9 KB, free 366.2 MB) 17/05/30 02:34:52 INFO memory.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.5 KB, free 366.2 MB) 17/05/30 02:34:52 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on slave2:44524 (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:34:52 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1012 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 16 (MapPartitionsRDD[23] at map at CaffeOnSpark.scala:451) 17/05/30 02:34:52 INFO cluster.YarnClusterScheduler: Adding task set 16.0 with 2 tasks 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 32, slave2, partition 0, PROCESS_LOCAL, 5223 bytes) 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 33, slave1, partition 1, PROCESS_LOCAL, 5223 bytes) 17/05/30 02:34:52 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 32 on executor id: 1 hostname: slave2. 17/05/30 02:34:52 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 33 on executor id: 2 hostname: slave1. 17/05/30 02:34:52 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on slave1:37625 (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:34:52 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on slave2:42368 (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 32) in 80 ms on slave2 (1/2) 17/05/30 02:34:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 33) in 83 ms on slave1 (2/2) 17/05/30 02:34:52 INFO cluster.YarnClusterScheduler: Removed TaskSet 16.0, whose tasks have all completed, from pool 17/05/30 02:34:52 INFO scheduler.DAGScheduler: ResultStage 16 (count at CaffeOnSpark.scala:456) finished in 0.086 s 17/05/30 02:34:52 INFO scheduler.DAGScheduler: Job 16 finished: count at CaffeOnSpark.scala:456, took 0.093833 s 17/05/30 02:34:52 INFO caffe.CaffeOnSpark: Schema:StructType(StructField(SampleID,StringType,false), StructField(accuracy,ArrayType(FloatType,true),false), StructField(loss,ArrayType(FloatType,true),false), StructField(label,ArrayType(FloatType,true),false)) 17/05/30 02:34:52 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5f6dfc3e{/SQL,null,AVAILABLE} 17/05/30 02:34:52 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7d200462{/SQL/json,null,AVAILABLE} 17/05/30 02:34:52 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@359df95b{/SQL/execution,null,AVAILABLE} 17/05/30 02:34:52 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1a5e1f09{/SQL/execution/json,null,AVAILABLE} 17/05/30 02:34:52 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5666ca89{/static/sql,null,AVAILABLE} 17/05/30 02:34:52 INFO internal.SharedState: Warehouse path is 'file:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/container_1496110651001_0002_01_000001/spark-warehouse'. 17/05/30 02:34:53 INFO codegen.CodeGenerator: Code generated in 156.553915 ms 17/05/30 02:34:53 INFO codegen.CodeGenerator: Code generated in 9.889899 ms 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO util.Utils: Fetching hdfs://master/projects/machine_learning/image_dataset/mnist_test_lmdb/data.mdb to /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/spark-af9b6dcf-5b64-42f5-b512-7c3e04466776/userFiles-75588463-5a16-4ad0-a5ec-b34b34c0c4a3/mnist_test_lmdb/fetchFileTemp4445742878623530741.tmp 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO util.Utils: Fetching hdfs://master/projects/machine_learning/image_dataset/mnist_test_lmdb/lock.mdb to /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/spark-af9b6dcf-5b64-42f5-b512-7c3e04466776/userFiles-75588463-5a16-4ad0-a5ec-b34b34c0c4a3/mnist_test_lmdb/fetchFileTemp5748581051992443231.tmp 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO spark.SparkContext: Added file hdfs:///projects/machine_learning/image_dataset/mnist_test_lmdb/ at hdfs:///projects/machine_learning/image_dataset/mnist_test_lmdb/ with timestamp 1496111693162 17/05/30 02:34:53 INFO caffe.LmdbRDD: local LMDB path:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1496110651001_0002/spark-af9b6dcf-5b64-42f5-b512-7c3e04466776/userFiles-75588463-5a16-4ad0-a5ec-b34b34c0c4a3/mnist_test_lmdb 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_15_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_15_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO caffe.LmdbRDD: 2 LMDB RDD partitions 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_15_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO spark.SparkContext: Starting job: count at CaffeOnSpark.scala:432 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_16_piece0 on slave2:44524 in memory (size: 1227.0 B, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_16_piece0 on slave1:37625 in memory (size: 1227.0 B, free: 366.3 MB) 17/05/30 02:34:53 INFO scheduler.DAGScheduler: Registering RDD 30 (count at CaffeOnSpark.scala:432) 17/05/30 02:34:53 INFO scheduler.DAGScheduler: Got job 17 (count at CaffeOnSpark.scala:432) with 1 output partitions 17/05/30 02:34:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 18 (count at CaffeOnSpark.scala:432) 17/05/30 02:34:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 17) 17/05/30 02:34:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 17) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_16_piece0 on slave2:42368 in memory (size: 1227.0 B, free: 366.3 MB) 17/05/30 02:34:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 17 (MapPartitionsRDD[30] at count at CaffeOnSpark.scala:432), which has no missing parents 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_17_piece0 on slave2:44524 in memory (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_17_piece0 on slave1:37625 in memory (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_17_piece0 on slave2:42368 in memory (size: 2.5 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO spark.ContextCleaner: Cleaned accumulator 1124 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO memory.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 18.6 KB, free 366.3 MB) 17/05/30 02:34:53 INFO memory.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 9.1 KB, free 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on slave2:44524 (size: 9.1 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1012 17/05/30 02:34:53 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 17 (MapPartitionsRDD[30] at count at CaffeOnSpark.scala:432) 17/05/30 02:34:53 INFO cluster.YarnClusterScheduler: Adding task set 17.0 with 2 tasks 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 34, slave2, partition 0, PROCESS_LOCAL, 5315 bytes) 17/05/30 02:34:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 17.0 (TID 35, slave1, partition 1, PROCESS_LOCAL, 5332 bytes) 17/05/30 02:34:53 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 34 on executor id: 1 hostname: slave2. 17/05/30 02:34:53 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 35 on executor id: 2 hostname: slave1. 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on slave2:44524 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on slave2:42368 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on slave1:37625 in memory (size: 2.4 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on slave1:37625 (size: 9.1 KB, free: 366.3 MB) 17/05/30 02:34:53 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on slave2:42368 (size: 9.1 KB, free: 366.3 MB)