Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error in executing the BwaAndMarkDuplicatesPipelineSpark #4112

Closed
Sun-shan opened this issue Jan 10, 2018 · 3 comments
Closed

Error in executing the BwaAndMarkDuplicatesPipelineSpark #4112

Sun-shan opened this issue Jan 10, 2018 · 3 comments
Labels
bug Spark

Comments

@Sun-shan
Copy link

Sun-shan commented Jan 10, 2018 •
edited by lbergelson
Loading

Hi, when I executed the BwaAndMarkDuplicatesPipelineSpark command, there are some issues, I am not sure the issue related to the Spark environment or to the files I used.

The "ucsc.hg19.fasta.img" is generated by the "gatk-launch BwaMemIndexImageCreator ucsc.hg19.fasta" command

The "1982.unmapped.bam" is generated by the picard command FastqToSam.jar 🔢

"java -jar /opt/NfsDir/BioDir/picard-tools-1.119/FastqToSam.jar F1=/opt/NfsDir/UserDir/lvxy/pipeline_test/BRCA/1982.R1.clean.fastq.gz F2=/opt/NfsDir/UserDir/lvxy/pipeline_test/BRCA/1982.R2.clean.fastq.gz V=Standard O=/opt/NfsDir/UserDir/wujh/1982.unmapped.bam SM=R1"

Are there some examples I can follow?

[sun@tele-1 download]/opt/NfsDir/BioDir/GATK4/gatk/gatk-launch BwaAndMarkDuplicatesPipelineSpark --bwamemIndexImage hdfs:///user/sun/ucsc.hg19.fasta.img -I hdfs:///user/sun/1982.unmapped.bam -R hdfs:///user/sun/ucsc.hg19.fasta -O hdfs:///user/sun/17F02897_17F02897M_WES_img.bwa.bam -- --sparkRunner SPARK --sparkMaster yarn --sparkSubmitCommand spark2-submit --driver-memory 4G --num-executors 4 --executor-cores 6 --executor-memory 16G --conf spark.dynamicAllocation.enabled=false
Using GATK jar /opt/NfsDir/BioDir/GATK4/gatk/build/libs/gatk-package-4.beta.5-50-g8d666b6-SNAPSHOT-spark.jar
Running:
    spark2-submit --master yarn --conf spark.driver.userClassPathFirst=true --conf spark.io.compression.codec=lzf --conf spark.driver.maxResultSize=0 --conf spark.executor.extraJavaOptions=-DGATK_STACKTRACE_ON_USER_EXCEPTION=true -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=false -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true  --conf spark.driver.extraJavaOptions=-DGATK_STACKTRACE_ON_USER_EXCEPTION=true -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=false -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true  --conf spark.kryoserializer.buffer.max=512m --conf spark.yarn.executor.memoryOverhead=600 --driver-memory 4G --num-executors 4 --executor-cores 6 --executor-memory 16G --conf spark.dynamicAllocation.enabled=false /opt/NfsDir/BioDir/GATK4/gatk/build/libs/gatk-package-4.beta.5-50-g8d666b6-SNAPSHOT-spark.jar BwaAndMarkDuplicatesPipelineSpark --bwamemIndexImage hdfs:///user/sun/ucsc.hg19.fasta.img -I hdfs:///user/sun/1982.unmapped.bam -R hdfs:///user/sun/ucsc.hg19.fasta -O hdfs:///user/sun/17F02897_17F02897M_WES_img.bwa.bam --sparkMaster yarn
WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
18:30:33.354 WARN  SparkContextFactory - Environment variables HELLBENDER_TEST_PROJECT and HELLBENDER_JSON_SERVICE_ACCOUNT_KEY must be set or the GCS hadoop connector will not be configured properly
18:30:33.534 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/opt/NfsDir/BioDir/GATK4/gatk/build/libs/gatk-package-4.beta.5-50-g8d666b6-SNAPSHOT-spark.jar!/com/intel/gkl/native/libgkl_compression.so
[January 9, 2018 6:30:33 PM CST] BwaAndMarkDuplicatesPipelineSpark  --bwamemIndexImage hdfs:///user/sun/ucsc.hg19.fasta.img --output hdfs:///user/sun/17F02897_17F02897M_WES_img.bwa.bam --reference hdfs:///user/sun/ucsc.hg19.fasta --input hdfs:///user/sun/1982.unmapped.bam --sparkMaster yarn  --duplicates_scoring_strategy SUM_OF_BASE_QUALITIES --readValidationStringency SILENT --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --interval_merging_rule ALL --bamPartitionSize 0 --disableSequenceDictionaryValidation false --shardedOutput false --numReducers 0 --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --gcs_max_retries 20 --disableToolDefaultReadFilters false
[January 9, 2018 6:30:33 PM CST] Executing as sun@tele-1 on Linux 3.10.0-514.10.2.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14; Version: 4.beta.5-50-g8d666b6-SNAPSHOT
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 1
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : false
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - Deflater: IntelDeflater
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - Inflater: IntelInflater
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - GCS max retries/reopens: 20
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - Using google-cloud-java patch c035098b5e62cb4fe9155eff07ce88449a361f5d from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - Initializing engine
18:30:54.424 INFO  BwaAndMarkDuplicatesPipelineSpark - Done initializing engine
18/01/09 18:30:54 INFO spark.SparkContext: Running Spark version 2.2.0.cloudera1
18/01/09 18:30:54 INFO spark.SparkContext: Submitted application: BwaAndMarkDuplicatesPipelineSpark
18/01/09 18:30:54 INFO spark.SecurityManager: Changing view acls to: sun
18/01/09 18:30:54 INFO spark.SecurityManager: Changing modify acls to: sun
18/01/09 18:30:54 INFO spark.SecurityManager: Changing view acls groups to: 
18/01/09 18:30:54 INFO spark.SecurityManager: Changing modify acls groups to: 
18/01/09 18:30:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sun); groups with view permissions: Set(); users  with modify permissions: Set(sun); groups with modify permissions: Set()
18/01/09 18:30:55 INFO util.Utils: Successfully started service 'sparkDriver' on port 38793.
18/01/09 18:30:55 INFO spark.SparkEnv: Registering MapOutputTracker
18/01/09 18:30:55 INFO spark.SparkEnv: Registering BlockManagerMaster
18/01/09 18:30:55 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/01/09 18:30:55 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/01/09 18:30:55 INFO storage.DiskBlockManager: Created local directory at /tmp/sun/blockmgr-b03058dc-763a-449c-bd05-18f3304c01ea
18/01/09 18:30:55 INFO memory.MemoryStore: MemoryStore started with capacity 2004.6 MB
18/01/09 18:30:55 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/01/09 18:30:55 INFO util.log: Logging initialized @25356ms
18/01/09 18:30:55 INFO server.Server: jetty-9.3.z-SNAPSHOT
18/01/09 18:30:55 INFO server.Server: Started @25495ms
18/01/09 18:30:55 INFO server.AbstractConnector: Started ServerConnector@283ab206{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/01/09 18:30:55 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@418f0534{/jobs,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@134a8ead{/jobs/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54247647{/jobs/job,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5463f035{/jobs/job/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@44fd7ba4{/stages,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@69d103f0{/stages/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@74fb5b59{/stages/stage,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@26fadd98{/stages/stage/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3db6dd52{/stages/pool,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6ef4cbe1{/stages/pool/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2baac4a7{/storage,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6bce4140{/storage/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5882b202{/storage/rdd,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@b506ed0{/storage/rdd/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@65f3e805{/environment,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@10618775{/environment/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@20a3e10c{/executors,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5e2a6991{/executors/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6f96dd64{/executors/threadDump,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@409732fb{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5e99e2cb{/static,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@478967eb{/,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7f2b39a{/api,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@18c880ea{/jobs/job/kill,null,AVAILABLE,@Spark}
18/01/09 18:30:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6afbe6a1{/stages/stage/kill,null,AVAILABLE,@Spark}
18/01/09 18:30:56 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.4:4040
18/01/09 18:30:56 INFO spark.SparkContext: Added JAR file:/opt/NfsDir/BioDir/GATK4/gatk/build/libs/gatk-package-4.beta.5-50-g8d666b6-SNAPSHOT-spark.jar at spark://192.168.1.4:38793/jars/gatk-package-4.beta.5-50-g8d666b6-SNAPSHOT-spark.jar with timestamp 1515493856032
18/01/09 18:30:56 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.1-hadoop2
18/01/09 18:30:57 INFO client.RMProxy: Connecting to ResourceManager at tele-1/192.168.1.4:8032
18/01/09 18:30:57 INFO yarn.Client: Requesting a new application from cluster with 4 NodeManagers
18/01/09 18:30:58 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (18432 MB per container)
18/01/09 18:30:58 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
18/01/09 18:30:58 INFO yarn.Client: Setting up container launch context for our AM
18/01/09 18:30:58 INFO yarn.Client: Setting up the launch environment for our AM container
18/01/09 18:30:58 INFO yarn.Client: Preparing resources for our AM container
18/01/09 18:30:59 INFO yarn.Client: Uploading resource file:/tmp/sun/spark-5a3e539e-2e2b-4da2-b218-2bda166bd4c0/__spark_conf__7100950787185363106.zip -> hdfs://tele-1:8020/user/sun/.sparkStaging/application_1515493209401_0001/__spark_conf__.zip
18/01/09 18:31:00 INFO spark.SecurityManager: Changing view acls to: sun
18/01/09 18:31:00 INFO spark.SecurityManager: Changing modify acls to: sun
18/01/09 18:31:00 INFO spark.SecurityManager: Changing view acls groups to: 
18/01/09 18:31:00 INFO spark.SecurityManager: Changing modify acls groups to: 
18/01/09 18:31:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(sun); groups with view permissions: Set(); users  with modify permissions: Set(sun); groups with modify permissions: Set()
18/01/09 18:31:00 INFO yarn.Client: Submitting application application_1515493209401_0001 to ResourceManager
18/01/09 18:31:00 INFO impl.YarnClientImpl: Submitted application application_1515493209401_0001
18/01/09 18:31:00 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1515493209401_0001 and attemptId None
18/01/09 18:31:01 INFO yarn.Client: Application report for application_1515493209401_0001 (state: ACCEPTED)
18/01/09 18:31:01 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.users.sun
	 start time: 1515493860237
	 final status: UNDEFINED
	 tracking URL: http://tele-1:8088/proxy/application_1515493209401_0001/
	 user: sun
18/01/09 18:31:02 INFO yarn.Client: Application report for application_1515493209401_0001 (state: ACCEPTED)
18/01/09 18:31:03 INFO yarn.Client: Application report for application_1515493209401_0001 (state: ACCEPTED)
18/01/09 18:31:04 INFO yarn.Client: Application report for application_1515493209401_0001 (state: ACCEPTED)
18/01/09 18:31:05 INFO yarn.Client: Application report for application_1515493209401_0001 (state: ACCEPTED)
18/01/09 18:31:05 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
18/01/09 18:31:05 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> tele-1, PROXY_URI_BASES -> http://tele-1:8088/proxy/application_1515493209401_0001), /proxy/application_1515493209401_0001
18/01/09 18:31:05 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
18/01/09 18:31:06 INFO yarn.Client: Application report for application_1515493209401_0001 (state: RUNNING)
18/01/09 18:31:06 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 192.168.1.4
	 ApplicationMaster RPC port: 0
	 queue: root.users.sun
	 start time: 1515493860237
	 final status: UNDEFINED
	 tracking URL: http://tele-1:8088/proxy/application_1515493209401_0001/
	 user: sun
18/01/09 18:31:06 INFO cluster.YarnClientSchedulerBackend: Application application_1515493209401_0001 has started running.
18/01/09 18:31:06 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44190.
18/01/09 18:31:06 INFO netty.NettyBlockTransferService: Server created on 192.168.1.4:44190
18/01/09 18:31:06 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/01/09 18:31:06 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.4, 44190, None)
18/01/09 18:31:06 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.4:44190 with 2004.6 MB RAM, BlockManagerId(driver, 192.168.1.4, 44190, None)
18/01/09 18:31:06 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.4, 44190, None)
18/01/09 18:31:06 INFO storage.BlockManager: external shuffle service port = 7337
18/01/09 18:31:06 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.4, 44190, None)
18/01/09 18:31:06 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60c8909a{/metrics/json,null,AVAILABLE,@Spark}
18/01/09 18:31:06 INFO scheduler.EventLoggingListener: Logging events to hdfs://tele-1:8020/user/spark/spark2ApplicationHistory/application_1515493209401_0001
18/01/09 18:31:09 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000002 on host: tele-6. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:09 INFO storage.BlockManagerMaster: Removal of executor 1 requested
18/01/09 18:31:09 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 1
18/01/09 18:31:09 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
18/01/09 18:31:12 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000003 on host: tele-2. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:12 INFO storage.BlockManagerMaster: Removal of executor 2 requested
18/01/09 18:31:12 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 2
18/01/09 18:31:12 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000004 on host: tele-6. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000004
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:12 INFO storage.BlockManagerMaster: Removal of executor 3 requested
18/01/09 18:31:12 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 3
18/01/09 18:31:12 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster.
18/01/09 18:31:12 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 3 from BlockManagerMaster.
18/01/09 18:31:15 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000005 on host: tele-2. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000005
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:15 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000006 on host: tele-6. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000006
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:15 INFO storage.BlockManagerMaster: Removal of executor 4 requested
18/01/09 18:31:15 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 4
18/01/09 18:31:15 INFO storage.BlockManagerMaster: Removal of executor 5 requested
18/01/09 18:31:15 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 5
18/01/09 18:31:15 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.
18/01/09 18:31:15 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 5 from BlockManagerMaster.
18/01/09 18:31:18 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000007 on host: tele-2. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000007
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:18 INFO storage.BlockManagerMaster: Removal of executor 6 requested
18/01/09 18:31:18 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 6 from BlockManagerMaster.
18/01/09 18:31:18 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 6
18/01/09 18:31:18 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000008 on host: tele-6. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000008
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:18 INFO storage.BlockManagerMaster: Removal of executor 7 requested
18/01/09 18:31:18 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 7 from BlockManagerMaster.
18/01/09 18:31:18 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 7
18/01/09 18:31:21 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000009 on host: tele-2. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000009
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:21 INFO storage.BlockManagerMaster: Removal of executor 8 requested
18/01/09 18:31:21 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 8
18/01/09 18:31:21 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
18/01/09 18:31:21 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1515493209401_0001_01_000010 on host: tele-6. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1515493209401_0001_01_000010
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
	at org.apache.hadoop.util.Shell.run(Shell.java:504)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

18/01/09 18:31:21 INFO storage.BlockManagerMaster: Removal of executor 9 requested
18/01/09 18:31:21 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 9
18/01/09 18:31:21 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 9 from BlockManagerMaster.
18/01/09 18:31:26 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
18/01/09 18:31:26 INFO server.AbstractConnector: Stopped Spark@283ab206{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/01/09 18:31:26 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.4:4040
18/01/09 18:31:26 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
18/01/09 18:31:26 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
18/01/09 18:31:26 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
18/01/09 18:31:26 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
18/01/09 18:31:26 INFO cluster.YarnClientSchedulerBackend: Stopped
18/01/09 18:31:26 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/01/09 18:31:26 INFO memory.MemoryStore: MemoryStore cleared
18/01/09 18:31:26 INFO storage.BlockManager: BlockManager stopped
18/01/09 18:31:26 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
18/01/09 18:31:26 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/01/09 18:31:26 INFO spark.SparkContext: Successfully stopped SparkContext
18:31:26.896 INFO  BwaAndMarkDuplicatesPipelineSpark - Shutting down engine
[January 9, 2018 6:31:26 PM CST] org.broadinstitute.hellbender.tools.spark.pipelines.BwaAndMarkDuplicatesPipelineSpark done. Elapsed time: 0.89 minutes.
Runtime.totalMemory()=881328128
***********************************************************************

A USER ERROR has occurred: Input files reference and reads have incompatible contigs: No overlapping contigs found.
  reference contigs = [chrM, chr1, chr2, chr3, chr4, chr5, chr6, chr7, chr8, chr9, chr10, chr11, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr19, chr20, chr21, chr22, chrX, chrY, chr1_gl000191_random, chr1_gl000192_random, chr4_ctg9_hap1, chr4_gl000193_random, chr4_gl000194_random, chr6_apd_hap1, chr6_cox_hap2, chr6_dbb_hap3, chr6_mann_hap4, chr6_mcf_hap5, chr6_qbl_hap6, chr6_ssto_hap7, chr7_gl000195_random, chr8_gl000196_random, chr8_gl000197_random, chr9_gl000198_random, chr9_gl000199_random, chr9_gl000200_random, chr9_gl000201_random, chr11_gl000202_random, chr17_ctg5_hap1, chr17_gl000203_random, chr17_gl000204_random, chr17_gl000205_random, chr17_gl000206_random, chr18_gl000207_random, chr19_gl000208_random, chr19_gl000209_random, chr21_gl000210_random, chrUn_gl000211, chrUn_gl000212, chrUn_gl000213, chrUn_gl000214, chrUn_gl000215, chrUn_gl000216, chrUn_gl000217, chrUn_gl000218, chrUn_gl000219, chrUn_gl000220, chrUn_gl000221, chrUn_gl000222, chrUn_gl000223, chrUn_gl000224, chrUn_gl000225, chrUn_gl000226, chrUn_gl000227, chrUn_gl000228, chrUn_gl000229, chrUn_gl000230, chrUn_gl000231, chrUn_gl000232, chrUn_gl000233, chrUn_gl000234, chrUn_gl000235, chrUn_gl000236, chrUn_gl000237, chrUn_gl000238, chrUn_gl000239, chrUn_gl000240, chrUn_gl000241, chrUn_gl000242, chrUn_gl000243, chrUn_gl000244, chrUn_gl000245, chrUn_gl000246, chrUn_gl000247, chrUn_gl000248, chrUn_gl000249]
  reads contigs = []

***********************************************************************
org.broadinstitute.hellbender.exceptions.UserException$IncompatibleSequenceDictionaries: Input files reference and reads have incompatible contigs: No overlapping contigs found.
  reference contigs = [chrM, chr1, chr2, chr3, chr4, chr5, chr6, chr7, chr8, chr9, chr10, chr11, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr19, chr20, chr21, chr22, chrX, chrY, chr1_gl000191_random, chr1_gl000192_random, chr4_ctg9_hap1, chr4_gl000193_random, chr4_gl000194_random, chr6_apd_hap1, chr6_cox_hap2, chr6_dbb_hap3, chr6_mann_hap4, chr6_mcf_hap5, chr6_qbl_hap6, chr6_ssto_hap7, chr7_gl000195_random, chr8_gl000196_random, chr8_gl000197_random, chr9_gl000198_random, chr9_gl000199_random, chr9_gl000200_random, chr9_gl000201_random, chr11_gl000202_random, chr17_ctg5_hap1, chr17_gl000203_random, chr17_gl000204_random, chr17_gl000205_random, chr17_gl000206_random, chr18_gl000207_random, chr19_gl000208_random, chr19_gl000209_random, chr21_gl000210_random, chrUn_gl000211, chrUn_gl000212, chrUn_gl000213, chrUn_gl000214, chrUn_gl000215, chrUn_gl000216, chrUn_gl000217, chrUn_gl000218, chrUn_gl000219, chrUn_gl000220, chrUn_gl000221, chrUn_gl000222, chrUn_gl000223, chrUn_gl000224, chrUn_gl000225, chrUn_gl000226, chrUn_gl000227, chrUn_gl000228, chrUn_gl000229, chrUn_gl000230, chrUn_gl000231, chrUn_gl000232, chrUn_gl000233, chrUn_gl000234, chrUn_gl000235, chrUn_gl000236, chrUn_gl000237, chrUn_gl000238, chrUn_gl000239, chrUn_gl000240, chrUn_gl000241, chrUn_gl000242, chrUn_gl000243, chrUn_gl000244, chrUn_gl000245, chrUn_gl000246, chrUn_gl000247, chrUn_gl000248, chrUn_gl000249]
  reads contigs = []
	at org.broadinstitute.hellbender.utils.SequenceDictionaryUtils.validateDictionaries(SequenceDictionaryUtils.java:163)
	at org.broadinstitute.hellbender.utils.SequenceDictionaryUtils.validateDictionaries(SequenceDictionaryUtils.java:98)
	at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.validateToolInputs(GATKSparkTool.java:469)
	at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.runPipeline(GATKSparkTool.java:361)
	at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:38)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:119)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:176)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:195)
	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:131)
	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:152)
	at org.broadinstitute.hellbender.Main.main(Main.java:233)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/01/09 18:31:26 INFO util.ShutdownHookManager: Shutdown hook called
18/01/09 18:31:26 INFO util.ShutdownHookManager: Deleting directory /tmp/sun/spark-5a3e539e-2e2b-4da2-b218-2bda166bd4c0
@lbergelson
Copy link
Member

@Sun-shan Hi, could you try running with the --disable-sequence-dictionary-validation command?

@lbergelson
Copy link
Member

Opening a new issue here #4131 to track this bug.

@lbergelson
Copy link
Member

This should have been resolved when #4131 was. Please reopen if it isn't.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Spark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lbergelson @Sun-shan