From 85201e68ccf14fda3ec0163ce83ec6bcff76a0f1 Mon Sep 17 00:00:00 2001
From: Wei Shen <shenwei356@gmail.com>
Date: Fri, 1 Nov 2024 19:15:37 +0800
Subject: [PATCH] v2.9.0

---
 download/index.html      |  64 +++++++++++++++++++++++++--------------
 search/search_index.json |   2 +-
 sitemap.xml.gz           | Bin 127 -> 127 bytes
 usage/index.html         |  29 ++++++++++++------
 4 files changed, 63 insertions(+), 32 deletions(-)

diff --git a/download/index.html b/download/index.html
index 5ab4591..b823c80 100644
--- a/download/index.html
+++ b/download/index.html
@@ -647,18 +647,22 @@ <h1 id="download">Download</h1>
 </ol>
 <h2 id="current-version">Current Version</h2>
 <ul>
-<li><a href="https://github.com/shenwei356/seqkit/releases/tag/v2.8.2">SeqKit v2.8.2</a> - 2024-05-17
-<a href="https://github.com/shenwei356/seqkit/releases/tag/v2.8.2"><img alt="Github Releases (by Release)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/v2.8.2/total.svg" /></a><ul>
-<li><code>seqkit amplicon</code>:<ul>
-<li>Fix a big introduced in v2.7.0. When more than one pairs of primers are given, only the last one is used. <a href="https://github.com/shenwei356/seqkit/issues/457">#457</a></li>
+<li><a href="https://github.com/shenwei356/seqkit/releases/tag/v2.9.0">SeqKit v2.9.0</a> - 2024-11-01
+<a href="https://github.com/shenwei356/seqkit/releases/tag/v2.9.0"><img alt="Github Releases (by Release)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/v2.9.0/total.svg" /></a><ul>
+<li><code>seqkit</code>:<ul>
+<li><strong>Fix sequence ID parsing with the default regular expression (in this case, we actually use bytes.Index instead) for a rare case: "xxx\tyyy zzz" was wrongly parsed as "xxx\tyyy"</strong>. <a href="https://github.com/shenwei356/seqkit/issues/486">#486</a></li>
 </ul>
 </li>
-<li><code>seqkit translate</code>:<ul>
-<li>Add option <code>-e/--skip-translate-errors</code> to skip translate error and output empty sequence. <a href="https://github.com/shenwei356/seqkit/pull/458">#458</a></li>
+<li><code>seqkit locate</code>:<ul>
+<li><strong>Fix <code>-G/--non-greedy</code> for tandem repeats</strong>, e.g., ATTCGATTCGATTCG (ATTCGx3).</li>
 </ul>
 </li>
-<li><code>seqkit split</code>:<ul>
-<li>Add flag <code>-I/--ignore-case</code> for <code>-i/--by-id</code>. <a href="https://github.com/shenwei356/seqkit/issues/462">#462</a></li>
+<li><code>seqkit grep/subseq</code>:<ul>
+<li>Fix negative regions longer than sequence length. <a href="https://github.com/shenwei356/seqkit/issues/479">#479</a>.</li>
+</ul>
+</li>
+<li><code>seqkit stats</code>:<ul>
+<li>Add an extra column <code>sum_n</code> to count the number of ambiguous characters. <a href="https://github.com/shenwei356/seqkit/issues/490">#490</a></li>
 </ul>
 </li>
 </ul>
@@ -678,44 +682,44 @@ <h3 id="links">Links</h3>
 <tr>
 <td style="text-align: left;">Linux</td>
 <td style="text-align: left;">32-bit</td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_linux_386.tar.gz">seqkit_linux_386.tar.gz</a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_linux_386.tar.gz">中国镜像</a></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_linux_386.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_linux_386.tar.gz.svg?maxAge=3600" /></a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_linux_386.tar.gz">seqkit_linux_386.tar.gz</a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_linux_386.tar.gz">中国镜像</a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_linux_386.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_linux_386.tar.gz.svg?maxAge=3600" /></a></td>
 </tr>
 <tr>
 <td style="text-align: left;">Linux</td>
 <td style="text-align: left;"><strong>64-bit</strong></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_linux_amd64.tar.gz"><strong>seqkit_linux_amd64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_linux_amd64.tar.gz">中国镜像</a></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_linux_amd64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_linux_amd64.tar.gz.svg?maxAge=3600" /></a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_linux_amd64.tar.gz"><strong>seqkit_linux_amd64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_linux_amd64.tar.gz">中国镜像</a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_linux_amd64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_linux_amd64.tar.gz.svg?maxAge=3600" /></a></td>
 </tr>
 <tr>
 <td style="text-align: left;">Linux</td>
 <td style="text-align: left;"><strong>arm64</strong></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_linux_arm64.tar.gz"><strong>seqkit_linux_arm64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_linux_arm64.tar.gz">中国镜像</a></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_linux_arm64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_linux_arm64.tar.gz.svg?maxAge=3600" /></a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_linux_arm64.tar.gz"><strong>seqkit_linux_arm64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_linux_arm64.tar.gz">中国镜像</a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_linux_arm64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_linux_arm64.tar.gz.svg?maxAge=3600" /></a></td>
 </tr>
 <tr>
 <td style="text-align: left;">macOS</td>
 <td style="text-align: left;"><strong>64-bit</strong></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_darwin_amd64.tar.gz"><strong>seqkit_darwin_amd64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_darwin_amd64.tar.gz">中国镜像</a></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_darwin_amd64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_darwin_amd64.tar.gz.svg?maxAge=3600" /></a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_darwin_amd64.tar.gz"><strong>seqkit_darwin_amd64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_darwin_amd64.tar.gz">中国镜像</a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_darwin_amd64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_darwin_amd64.tar.gz.svg?maxAge=3600" /></a></td>
 </tr>
 <tr>
 <td style="text-align: left;">macOS</td>
 <td style="text-align: left;"><strong>arm64</strong></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_darwin_arm64.tar.gz"><strong>seqkit_darwin_arm64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_darwin_arm64.tar.gz">中国镜像</a></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_darwin_arm64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_darwin_arm64.tar.gz.svg?maxAge=3600" /></a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_darwin_arm64.tar.gz"><strong>seqkit_darwin_arm64.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_darwin_arm64.tar.gz">中国镜像</a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_darwin_arm64.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_darwin_arm64.tar.gz.svg?maxAge=3600" /></a></td>
 </tr>
 <tr>
 <td style="text-align: left;">Windows</td>
 <td style="text-align: left;">32-bit</td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_windows_386.exe.tar.gz">seqkit_windows_386.exe.tar.gz</a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_windows_386.exe.tar.gz">中国镜像</a></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_windows_386.exe.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_windows_386.exe.tar.gz.svg?maxAge=3600" /></a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_windows_386.exe.tar.gz">seqkit_windows_386.exe.tar.gz</a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_windows_386.exe.tar.gz">中国镜像</a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_windows_386.exe.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_windows_386.exe.tar.gz.svg?maxAge=3600" /></a></td>
 </tr>
 <tr>
 <td style="text-align: left;">Windows</td>
 <td style="text-align: left;"><strong>64-bit</strong></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_windows_amd64.exe.tar.gz"><strong>seqkit_windows_amd64.exe.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_windows_amd64.exe.tar.gz">中国镜像</a></td>
-<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.8.2/seqkit_windows_amd64.exe.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_windows_amd64.exe.tar.gz.svg?maxAge=3600" /></a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_windows_amd64.exe.tar.gz"><strong>seqkit_windows_amd64.exe.tar.gz</strong></a>, <br/> <a href="http://app.shenwei.me/data/seqkit/seqkit_windows_amd64.exe.tar.gz">中国镜像</a></td>
+<td style="text-align: left;"><a href="https://github.com/shenwei356/seqkit/releases/download/v2.9.0/seqkit_windows_amd64.exe.tar.gz"><img alt="Github Releases (by Asset)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/latest/seqkit_windows_amd64.exe.tar.gz.svg?maxAge=3600" /></a></td>
 </tr>
 </tbody>
 </table>
@@ -830,6 +834,22 @@ <h2 id="shell-completion">Shell-completion</h2>
 </code></pre>
 <h2 id="release-history">Release history</h2>
 <ul>
+<li><a href="https://github.com/shenwei356/seqkit/releases/tag/v2.8.2">SeqKit v2.8.2</a> - 2024-05-17
+<a href="https://github.com/shenwei356/seqkit/releases/tag/v2.8.2"><img alt="Github Releases (by Release)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/v2.8.2/total.svg" /></a><ul>
+<li><code>seqkit amplicon</code>:<ul>
+<li>Fix a big introduced in v2.7.0. When more than one pairs of primers are given, only the last one is used. <a href="https://github.com/shenwei356/seqkit/issues/457">#457</a></li>
+</ul>
+</li>
+<li><code>seqkit translate</code>:<ul>
+<li>Add option <code>-e/--skip-translate-errors</code> to skip translate error and output empty sequence. <a href="https://github.com/shenwei356/seqkit/pull/458">#458</a></li>
+</ul>
+</li>
+<li><code>seqkit split</code>:<ul>
+<li>Add flag <code>-I/--ignore-case</code> for <code>-i/--by-id</code>. <a href="https://github.com/shenwei356/seqkit/issues/462">#462</a></li>
+</ul>
+</li>
+</ul>
+</li>
 <li><a href="https://github.com/shenwei356/seqkit/releases/tag/v2.8.1">SeqKit v2.8.1</a> - 2024-04-07
 <a href="https://github.com/shenwei356/seqkit/releases/tag/v2.8.1"><img alt="Github Releases (by Release)" src="https://img.shields.io/github/downloads/shenwei356/seqkit/v2.8.1/total.svg" /></a><ul>
 <li><code>seqkit sana</code>:<ul>
diff --git a/search/search_index.json b/search/search_index.json
index a3802c8..c884782 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"SeqKit - a cross-platform and ultrafast toolkit for FASTA/Q file manipulation","text":"<ul> <li>Try SeqKit in your browser (Tutorials and Exercises provided by sandbox.bio)</li> <li>Documents: http://bioinf.shenwei.me/seqkit (Usage, FAQs, Tutorial, and  Benchmark)</li> <li>Source code: https://github.com/shenwei356/seqkit </li> <li>Latest version: </li> <li>Please cite: ,  </li> <li>Others:  </li> </ul>"},{"location":"#features","title":"Features","text":"<ul> <li>Easy to install (download)<ul> <li>Providing statically linked executable binaries for multiple platforms (Linux/Windows/macOS, amd64/arm64) </li> <li>Light weight and out-of-the-box, no dependencies, no compilation, no configuration</li> <li><code>conda install -c bioconda seqkit</code></li> </ul> </li> <li>Easy to use <ul> <li>Ultrafast (see technical-details and benchmark)</li> <li>Seamlessly parsing both FASTA and FASTQ formats</li> <li>Supporting (<code>gzip</code>/<code>xz</code>/<code>zstd</code>/<code>bzip2</code> compressed) STDIN/STDOUT and input/output file, easily integrated in pipe</li> <li>Reproducible results (configurable rand seed in <code>sample</code> and <code>shuffle</code>)</li> <li>Supporting custom sequence ID via regular expression</li> <li>Supporting Bash/Zsh autocompletion</li> </ul> </li> <li>Versatile commands (usages and examples)<ul> <li>Practical functions supported by 38 subcommands</li> </ul> </li> </ul>"},{"location":"#installation","title":"Installation","text":"<p>Go to Download Page for more download options and changelogs, or install via conda:</p> <pre><code>conda install -c bioconda seqkit\n</code></pre>"},{"location":"#subcommands","title":"Subcommands","text":"Category Command Function Input Strand-sensitivity Multi-threads Basic operation seq Transform sequences: extract ID/seq, filter by length/quality, remove gaps\u2026 FASTA/Q stats Simple statistics: #seqs, min/max_len, N50, Q20%, Q30%\u2026 FASTA/Q \u2713 subseq Get subsequences by region/gtf/bed, including flanking sequences FASTA/Q + or/and - sliding Extract subsequences in sliding windows FASTA/Q + only faidx Create the FASTA index file and extract subsequences (with more features than samtools faidx) FASTA + or/and - translate translate DNA/RNA to protein sequence FASTA/Q + or/and - watch  Monitoring and online histograms of sequence features FASTA/Q scat  Real time concatenation and streaming of fastx files FASTA/Q \u2713 Format conversion fq2fa Convert FASTQ to FASTA format FASTQ fx2tab Convert FASTA/Q to tabular format FASTA/Q fa2fq Retrieve corresponding FASTQ records by a FASTA file FASTA/Q + only tab2fx Convert tabular format to FASTA/Q format TSV convert Convert FASTQ quality encoding between Sanger, Solexa and Illumina FASTA/Q Searching grep Search sequences by ID/name/sequence/sequence motifs, mismatch allowed FASTA/Q + and - partly, -m locate Locate subsequences/motifs, mismatch allowed FASTA/Q + and - partly, -m amplicon Extract amplicon (or specific region around it), mismatch allowed FASTA/Q + and - partly, -m fish Look for short sequences in larger sequences FASTA/Q + and - Set operation sample Sample sequences by number or proportion FASTA/Q rmdup Remove duplicated sequences by ID/name/sequence FASTA/Q + and - common Find common sequences of multiple files by id/name/sequence FASTA/Q + and - duplicate Duplicate sequences N times FASTA/Q split Split sequences into files by id/seq region/size/parts (mainly for FASTA) FASTA preffered split2 Split sequences into files by size/parts (FASTA, PE/SE FASTQ) FASTA/Q head Print first N FASTA/Q records FASTA/Q head-genome Print sequences of the first genome with common prefixes in name FASTA/Q range Print FASTA/Q records in a range (start:end) FASTA/Q pair Patch up paired-end reads from two fastq files FASTA/Q Edit replace Replace name/sequence by regular expression FASTA/Q + only rename Rename duplicated IDs FASTA/Q concat Concatenate sequences with same ID from multiple files FASTA/Q + only restart Reset start position for circular genome FASTA/Q + only mutate Edit sequence (point mutation, insertion, deletion) FASTA/Q + only sana Sanitize broken single line FASTQ files FASTQ Ordering sort Sort sequences by id/name/sequence/length FASTA preffered shuffle Shuffle sequences FASTA preffered BAM processing bam Monitoring and online histograms of BAM record features BAM Miscellaneous sum Compute message digest for all sequences in FASTA/Q files FASTA/Q \u2713 merge-slides Merge sliding windows generated from seqkit sliding TSV <p>Notes:</p> <ul> <li>Strand-sensitivity:<ul> <li><code>+ only</code>: only processing on the positive/forward strand.</li> <li><code>+ and -</code>: searching on both strands.</li> <li><code>+ or/and -</code>: depends on users' flags/options/arguments.</li> </ul> </li> <li>Multiple-threads: Using the default 4 threads is fast enough for most commands, some commands can benefit from extra threads.</li> </ul>"},{"location":"#citation","title":"Citation","text":"<ol> <li>Wei Shen*, Botond Sipos, and Liuyang Zhao. 2024. SeqKit2: A Swiss Army Knife for Sequence and Alignment Processing. iMeta e191. doi:10.1002/imt2.191. </li> <li>Wei Shen, Shuai Le, Yan Li*, and Fuquan Hu*. SeqKit: a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PLOS ONE. doi:10.1371/journal.pone.0163962. </li> </ol>"},{"location":"#contributors","title":"Contributors","text":"<ul> <li>Wei Shen</li> <li>Botond Sipos: <code>bam</code>, <code>scat</code>, <code>fish</code>, <code>sana</code>, <code>watch</code>.</li> <li>others</li> </ul>"},{"location":"#acknowledgements","title":"Acknowledgements","text":"<p>We thank all users for their valuable feedback and suggestions. We thank all contributors for improving the code and documentation.</p> <p>We appreciate Klaus Post for his fantastic packages ( compress and pgzip ) which accelerate gzip file reading and writing.</p>"},{"location":"#contact","title":"Contact","text":"<p>Create an issue to report bugs, propose new functions or ask for help.</p>"},{"location":"#license","title":"License","text":"<p>MIT License</p>"},{"location":"#starchart","title":"Starchart","text":""},{"location":"benchmark/","title":"Benchmark","text":"<p>Note: these benchmarks was performed in 2016.</p>"},{"location":"benchmark/#softwares","title":"Softwares","text":"<ol> <li>seqkit. (Go).    Version v0.3.1.1.    Compiled with Go 1.7rc5.</li> <li>fasta_utilities. (Perl).    Version 3dcc0bc.    Lots of dependencies to install.</li> <li>fastx_toolkit. (Perl).    Version 0.0.13.    Can't handle multi-line FASTA files.</li> <li>seqmagick. (Python).    Version 0.6.1</li> <li>seqtk. (C).    Version 1.1-r92-dirty.</li> </ol> <p>Not used:</p> <ol> <li>pyfaidx. (Python).    Version 0.4.7.1. Not used, because it exhausted my memory (10G) when computing reverse-complement on a 5GB fasta file of 250 bp.</li> </ol> <p>A Python script memusg was used to compute running time and peak memory usage of a process.</p>"},{"location":"benchmark/#features","title":"Features","text":"Categories Features seqkit fasta_utilities fastx_toolkit pyfaidx seqmagick seqtk Formats supports Multi-line FASTA Yes Yes -- Yes Yes Yes FASTQ Yes Yes Yes -- Yes Yes Multi-line  FASTQ Yes Yes -- -- Yes Yes Validating sequences Yes -- Yes Yes -- -- Supporting RNA Yes Yes -- -- Yes Yes Functions Searching by motifs Yes Yes -- -- Yes -- Sampling Yes -- -- -- Yes Yes Extracting sub-sequence Yes Yes -- Yes Yes Yes Removing duplicates Yes -- -- -- Partly -- Splitting Yes Yes -- Partly -- -- Splitting by seq Yes -- Yes Yes -- -- Shuffling Yes -- -- -- -- -- Sorting Yes Yes -- -- Yes -- Locating motifs Yes -- -- -- -- -- Common sequences Yes -- -- -- -- -- Cleaning bases Yes Yes Yes Yes -- -- Transcription Yes Yes Yes Yes Yes Yes Translation -- Yes Yes Yes Yes -- Filtering by size Indirect Yes -- Yes Yes -- Renaming header Yes Yes -- -- Yes Yes Other features Cross-platform Yes Partly Partly Yes Yes Yes Reading STDIN Yes Yes Yes -- Yes Yes Reading gzipped file Yes Yes -- -- Yes Yes Writing gzip file Yes -- -- -- Yes -- <p>Note 2: See usage for detailed options of seqkit.</p>"},{"location":"benchmark/#datasets","title":"Datasets","text":"<p>All test data is available here: seqkit-benchmark-data.tar.gz  (2.2G)</p>"},{"location":"benchmark/#dataset_afa-large-number-of-short-sequences","title":"dataset_A.fa - large number of short sequences","text":"<p>Dataset A is reference genomes DNA sequences of gastrointestinal tract from NIH Human Microbiome Project: <code>Gastrointestinal_tract.nuc.fsa</code> (FASTA format, ~2.7G).</p>"},{"location":"benchmark/#dataset_bfa-small-number-of-large-sequences","title":"dataset_B.fa - small number of large sequences","text":"<p>Dataset B is Human genome from ensembl.</p> <ul> <li>Genome DNA:  <code>Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz</code> (Gzipped FASTA file, ~900M) . Decompress it and rename to dataset_B.fa (~2.9G).</li> <li>GTF file:  <code>Homo_sapiens.GRCh38.84.gtf.gz</code> (~44M)</li> <li>BED file: <code>Homo_sapiens.GRCh38.84.bed.gz</code> was converted from <code>Homo_sapiens.GRCh38.84.gtf.gz</code> by  <code>gtf2bed</code>  with command<pre><code>$ zcat Homo_sapiens.GRCh38.84.gtf.gz | gtf2bed --do-not-sort | gzip -c &gt; Homo_sapiens.GRCh38.84.bed.gz\n</code></pre> </li> </ul>"},{"location":"benchmark/#dataset_cfq-illumina-single-end-reads-se100","title":"dataset_C.fq \u2013 Illumina single end reads (SE100)","text":"<p>Dataset C is Illumina single end (SE 100bp) reads file (~2.2G).</p> <p>Summary</p> <pre><code>$ seqkit stat *.fa\nfile          format  type   num_seqs        sum_len  min_len       avg_len      max_len\ndataset_A.fa  FASTA   DNA      67,748  2,807,643,808       56      41,442.5    5,976,145\ndataset_B.fa  FASTA   DNA         194  3,099,750,718      970  15,978,096.5  248,956,422\ndataset_C.fq  FASTQ   DNA   9,186,045    918,604,500      100           100          100\n</code></pre>"},{"location":"benchmark/#sequence-id-list","title":"Sequence ID list","text":"<p>Parts of sequences IDs was sampled and shuffled from original data. They were used in test of extracting sequences by ID list.</p> <p>Commands:</p> <pre><code>$ seqkit sample -p 0.3  dataset_A.fa | seqkit seq --name --only-id | shuf &gt; ids_A.txt\n$ seqkit sample -p 0.3  dataset_B.fa | seqkit seq --name --only-id | shuf &gt; ids_B.txt    \n$ seqkit sample -p 0.03 dataset_C.fq | seqkit seq --name --only-id | shuf &gt; ids_C.txt\n</code></pre> <p>Numbers:</p> <pre><code>$ wc -l ids*.txt\n    20138 ids_A.txt\n    58 ids_B.txt\n2754516 ids_C.txt\n</code></pre>"},{"location":"benchmark/#bed-file","title":"BED file","text":"<p>Only BED data of chromosome 19 was used in test of subsequence with BED file:</p> <pre><code>$ zcat Homo_sapiens.GRCh38.84.bed.gz | grep -E \"^19\" | gzip -c &gt; chr19.bed.gz\n</code></pre>"},{"location":"benchmark/#platform","title":"Platform","text":"<p>PC:</p> <ul> <li>CPU: Intel Core i5-3320M @ 2.60GHz, two cores/4 threads</li> <li>RAM: DDR3 1600MHz, 12GB</li> <li>SSD: SAMSUNG 850 EVO 250G, SATA-3</li> <li>OS: Fedora 24 (Scientific KDE spin),  Kernal: 4.6.4-301.fc24.x86_64</li> </ul> <p>Softwares:</p> <ul> <li>Perl: perl 5, version 22, subversion 2 (v5.22.2) built for x86_64-linux-thread-multi</li> <li>Python: Python 2.7.11 (default, Jul 10 2016, 20:58:20) [GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux2</li> </ul>"},{"location":"benchmark/#tests","title":"Tests","text":"<p>Automatic benchmark and plotting scripts are available at:  https://github.com/shenwei356/seqkit/tree/master/benchmark.</p> <p>All tests were repeated 3 times, and average time and peak memory ware used for plotting.</p> <p>All data were readed once before tests began to minimize the influence of page cache.</p> <p>Output sequences of all softwares were not wrapped to fixed length.</p>"},{"location":"benchmark/#test-1-reverse-complement","title":"Test 1. Reverse Complement","text":"<p><code>revcom_biogo</code> (source,  binary ),  a tool written in Golang (compiled with Go 1.6.3) using biogo  (Version 7ebd71b)  package,  was also used for comparison of FASTA file parsing performance.</p> <p>Note that some softwares (fasta_utilities and biogo) have different converting rules of computing complement sequence on ambiguous bases, there fore the results are different from others.</p> <p>Commands</p>"},{"location":"benchmark/#test-2-extracting-sequences-by-id-list","title":"Test 2. Extracting sequences by ID list","text":"<p>Commands</p>"},{"location":"benchmark/#test-3-sampling-by-number","title":"Test 3. Sampling by number","text":"<p>Note that different softwares have different sampling strategies, the peak memory  depends on size of sampled sequences and the results may not be the same.</p> <p>Commands</p>"},{"location":"benchmark/#test-4-removing-duplicates-by-sequence-content","title":"Test 4. Removing duplicates by sequence content","text":"<p>Commands</p>"},{"location":"benchmark/#test-5-subsequence-with-bed-file","title":"Test 5. Subsequence with BED file","text":"<p>Commands</p>"},{"location":"benchmark/#results","title":"Results","text":"<p>seqkit version: v0.3.1.1</p> <p>FASTA:</p> <p></p> <p>FASTQ:</p> <p></p>"},{"location":"benchmark/#test-of-multiple-threads","title":"Test of multiple threads:","text":"<p>From the results, 2 threads/CPU is enough, so the default threads of seqkit is 2.</p> <p></p>"},{"location":"benchmark/#tests-on-different-file-sizes","title":"Tests on different file sizes","text":"<p>Files are generated by replicating Human genome chr1 for N times.</p> <p></p> Please enable JavaScript to view the comments powered by Disqus."},{"location":"bioinf/","title":"Bioinf","text":""},{"location":"download/","title":"Download","text":"<p>SeqKit is implemented in Go programming language, statically-linked executable binary files are freely available.</p> <p>Please cite:</p> <ol> <li>Wei Shen*, Botond Sipos, and Liuyang Zhao. 2024. SeqKit2: A Swiss Army Knife for Sequence and Alignment Processing. iMeta e191. doi:10.1002/imt2.191. </li> <li>Wei Shen, Shuai Le, Yan Li*, and Fuquan Hu*. SeqKit: a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PLOS ONE. doi:10.1371/journal.pone.0163962. </li> </ol>"},{"location":"download/#current-version","title":"Current Version","text":"<ul> <li>SeqKit v2.8.2 - 2024-05-17 <ul> <li><code>seqkit amplicon</code>:<ul> <li>Fix a big introduced in v2.7.0. When more than one pairs of primers are given, only the last one is used. #457</li> </ul> </li> <li><code>seqkit translate</code>:<ul> <li>Add option <code>-e/--skip-translate-errors</code> to skip translate error and output empty sequence. #458</li> </ul> </li> <li><code>seqkit split</code>:<ul> <li>Add flag <code>-I/--ignore-case</code> for <code>-i/--by-id</code>. #462</li> </ul> </li> </ul> </li> </ul>"},{"location":"download/#links","title":"Links","text":"OS Arch File, \u4e2d\u56fd\u955c\u50cf Download Count Linux 32-bit seqkit_linux_386.tar.gz,  \u4e2d\u56fd\u955c\u50cf Linux 64-bit seqkit_linux_amd64.tar.gz,  \u4e2d\u56fd\u955c\u50cf Linux arm64 seqkit_linux_arm64.tar.gz,  \u4e2d\u56fd\u955c\u50cf macOS 64-bit seqkit_darwin_amd64.tar.gz,  \u4e2d\u56fd\u955c\u50cf macOS arm64 seqkit_darwin_arm64.tar.gz,  \u4e2d\u56fd\u955c\u50cf Windows 32-bit seqkit_windows_386.exe.tar.gz,  \u4e2d\u56fd\u955c\u50cf Windows 64-bit seqkit_windows_amd64.exe.tar.gz,  \u4e2d\u56fd\u955c\u50cf <p>Notes</p> <ul> <li>please open an issuse to request binaries for other platforms.</li> <li>run <code>seqkit version</code> to check update !!!</li> <li>run <code>seqkit genautocomplete</code> to update shell autocompletion script !!!</li> </ul>"},{"location":"download/#installation","title":"Installation","text":""},{"location":"download/#method-1-download-binaries-latest-stable-version","title":"Method 1: Download binaries (latest stable version)","text":"<p>Just download compressed executable file of your operating system, and decompress it with <code>tar -zxvf *.tar.gz</code> command or other tools. And then:</p> <ol> <li> <p>For Linux-like systems</p> <ol> <li> <p>If you have root privilege simply copy it to <code>/usr/local/bin</code>:</p> <pre><code>sudo cp seqkit /usr/local/bin/\n</code></pre> </li> <li> <p>Or copy to anywhere in the environment variable <code>PATH</code>:</p> <pre><code>mkdir -p $HOME/bin/; cp seqkit $HOME/bin/\n</code></pre> </li> </ol> </li> <li> <p>For windows, just copy <code>seqkit.exe</code> to <code>C:\\WINDOWS\\system32</code>.</p> </li> </ol>"},{"location":"download/#method-2-install-via-conda-latest-stable-version","title":"Method 2: Install via conda (latest stable version)","text":"<pre><code>conda install -c bioconda seqkit\n</code></pre>"},{"location":"download/#method-3-install-via-homebrew-might-not-be-latest-stable-version","title":"Method 3: Install via homebrew (might not be latest stable version)","text":"<pre><code>brew install seqkit\n</code></pre>"},{"location":"download/#method-4-for-go-developer-latest-stabledev-version","title":"Method 4: For Go developer (latest stable/dev version)","text":"<pre><code>go get -u github.com/shenwei356/seqkit/v2/seqkit/\n</code></pre>"},{"location":"download/#method-5-docker-based-installation-might-not-be-latest-stable-versio","title":"Method 5: Docker based installation (might not be latest stable versio)","text":"<p>Install Docker</p> <p>git clone this repo:</p> <pre><code>git clone https://github.com/shenwei356/seqkit\n</code></pre> <p>Run the following commands:</p> <pre><code>cd seqkit\ndocker build -t shenwei356/seqkit .\ndocker run -it shenwei356/seqkit:latest\n</code></pre>"},{"location":"download/#method-6-compiling-from-source-latest-stabledev-version","title":"Method 6: Compiling from source (latest stable/dev version)","text":"<pre><code># ------------------- install golang -----------------\n\n# download Go from https://go.dev/dl\nwget https://go.dev/dl/go1.17.13.linux-amd64.tar.gz\n\ntar -zxf go1.17.13.linux-amd64.tar.gz -C $HOME/\n\n# or \n#   echo \"export PATH=$PATH:$HOME/go/bin\" &gt;&gt; ~/.bashrc\n#   source ~/.bashrc\nexport PATH=$PATH:$HOME/go/bin\n\n\n# ------------- the latest stable version -------------\n\ngo get -v -u github.com/shenwei356/seqkit/seqkit\n\n# The executable binary file is located in:\n#   ~/go/bin/seqkit\n# You can also move it to anywhere in the $PATH\nmkdir -p $HOME/bin\ncp ~/go/bin/seqkit $HOME/bin/\n\n# --------------- the development version --------------\n\ngit clone https://github.com/shenwei356/seqkit\ncd seqkit/seqkit/\ngo build\n\n# The executable binary file is located in:\n#   ./seqkit\n# You can also move it to anywhere in the $PATH\nmkdir -p $HOME/bin\ncp ./seqkit $HOME/bin/\n</code></pre>"},{"location":"download/#shell-completion","title":"Shell-completion","text":"<p>Supported shell: bash|zsh|fish|powershell</p> <p>Bash:</p> <pre><code># generate completion shell\nseqkit genautocomplete --shell bash\n\n# configure if never did.\n# install bash-completion if the \"complete\" command is not found.\necho \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" &gt;&gt; ~/.bash_completion\necho \"source ~/.bash_completion\" &gt;&gt; ~/.bashrc\n</code></pre> <p>Zsh:</p> <pre><code># generate completion shell\nseqkit genautocomplete --shell zsh --file ~/.zfunc/_seqkit\n\n# configure if never did\necho 'fpath=( ~/.zfunc \"${fpath[@]}\" )' &gt;&gt; ~/.zshrc\necho \"autoload -U compinit; compinit\" &gt;&gt; ~/.zshrc\n</code></pre> <p>fish:</p> <pre><code>seqkit genautocomplete --shell fish --file ~/.config/fish/completions/seqkit.fish\n</code></pre>"},{"location":"download/#release-history","title":"Release history","text":"<ul> <li>SeqKit v2.8.1 - 2024-04-07 <ul> <li><code>seqkit sana</code>:<ul> <li>Add support for FASTQ files with IDs in the separator (<code>+</code>, 3rd) lines.. #446, #429, #408</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>Add some docs to show how to keep the original order of sequences when extracting with BED: compress the input FASTA file. #451</li> </ul> </li> </ul> </li> <li>SeqKit v2.8.0 - 2024-03-11 <ul> <li><code>seqkit stats</code>:<ul> <li>Add column <code>N50_num</code>, an alias of L50, #15.</li> </ul> </li> <li><code>seqkit seq/locate/fish/watch</code>:<ul> <li>Removing the flag <code>-V/--validate-seq-length</code>. Now the whole sequence will be checked if <code>-v/--validate-seq</code> is given.</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>Fix the speed problem, introduced in v2.7.0. #439.</li> <li>Slightly faster by reusing objects.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>Change the threshold sequence length for parallelizing complement sequence computation, 1kb-&gt;1Mb.</li> </ul> </li> </ul> </li> <li>SeqKit v2.7.0 - 2024-01-31 <ul> <li><code>seqkit</code>:<ul> <li>Grouping subcommands in help message, which is intuitive for beginners.</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>New flag: <code>-D/--allow-duplicated-patterns</code> for outputting records multiple times when duplicated patterns are given. #427</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>Use the ID regular expression from the option <code>--id-regexp</code> to create FASTA index file. This solves the panic happened for sequences containing tabs in the headers. #432</li> </ul> </li> <li><code>seqkit split/sort/shuffle</code>:<ul> <li>When using the two-pass mode (<code>-2/--two-pass</code>), replace possible tabs in the sequence header.</li> </ul> </li> <li><code>seqkit rmdup</code>:<ul> <li>Write an empty file of duplicate numbers and lists of IDs even if there's no duplicates when using <code>-D/--dup-num-file</code>. #436</li> </ul> </li> <li><code>seqkit stats</code>:<ul> <li>New flag <code>-S/--skip-file-check</code> to skip input file checking when given files or a file list. It's very useful if you run it with millions of files.</li> </ul> </li> </ul> </li> <li>SeqKit v2.6.1 - 2023-11-18 <ul> <li><code>seqkit</code>:<ul> <li>fix panic of nil pointer introduced in v2.6.0, which happens when handling multiple input files and some of them have file sizes of zero.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>fix panic (close of closed channel) when using <code>-v</code> to checking sequences.</li> </ul> </li> </ul> </li> <li>SeqKit v2.6.0 - 2023-11-09 <ul> <li><code>seqkit</code>:<ul> <li>add the shortcut <code>-X</code> for the flag <code>--infile-list</code>.</li> </ul> </li> <li><code>seqkit common</code>:<ul> <li>add a new flag <code>-e/--check-embedded-seqs</code> for detecting embedded sequences.</li> <li>for matching by sequences: reduced the memory occupation and corrected numbers in the log. #416</li> </ul> </li> <li><code>seqkit stat</code>:<ul> <li>add a new column <code>AvgQual</code> for average quality score. #411</li> </ul> </li> <li><code>seqkit split2</code>:<ul> <li>fix the panic for invalid input.</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>add a new flag <code>-R/--region-coord</code> for appending coordinates to sequence ID for <code>-r/--region</code>. #413</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>add a new flag <code>-s/--max-len-to-show</code> to show at most X characters for the search pattern or matched sequences.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>change the nucleotide color theme. #412</li> </ul> </li> </ul> </li> <li>SeqKit v2.5.1 - 2023-08-09 <ul> <li><code>seqkit stats</code>:<ul> <li>fix a concurrency bug (file name error) introduced in v2.5.0. #405</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>sequence/chromosome IDs are case-sensitive now. #400</li> </ul> </li> </ul> </li> <li>SeqKit v2.5.0 - 2023-07-16 <ul> <li>new command <code>seqkit merge-slides</code>: merge sliding windows generated from seqkit sliding. #390</li> <li><code>seqkit stats</code>:<ul> <li>added a new flag <code>-N/--N</code> for appending other N50-like stats as new columns. #393</li> <li>added a progress bar for &gt; 1 input files.</li> <li>write the result of each file immediately (no output buffer) when using <code>-T/--tabular</code>.</li> </ul> </li> <li><code>seqkit translate</code>:<ul> <li>add options <code>-s/--out-subseqs</code> and <code>-m/--min-len</code> to write ORFs longer than <code>x</code> amino acids as individual records. #389</li> </ul> </li> <li><code>seqkit sum</code>:<ul> <li>do not remove possible '*' by default and delete confusing warnings. Thanks to @photocyte. #399</li> <li>added a progress bar for &gt; 1 input files.</li> </ul> </li> <li><code>seqkit pair</code>:<ul> <li>remove the restriction of requiring FASTQ format, i.e., FASTA files are also supported.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>update help messages. #387</li> </ul> </li> <li><code>seqkit fxtab</code>:<ul> <li>faster alphabet computation (<code>-a/--alphabet</code>) with a new data structure. Thanks to @elliotwutingfeng #388</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>accept reverse coordinates in BED/GTF. #392</li> </ul> </li> </ul> </li> <li>SeqKit v2.4.0 - 2023-03-17 <ul> <li><code>seqkit</code>:<ul> <li>support <code>bzip2</code> format. #361</li> <li>support setting compression level for <code>gzip</code>, <code>zstd</code>, and <code>bzip2</code> format via <code>--compress-level</code>. #320</li> <li>the global flag <code>--infile-list</code> accepts stdin (<code>-</code>) now.</li> <li>wrap the help message of flags.</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>do not remove embeded regions when searching with regular expressions. #368</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>fix BED coordinates for amplicons found in the minus strand. #367</li> </ul> </li> <li><code>seqkit split</code>:<ul> <li>fix forgetting to add extension for <code>--two-pass</code>. #332</li> </ul> </li> <li><code>seqkit stats</code>:<ul> <li>fix compute Q1 and Q3 of sequence length for one record. #353</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>fix count number (<code>-C</code>) for matching with mismatch (<code>-m &gt; 0</code>). #370</li> </ul> </li> <li><code>seqkit replace</code>:<ul> <li>add some flags to match partly records to edit; these flags are transplanted from <code>seqkit grep</code>. #348</li> </ul> </li> <li><code>seqkit faidx</code>:<ul> <li>allow empty lines at the end of sequences.</li> </ul> </li> <li><code>seqkit faidx/sort/shuffle/split/subseq</code>:<ul> <li>new flag <code>-U/--update-faidx</code>: update the FASTA index file if it exists, to guarantee the index file matches the FASTA files. #364</li> <li>improve log info and update help message. #365</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>allow filtering sequences of length zero. thanks to @penglbio.</li> </ul> </li> <li><code>seqkit rename</code>:<ul> <li>new flag <code>-s/--separator</code> for setting separator between original ID/name and the counter (default \"_\"). #360</li> <li>new flag <code>-N/--start-num</code> for setting starting count number for duplicated IDs/names (default 2). #360</li> <li>new flag <code>-1/--rename-1st-rec</code> for renaming the first record as well. #360</li> <li>do not append space if there's no description after the sequene ID.</li> </ul> </li> <li><code>seqkit sliding</code>:<ul> <li>new flag <code>-S/--suffix</code> for change the suffix added to the sequence ID (default: \"_sliding\").</li> </ul> </li> </ul> </li> <li>SeqKit v2.3.1 - 2022-09-22 <ul> <li><code>seqkit grep/locate</code>: fix bug of FMIndex building for empty sequences. #321</li> <li><code>seqkit split2</code>: fix bug of splitting two FASTA files. #325</li> <li><code>seqkit faidx</code>: <code>--id-regexp</code> works now.</li> </ul> </li> <li>SeqKit v2.3.0 - 2022-08-12 <ul> <li><code>seqkit grep/rename</code>:<ul> <li>reduce memory comsumption for a lot of searching patterns, and it's faster. #305</li> <li>2X faster <code>-s/--by-seq</code>.</li> </ul> </li> <li><code>seqkit split</code><ul> <li>fix outputting an empty file when the number of sequence equal to the split size. #293</li> <li>add options to set output file prefix and extention. #296</li> </ul> </li> <li><code>seqkit split2</code><ul> <li>reduce memory consumption. #304</li> <li>add options to set output file prefix</li> </ul> </li> <li><code>seqkit stats</code>:<ul> <li>add GC content. #294</li> </ul> </li> </ul> </li> <li> <p>SeqKit v2.2.0 - 2020-03-14 </p> <ul> <li><code>seqkit</code>:<ul> <li>add support of <code>xz</code> and <code>zstd</code> input/output formats. #274</li> <li>fix panic when reading records with header of <code>ID</code> + blanks.</li> </ul> </li> <li>new command <code>seqkit sum</code>: computing message digest for all sequences in FASTA/Q files.   The idea comes from @photocyte and the format borrows from seqhash #262</li> <li>new command <code>seqkit fa2fq</code>: retrieving corresponding FASTQ records by a FASTA file</li> <li><code>seqkit split2</code>:<ul> <li>new flag <code>-e/--extension</code> for forcing compresson or changing compression format. #276</li> <li>support changing output prefix via <code>-o/--out-file</code>. #275</li> </ul> </li> <li><code>seqkit concat</code>: <ul> <li>fix handling of multiple seqs with the same ID in one file. #269</li> <li>performaning out/full join. #270</li> <li>preserve the comments. #271</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>parallelizing <code>-F/--use-fmi</code> and <code>-m</code> for large number of search patterns.</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>new flag <code>-M/--output-mismatches</code> to append the total mismatches and mismatches of 5' end and 3' end. #286</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>detect FASTA/Q symbol <code>@</code> and <code>&gt;</code> in the searching patterns and show warnings.</li> <li>add new flag <code>-C/--count</code>, like <code>grep -c</code> in GNU grep. #267</li> </ul> </li> <li><code>seqkit range</code>:<ul> <li>support removing leading 100 seqs (<code>seqkit range -r 101:-1</code> == <code>tail -n +101</code>). #279</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>report error when no options were given.</li> </ul> </li> <li>update doc:<ul> <li><code>seqkit head</code>: add doc for \"seqkit tail\": <code>seqkit range -N:-1 seqs.fasta</code>. #272</li> <li><code>seqkit rmdup</code>: add the note of only the first record being saved for duplicates. #265</li> </ul> </li> </ul> </li> <li> <p>SeqKit v2.1.0 - 2021-11-15 </p> <ul> <li><code>seqkit seq</code>:<ul> <li>fix filtering by average quality <code>-Q/-R</code>. #257</li> </ul> </li> <li><code>seqkit convert</code>:<ul> <li>fix quality encoding checking, change default value of <code>-N/--thresh-B-in-n-most-common</code> from <code>4</code> to <code>2</code>.  #254 and #239</li> </ul> </li> <li><code>seqkit split</code>:<ul> <li>fix writing an extra empty file when using <code>--two-pass</code>#244</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>fix <code>--bed</code> which fail to recognize strand <code>.</code>.</li> </ul> </li> <li><code>seqkit fq2fa</code>: <ul> <li>faster, and do not wrap sequences.</li> </ul> </li> <li><code>seqkit grep/locate/mutate</code>:<ul> <li>detect unquoted comma and show warning message, e.g., <code>-p 'A{2,}'</code>. #250</li> </ul> </li> </ul> </li> <li>SeqKit v2.0.0 - 2021-08-27 <ul> <li>Performance improvements<ul> <li><code>seqkit</code>:<ul> <li>faster FASTA/Q reading and writing, especially on FASTQ, see the benchmark.<ul> <li>reading (plain text): 4X faster. <code>seqkit stats dataset_C.fq</code></li> <li>reading (gzip files): 45% faster. <code>seqkit stats dataset_C.fq.gz</code></li> <li>reading + writing (plain text): 3.5X faster. <code>seqkit grep -p . -v  dataset_C.fq -o t</code></li> <li>reading + writing (gzip files): 2.2X faster. <code>seqkit grep -p . -v  dataset_C.fq.gz -o t.gz</code></li> </ul> </li> <li>change default value of <code>-j/--threads</code> from 2 to 4, which is faster for writting gzip files.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>fix writing speed, which was slowed down in v0.12.1.</li> </ul> </li> </ul> </li> <li>Breaking changes<ul> <li><code>seqkit grep/rmdup/common</code>: <ul> <li>consider reverse complement sequence by default for comparing by sequence, add flag <code>-P/--only-positive-strand</code>. #215</li> </ul> </li> <li><code>seqkit rename</code>:<ul> <li>rename ID only, do not append original header to new ID. #236</li> </ul> </li> <li><code>seqkit fx2tab</code>:<ul> <li>for <code>-s/--seq-hash</code>: outputing MD5 instead of hash value (integers) of xxhash. #219</li> </ul> </li> </ul> </li> <li>Bugfixes<ul> <li><code>seqkit seq</code>:<ul> <li>fix failing to output gzipped format for file name with extension of <code>.gz</code> since v0.12.1.</li> </ul> </li> <li><code>seqkit tab2fx</code>:<ul> <li>fix bug for very long sequences. #214</li> </ul> </li> <li><code>seqkit fish</code>:<ul> <li>fix range check. #213</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>it's not exactly a bug: forgot to use multi-threads for <code>-m</code> &gt; 0.</li> </ul> </li> </ul> </li> <li>New features/enhancements<ul> <li><code>seqkit grep</code>: <ul> <li>allow empty pattern files.</li> </ul> </li> <li><code>seqkit faidx</code>:<ul> <li>support region with <code>begin &gt; end</code>, i.e., returning reverse complement sequence</li> <li>add new flag <code>-l/--region-file</code>:  file containing a list of regions.</li> </ul> </li> <li><code>seqkit fx2tab</code>:<ul> <li>new flag <code>-Q/--no-qual</code> for disabling outputing quality even for FASTQ file. #221</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>new flag <code>-u/--save-unmatched</code> for saving records that do not match any primer.</li> </ul> </li> <li><code>seqkit sort</code>:<ul> <li>new flag <code>-b/--by-bases</code> for sorting by non-gap bases, for multiple sequence alignment files.#216</li> </ul> </li> </ul> </li> </ul> </li> <li>SeqKit v0.16.1 - 2021-05-20 <ul> <li><code>seqkit shuffle --two-pass</code>: fix bug introduced in #173 . #209</li> <li><code>seqkit pair</code>: fix a dangerous bug: when input files are not in current directory, input files were overwritten.</li> </ul> </li> <li>SeqKit v0.16.0 - 2021-04-16 <ul> <li>new command <code>seqkit head-genome</code>:<ul> <li>print sequences of the first genome with common prefixes in name</li> </ul> </li> <li><code>seqkit grep/locate/amplicon -m</code><ul> <li>much faster (300-400x) searching with mismatch allowed by optimizing FM-indexing and parallelization.</li> <li>new flag <code>-I/--immediate-output</code>.</li> </ul> </li> <li><code>seqkit grep/locate</code>:<ul> <li>fix bug of <code>-m</code> when querying contains letters not in alphabet, usually for protein sequences. #178, #179</li> <li>onply search on positive strand when searching unlimited or protein sequences.</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>removing debug info for <code>-r</code> introduced in a0f6b6e. #180</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>fix bug of <code>-m</code>, when mismatch is allowed.</li> </ul> </li> <li><code>seqkit fx2tab</code>:<ul> <li>new flag <code>-C/--base-count</code> for counting bases. #183</li> </ul> </li> <li><code>seqkit tab2fx</code>:<ul> <li>fix a rare bug. #197</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>fix bug for BED with empty columns. #195</li> </ul> </li> <li><code>seqkit genautocomplete</code>: <ul> <li>support bash|zsh|fish|powershell.</li> </ul> </li> </ul> </li> <li>SeqKit v0.15.0 - 2021-01-12 <ul> <li><code>seqkit grep/locate</code>: update help message.</li> <li><code>seqkit grep</code>: search on both strand when searching by sequence.</li> <li><code>seqkit split2</code>: fix redundant log when using <code>-s</code>.</li> <li><code>seqkit bam</code>: new field <code>RightSoftClipSeq</code>. #172</li> <li><code>seqkit sample -2</code>: remove extra <code>\\n</code>. #173</li> <li><code>seqkit split2 -l</code>: fix bug for splitting by accumulative length, this bug occurs when the first record is longer than <code>-l</code>, no sequences are lost.</li> </ul> </li> <li>SeqKit v0.14.0 - 2020-10-30 <ul> <li>new command <code>seqkit pair</code>: match up paired-end reads from two fastq files, faster than fastq-pair.</li> <li><code>seqkit translate</code>: new flag <code>-F/--append-fram</code> for optional adding frame info to ID. #159</li> <li><code>seqkit stats</code>: reduce memory usage when using <code>-a</code> for calculating N50. #153</li> <li><code>seqkit mutate</code>: fix inserting sequence <code>-i/--insertion</code>,     this bug occurs when <code>insert site</code> is big in some cases, don't worry if no error reported.</li> <li><code>seqkit replace</code>:<ul> <li>new flag <code>-U/--keep-untouched</code>: do not change anything when no value found for the key (only for sequence name).</li> <li>do no support editing FASTQ sequence.</li> </ul> </li> <li><code>seqkit grep/locate</code>: new flag <code>--circular</code> for supporting circular genome. #158</li> </ul> </li> <li>SeqKit v0.13.2 - 2020-07-13 <ul> <li><code>seqkit sana</code>: fix bug causing hanging on empty files. #149</li> </ul> </li> <li>SeqKit v0.13.1 - 2020-07-09 <ul> <li><code>seqkit sana</code>: fix bug causing hanging on empty files. #148</li> </ul> </li> <li>SeqKit v0.13.0 - 2020-07-07 <ul> <li><code>seqkit</code>: fix a rare FASTA/Q parser bug. #127</li> <li><code>seqkit seq</code>: output sequence or quality in single line when <code>-s/--seq</code> or <code>-q/--qual</code> is on. #132</li> <li><code>seqkit translate</code>: delete debug info, #133, and fix typo. #134</li> <li><code>seqkit split2</code>: tiny performance improvement. #137</li> <li><code>seqkit stats</code>: new flag <code>-i/--stdin-label</code> for replacing default \"-\" for stdin. #139</li> <li><code>seqkit fx2tab</code>: new flag <code>-s/--seq-hash</code> for printing hash of sequence (case sensitive). #144</li> <li><code>seqkit amplicon</code>:<ul> <li>fix bug of missing searching reverse strand. #140</li> <li>supporting degenerate bases now. #83</li> <li>new flag <code>-p/--primer-file</code> for reading list of primer pairs. #142</li> <li>new flag <code>--bed</code> for outputing in BED6+1 format. #141</li> </ul> </li> <li>New features and improvements by @bsipos. #130, #147<ul> <li>new command <code>seqkit scat</code>, for real-time robust concatenation of fastx files. </li> <li>Rewrote the parser behind the <code>sana</code> subcommand, now it supports robust parsing of fasta file as well.</li> <li>Added a \"toolbox\" feature to the <code>bam</code> subcommand (<code>-T</code>), which is a collection of filters acting on streams of BAM records configured through a YAML string (see the docs for more).</li> <li>Added the <code>SEQKIT_THREADS</code> environmental variable to override the default number of threads.</li> </ul> </li> </ul> </li> <li>SeqKit v0.12.1 - 2020-04-21 <ul> <li><code>seqkit bam</code>: add colorised and pretty printed output, by @bsipos. #110</li> <li><code>seqkit locate/grep</code>: fix bug of <code>-m</code>, when query contains letters not in subject sequences. #124</li> <li><code>seqkit split2</code>: new flag <code>-l/--by-length</code> for splitting into chunks of N bases.</li> <li><code>seqkit fx2tab</code>:<ul> <li>new flag <code>-I/--case-sensitive</code> for calculating case sensitive base content. #108</li> <li>add missing column name for averge quality for <code>-H -q</code>. #115</li> <li>fix output of <code>-n/--only-name</code>, do not write empty columns of sequence and quality. #104, #115</li> </ul> </li> <li><code>seqkit seq</code>: new flag <code>-k/--color</code>: colorize sequences.</li> </ul> </li> <li>SeqKit v0.12.0 - 2020-02-18 <ul> <li><code>seqkit</code>:<ul> <li>fix checking input file existence.</li> <li>new global flag <code>--infile-list</code> for long list of input files, if given, they are appended to files from cli arguments.</li> </ul> </li> <li><code>seqkit faidx</code>: supporting \"truncated\" (no ending newline charactor) file.</li> <li><code>seqkit seq</code>:<ul> <li>do not force switching on <code>-g</code> when using <code>-m/-M</code>.</li> <li>show recommendation if flag <code>-t/--seq-type</code> is not DNA/RNA when computing complement sequence. #103</li> </ul> </li> <li><code>seqkit translate</code>: supporting multiple frames. #96</li> <li><code>seqkit grep/locate</code>:<ul> <li>add detection and warning for space existing in search pattern/sequence.</li> <li>speed improvement (2X) for <code>-m/--max-mismatch</code>. shenwei356/bwt/issues/3</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>new flag <code>-M/--hide-matched</code> for hiding matched sequences. #98</li> <li>new flag <code>-r/--use-regexp</code> for explicitly using regular expression, so improve speed of default <code>index</code> operation. And you have to switch this on if using regexp now. #101</li> <li>new flag <code>-F/--use-fmi</code> for improving search speed for lots of sequence patterns.</li> </ul> </li> <li><code>seqkit rename</code>: making IDs unique across multiple files, and can write into multiple files. #100</li> <li><code>seqkit sample</code>: fix stdin checking for flag <code>-2</code>. #102.</li> <li><code>seqkit rename/split/split2</code>: fix detection of existed outdir.</li> <li><code>split split</code>: fix bug of <code>seqkit split -i -2</code> and parallizing it.</li> <li><code>seqkit version</code>: checking update is optional (<code>-u</code>).</li> </ul> </li> <li>SeqKit v0.11.0 - 2019-09-25 <ul> <li><code>seqkit</code>: fix hanging when reading from truncated gzip file.</li> <li>new commands:<ul> <li><code>seqkit amplicon</code>: retrieve amplicon (or specific region around it) via primer(s).</li> </ul> </li> <li>new commands by @bsipos:<ul> <li><code>seqkit watch</code>: monitoring and online histograms of sequence features.</li> <li><code>seqkit sana</code>: sanitize broken single line fastq files.</li> <li><code>seqkit fish</code>: look for short sequences in larger sequences using local alignment.</li> <li><code>seqkit bam</code>: monitoring and online histograms of BAM record features.</li> </ul> </li> <li><code>seqkit grep/locate</code>: reduce memory occupation when using flag <code>-m/--max-mismatch</code>.</li> <li><code>seqkit seq</code>: fix panic of computing complement sequence for long sequences containing illegal letters without flag <code>-v</code> on. #84</li> </ul> </li> <li>SeqKit v0.10.2 - 2019-07-30 <ul> <li><code>seqkit</code>: fix bug of parsing sequence ID delimited by tab (<code>\\t</code>). #78</li> <li><code>seqkit grep</code>: better logic of <code>--delete-matched</code>.</li> <li><code>seqkit common/rmdup/split</code>: use xxhash to replace MD5 when comparing with sequence, discard flag <code>-m/--md5</code>.</li> <li><code>seqkit stats</code>: new flag <code>-b/--basename</code> for outputting basename instead of full path.</li> </ul> </li> <li>SeqKit v0.10.1 - 2019-02-27 <ul> <li><code>seqkit fx2tab</code>: new option <code>-q/--avg-qual</code> for outputting average read quality. #60</li> <li><code>seqkit grep/locate</code>: fix support of <code>X</code> when using <code>-d/--degenerate</code>. #61</li> <li><code>seqkit translate</code>:<ul> <li>new flag <code>-M/--init-codon-as-M</code> to translate initial codon at beginning to 'M'. #62</li> <li>translates <code>---</code> to <code>-</code> for aligned DNA/RNA, flag <code>-X</code> needed. #63</li> <li>supports codons containing ambiguous bases, e.g., <code>GGN-&gt;G</code>, <code>ATH-&gt;I</code>. #64</li> <li>new flag <code>-l/--list-transl-table</code> to show details of translate table N</li> <li>new flag <code>-l/--list-transl-table-with-amb-codons</code> to show details of translate table N (including ambigugous codons)</li> </ul> </li> <li><code>seqkit split/split2</code>, fix bug of ignoring <code>-O</code> when reading from stdin.</li> </ul> </li> <li>SeqKit v0.10.0 - 2018-12-24 <ul> <li><code>seqkit</code>: report error when input is directory.</li> <li>new command <code>seqkit mutate</code>: edit sequence (point mutation, insertion, deletion).</li> </ul> </li> <li>SeqKit v0.9.3 - 2018-12-02 <ul> <li><code>seqkit stats</code>: fix panic for empty file. #57</li> <li><code>seqkit translate</code>: add flag <code>-x/--allow-unknown-codon</code> to translate unknown codon to <code>X</code>.</li> </ul> </li> <li>SeqKit v0.9.2 - 2018-11-16 <ul> <li><code>seqkit</code>: stricter checking for value of global flag <code>-t/--seq-type</code>.</li> <li><code>seqkit sliding</code>: fix bug for flag <code>-g/--greedy</code>. #54</li> <li><code>seqkit translate</code>: fix bug for frame &lt; 0. #55</li> <li><code>seqkit seq</code>: add TAB to default blank characters (flag <code>-G/--gap-letters</code>), and fix filter result when using flag <code>-g/--remove-gaps</code> along with <code>-m/--min-len</code> or <code>-M/--max-len</code></li> </ul> </li> <li>SeqKit v0.9.1 - 2018-10-12 <ul> <li><code>seqkit faidx</code>: fix bug of retrieving subsequence with multiple regions on same sequence. #48</li> <li><code>seqkit sort/shuffle/split</code>: fix bug when using <code>-2/--two-pass</code> to process <code>.gz</code> files. #52</li> </ul> </li> <li>SeqKit v0.9.0 - 2018-09-26 <ul> <li><code>seqkit</code>: better handle of empty file, no error message shown. #36</li> <li>new subcommand <code>seqkit split2</code>: split sequences into files by size/parts (FASTA, PE/SE FASTQ).  #35</li> <li>new subcommand <code>seqkit translate</code>: translate DNA/RNA to protein sequence. #28</li> <li><code>seqkit sort</code>: fix bug when using <code>-2 -i</code>, and add support for sorting in natural order. #39</li> <li><code>seqkit grep</code> and <code>seqkit locate</code>: add experimental support of mismatch when searching subsequences. #14</li> <li><code>seqkit stats</code>: add stats of Q20 and Q30 for FASTQ. #45</li> </ul> </li> <li>SeqKit v0.8.1 - 2018-06-29 <ul> <li><code>seqkit</code>: do not call <code>pigz</code> or <code>gzip</code> for decompressing gzipped file any more. But you can still utilize <code>pigz</code> or <code>gzip</code> by <code>pigz -d -c seqs.fq.gz | seqkit xxx</code>.</li> <li><code>seqkit subseq</code>: fix bug of missing quality when using <code>--gtf</code> or <code>--bed</code></li> <li><code>seqkit stats</code>: parallelize counting files, it's much faster for lots of small files, especially for files on SSD</li> </ul> </li> <li>SeqKit v0.8.0 - 2018-03-22 </li> <li><code>seqkit</code>, stricter FASTA/Q format requirement, i.e., must starting with <code>&gt;</code> or <code>@</code>.</li> <li><code>seqkit</code>, fix output format for FASTQ files containing zero-length records, yes this happens.</li> <li><code>seqkit</code>, add amino acid code <code>O</code> (pyrrolysine) and <code>U</code> (selenocysteine).</li> <li><code>seqkit replace</code>, add flag <code>--nr-width</code> to fill leading 0s for <code>{nr}</code>, useful for preparing sequence submission (\"&gt;strain_00001 XX\", \"&gt;strain_00002 XX\").</li> <li><code>seqkit subseq</code>, require BED file to be tab-delimited.</li> <li>SeqKit v0.7.2 - 2017-12-03 <ul> <li><code>seqkit tab2fx</code>: fix a concurrency bug that occurs in low proprobability when only 1-column data provided.</li> <li><code>seqkit stats</code>: add quartiles of sequence length</li> <li><code>seqkit faidx</code>: add support for retrieving subsequence using seq ID and region, which is similar with \"samtools faidx\" but has some extra features</li> </ul> </li> <li>SeqKit v0.7.1 - 2017-09-22 <ul> <li><code>seqkit convert</code>: fix bug of read quality containing only 3 or less values.  shenwei356/bio/issues/3</li> <li><code>seqkit stats</code>: add option <code>-T/--tabular</code> to output in machine-friendly tabular format.   #23</li> <li><code>seqkit common</code>: increase speed and decrease memory occupation, and add some notes.</li> <li>fix some typos. #22</li> <li>suggestion: please install pigz to gain better parsing performance for gzipped data.</li> </ul> </li> <li>SeqKit v0.7.0 - 2017-08-12 <ul> <li>add new command <code>convert</code> for converting FASTQ quality encoding between Sanger, Solexa and Illumina. Thanks suggestion from @cviner ( #18). usage &amp; example.</li> <li>add new command <code>range</code> for printing FASTA/Q records in a range (start:end). #19. usage &amp; example.</li> <li>add new command <code>concate</code> for concatenating sequences with same ID from multiple files. usage &amp; example.</li> </ul> </li> <li>SeqKit v0.6.0 - 2017-06-21 <ul> <li>add new command <code>genautocomplete</code> to generate shell autocompletion script! (#17)</li> <li>add new command <code>seqkit dup</code> for duplicating sequences (#16)</li> <li><code>seqkit stats -a</code> does not show L50 which may brings confusion (#15)</li> <li><code>seqkit subseq --bed</code>: more robust for bad BED files</li> </ul> </li> <li>SeqKit v0.5.5 - 2017-05-10 <ul> <li>Increasing speed of reading <code>.gz</code> file by utilizing <code>gzip</code> (1.3X),     it would be much faster if you installed <code>pigz</code> (2X).</li> <li>Fixing colorful output in Windows</li> <li><code>seqkit locate</code>: add flag <code>--gtf</code> and <code>--bed</code> to output GTF/BED6 format,     so the result can be used in <code>seqkit subseq</code>.</li> <li><code>seqkit subseq</code>: fix bug of <code>--bed</code>, add checking coordinate.</li> </ul> </li> <li>SeqKit v0.5.4 - 2017-04-11 <ul> <li><code>seqkit subseq --gtf</code>, add flag <code>--gtf-tag</code> to set tag that's outputted as sequence comment</li> <li>fix <code>seqkit split</code> and <code>seqkit sample</code>: forget not to wrap sequence and quality in output for FASTQ format</li> <li>compile with go1.8.1</li> </ul> </li> <li>SeqKit v0.5.3 - 2017-04-01 <ul> <li><code>seqkit grep</code>: fix bug when using <code>seqkit grep -r -f patternfile</code>:   all records will be retrived due to failing to discarding the blank pattern (<code>\"\"</code>). #11</li> </ul> </li> <li>SeqKit v0.5.2 - 2017-03-24 <ul> <li><code>seqkit stats -a</code> and <code>seqkit seq -g -G</code>: change default gap letters from '- ' to '- .'</li> <li><code>seqkit subseq</code>: fix bug of range overflow when using <code>-d/--down-stream</code>     or <code>-u/--up-stream</code> for retieving subseq using BED (<code>--beb</code>) or GTF (<code>--gtf</code>) file.</li> <li><code>seqkit locate</code>: add flag <code>-G/--non-greedy</code>, non-greedy mode,  faster but may miss motifs overlaping with others.</li> </ul> </li> <li>SeqKit v0.5.1 - 2017-03-12 <ul> <li><code>seqkit restart</code>: fix bug of flag parsing</li> </ul> </li> <li>SeqKit v0.5.0 - 2017-03-11 <ul> <li>new command <code>seqkit restart</code>, for resetting start position for circular genome.</li> <li><code>seqkit sliding</code>: add flag <code>-g/--greedy</code>, exporting last subsequences even shorter than windows size.</li> <li><code>seqkit seq</code>:<ul> <li>add flag <code>-m/--min-len</code> and <code>-M/--max-len</code> to filter sequences by length.</li> <li>rename flag <code>-G/--gap-letter</code> to <code>-G/--gap-letters</code>.</li> </ul> </li> <li><code>seqkit stat</code>:<ul> <li>renamed to <code>seqkit stats</code>, don't worry, old name is still available as an alias.</li> <li>add new flag <code>-a/all</code>, for all statistics, including <code>sum_gap</code>, <code>N50</code>, and <code>L50</code>.</li> </ul> </li> </ul> </li> <li>SeqKit v0.4.5 - 2017-02-26 <ul> <li><code>seqkit seq</code>: fix bug of failing to reverse quality of FASTQ sequence</li> </ul> </li> <li>SeqKit v0.4.4 - 2017-02-17 <ul> <li><code>seqkit locate</code>: fix bug of missing regular-expression motifs containing  non-DNA characters (e.g., <code>ACT.{6,7}CGG</code>) from motif file (<code>-f</code>).</li> <li>compiled with go v1.8.</li> </ul> </li> <li>SeqKit v0.4.3 - 2016-12-22 <ul> <li>fix bug of <code>seqkit stat</code>: <code>min_len</code> always be <code>0</code> in versions: v0.4.0, v0.4.1, v0.4.2</li> </ul> </li> <li>SeqKit v0.4.2 - 2016-12-21 <ul> <li>fix header information of <code>seqkit subseq</code> when restriving up- and down-steam sequences using GTF/BED file.</li> </ul> </li> <li>SeqKit v0.4.1 - 2016-12-16 <ul> <li>enchancement: remove redudant regions for <code>seqkit locate</code>.</li> </ul> </li> <li>SeqKit v0.4.0 - 2016-12-07 <ul> <li>fix bug of <code>seqkit locate</code>, e.g, only find two locations (<code>1-4</code>, <code>7-10</code>, missing <code>4-7</code>) of <code>ACGA</code> in <code>ACGACGACGA</code>.</li> <li>better output of <code>seqkit stat</code> for empty file.</li> </ul> </li> <li>SeqKit v0.3.9 - 2016-12-04 <ul> <li>fix bug of region selection for blank sequences. affected commands include <code>seqkit subseq --region</code>, <code>seqkit grep --region</code>, <code>seqkit split --by-region</code>.</li> <li>compile with go1.8beta1.</li> </ul> </li> <li>SeqKit v0.3.8.1 - 2016-11-25 <ul> <li>enhancement and bugfix of <code>seqkit common</code>: two or more same files allowed, fix log information of number of extracted sequences in the first file.</li> </ul> </li> <li>SeqKit v0.3.8 - 2016-12-24 <ul> <li>enhancement of <code>seqkit common</code>: better handling of files containing replicated sequences</li> </ul> </li> <li>SeqKit v0.3.7 - 2016-12-23 <ul> <li>fix bug in <code>seqkit split --by-id</code> when sequence ID contains invalid characters for system path.</li> <li>add more flags validation for <code>seqkit replace</code>.</li> <li>enhancement: raise error when key pattern matches multiple targes in cases of replacing with key-value files and more controls are added.</li> <li>changes: do not wrap sequence and quality in output for FASTQ  format.</li> </ul> </li> <li>SeqKit v0.3.6 - 2016-11-03 <ul> <li>add new feature for <code>seqkit grep</code>: new flag <code>-R</code> (<code>--region</code>) for specifying sequence region for searching.</li> </ul> </li> <li>SeqKit v0.3.5 - 2016-10-30 <ul> <li>fig bug of <code>seqkit grep</code>: flag <code>-i</code> (<code>--ignore-case</code>) did not work when not using regular expression</li> </ul> </li> <li>SeqKit v0.3.4.1 - 2016-09-21 <ul> <li>improve performance of reading (~10%) and writing (100%) gzip-compressed file by using <code>github.com/klauspost/pgzip</code> package</li> <li>add citation</li> </ul> </li> <li>SeqKit v0.3.4 - 2016-09-17   <ul> <li>bugfix: <code>seq</code> wrongly handles only the first one sequence file when multiple files given</li> <li>new feature: <code>fx2tab</code> can output alphabet letters of a sequence by flag <code>-a</code> (<code>--alphabet</code>)</li> <li>new feature: new flag <code>-K</code> (<code>--keep-key</code>) for <code>replace</code>,  when replacing with key-value file, one can choose keeping the key as value or not.</li> </ul> </li> <li>SeqKit v0.3.3 - 2016-08-18   <ul> <li>fix bug of <code>seqkit replace</code>, wrongly starting from 2 when using <code>{nr}</code>   in <code>-r</code> (<code>--replacement</code>)</li> <li>new feature: <code>seqkit replace</code> supports replacement symbols <code>{nr}</code> (record number)   and <code>{kv}</code> (corresponding value of the key ($1) by key-value file)</li> </ul> </li> <li>SeqKit v0.3.2 - 2016-08-13   <ul> <li>fix bug of <code>seqkit split</code>, error when target file is in a directory.</li> <li>improve performance of <code>seqkit spliding</code> for big sequences, and output   last part even if it's shorter than window sze,   output of FASTQ is also supported.</li> </ul> </li> <li>SeqKit v0.3.1.1 - 2016-08-07   <ul> <li>compile with go1.7rc5, with higher performance and smaller size of binary file</li> </ul> </li> <li>SeqKit v0.3.1 - 2016-08-02   <ul> <li>improve speed of <code>seqkit locate</code></li> </ul> </li> <li>SeqKit v0.3.0 - 2016-07-28   <ul> <li>use fork of github.com/brentp/xopen, using <code>zcat</code> for speedup of .gz file   reading on *nix systems.</li> <li>improve speed of parsing sequence ID when creating FASTA index</li> <li>reduce memory usage of <code>seqkit subseq --gtf</code></li> <li>fix bug of <code>seqkit subseq</code> when using flag <code>--id-ncbi</code></li> <li>fix bug of <code>seqkit split</code>, outdir error</li> <li>fix bug of <code>seqkit seq -p</code>, last base is wrongly failed to convert when   sequence length is odd.</li> <li>add \"sum_len\" result for output of <code>seqkit stat</code></li> </ul> </li> <li>seqkit v0.2.9 - 2016-07-24   <ul> <li>fix minor bug of <code>seqkit split</code> and <code>seqkit shuffle</code>,   header name error due to improper use of pointer</li> <li>add option <code>-O (--out-dir)</code> to <code>seqkit split</code></li> </ul> </li> <li>seqkit v0.2.8 - 2016-07-19   <ul> <li>improve speed of parsing sequence ID, not using regular expression for default <code>--id-regexp</code></li> <li>improve speed of record outputing for small-size sequences</li> <li>fix minor bug: <code>seqkit seq</code> for blank record</li> <li>update benchmark result</li> </ul> </li> <li>seqkit v0.2.7 - 2016-07-18   <ul> <li>reduce memory usage by optimize the outputing of sequences.   detail: using <code>BufferedByteSliceWrapper</code> to resuse bytes.Buffer.</li> <li>reduce memory usage and improve speed by using custom buffered  reading mechanism, instead of using standard library <code>bufio</code>,   which is slow for large genome sequence.</li> <li>discard strategy of \"buffer\" and \"chunk\" of FASTA/Q records,   just parse records one by one.</li> <li>delete global flags <code>-c (--chunk-size)</code> and <code>-b (--buffer-size)</code>.</li> <li>add function testing scripts</li> </ul> </li> <li>seqkit v0.2.6 - 2016-07-01   <ul> <li>fix bug of <code>seqkit subseq</code>: Inplace subseq method leaded to wrong result</li> </ul> </li> <li>seqkit v0.2.5.1 <ul> <li>fix a bug of <code>seqkit subseq</code>: chromesome name was not be converting to lower case when using <code>--gtf</code> or <code>--bed</code></li> </ul> </li> <li>seqkit v0.2.5 - 2016-07-01   <ul> <li>fix a serious bug brought in <code>v0.2.3</code>, using unsafe method to convert <code>string</code> to <code>[]byte</code></li> <li>add awk-like built-in variable of record number (<code>{NR}</code>) for <code>seqkit replace</code></li> </ul> </li> <li>seqkit v0.2.4.1 - 2016-06-12   <ul> <li>fix several bugs from library <code>bio</code>, affected situations:<ul> <li>Locating patterns in sequences by pattern FASTA file: <code>seqkit locate -f</code></li> <li>Reading FASTQ file with record of which the quality starts with <code>+</code></li> </ul> </li> <li>add command <code>version</code></li> </ul> </li> <li>seqkit v0.2.4 - 2016-05-31   <ul> <li>add subcommand <code>head</code></li> </ul> </li> <li>seqkit v0.2.3 - 2016-05-08   <ul> <li>reduce memory occupation by avoid copy data when convert <code>string</code> to <code>[]byte</code></li> <li>speedup reverse-complement by avoid repeatly calling functions</li> </ul> </li> <li>seqkit v0.2.2 - 2016-05-06   <ul> <li>reduce memory occupation of subcommands that use FASTA index</li> </ul> </li> <li>seqkit v0.2.1 - 2016-05-02   <ul> <li>improve performance of outputing.</li> <li>fix bug of <code>seqkit seq -g</code> for FASTA fromat</li> <li>some other minor fix of code and docs</li> <li>update benchmark results</li> </ul> </li> <li>seqkit v0.2.0 - 2016-04-29   <ul> <li>reduce memory usage of writing output</li> <li>fix bug of <code>subseq</code>, <code>shuffle</code>, <code>sort</code> when reading from stdin</li> <li>reduce memory usage of <code>faidx</code></li> <li>make validating sequences an optional option in <code>seq</code> command, it saves some time.</li> </ul> </li> <li>seqkit v0.1.9 - 2016-04-26   <ul> <li>using custom FASTA index file extension: <code>.seqkit.fai</code></li> <li>reducing memory usage of <code>sample --number --two-pass</code></li> <li>change default CPU number to 2 for multi-cpus computer, and 1 for single-CPU computer</li> </ul> </li> <li>seqkit v0.1.8 - 2016-04-24   <ul> <li>add subcommand <code>rename</code> to rename duplicated IDs</li> <li>add subcommand <code>faidx</code> to create FASTA index file</li> <li>utilize faidx to improve performance of <code>subseq</code></li> <li><code>shuffle</code>, <code>sort</code> and split support two-pass mode (by flag <code>-2</code>) with faidx to reduce memory usage.</li> <li>document update</li> </ul> </li> <li>seqkit v0.1.7 - 2016-04-21   <ul> <li>add support for (multi-line) FASTQ format</li> <li>update document, add technical details</li> <li>rename subcommands <code>fa2tab</code> and <code>tab2fa</code> to <code>fx2tab</code> and <code>tab2fx</code></li> <li>add subcommand <code>fq2fa</code></li> <li>add column \"seq_format\" to <code>stat</code></li> <li>add global flag <code>-b</code> (<code>--bufer-size</code>)</li> <li>little change of flag in <code>subseq</code> and some other commands</li> </ul> </li> <li>seqkit v0.1.6 - 2016-04-07   <ul> <li>add subcommand <code>replace</code></li> </ul> </li> <li>seqkit v0.1.5.2 - 2016-04-06   <ul> <li>fix bug of <code>grep</code>, when not using flag <code>-r</code>, flag <code>-i</code> will not take effect.</li> </ul> </li> <li>seqkit v0.1.5.1 <ul> <li>fix result of <code>seqkit sample -n</code></li> <li>fix benchmark script</li> </ul> </li> <li>seqkit v0.1.5 - 2016-03-29   <ul> <li>add global flag <code>--id-ncbi</code></li> <li>add flag <code>-d</code> (<code>--dup-seqs-file</code>) and <code>-D</code> (<code>--dup-num-file</code>) for subcommand <code>rmdup</code></li> <li>make using MD5 as an optional flag <code>-m</code> (<code>--md5</code>) in subcommand <code>rmdup</code> and <code>common</code></li> <li>fix file name suffix of <code>seqkit split</code> result</li> <li>minor modification of <code>sliding</code> output</li> </ul> </li> <li>seqkit v0.1.4.1 - 2016-03-27   <ul> <li>change alignment of <code>stat</code> output</li> <li>preciser CPUs number control</li> </ul> </li> <li>seqkit v0.1.4 - 2016-03-25   <ul> <li>add subcommand <code>sort</code></li> <li>improve subcommand <code>subseq</code>: supporting of getting subsequences by GTF and BED files</li> <li>change name format of <code>sliding</code> result</li> <li>prettier output of <code>stat</code></li> </ul> </li> <li>seqkit v0.1.3.1 - 2016-03-16   <ul> <li>Performance improvement by reducing time of cleaning spaces</li> <li>Document update</li> </ul> </li> <li>seqkit v0.1.3 - 2016-03-15   <ul> <li>Further performance improvement</li> <li>Rename sub command <code>extract</code> to <code>grep</code></li> <li>Change default value of flag <code>--threads</code> back CPU number of current device,   change default value of flag <code>--chunk-size</code> back 10000 sequences.</li> <li>Update benchmark</li> </ul> </li> <li>seqkit v0.1.2 - 2016-03-14   <ul> <li>Add flag <code>--dna2rna</code> and <code>--rna2dna</code> to subcommand <code>seq</code>.</li> </ul> </li> <li>seqkit v0.1.1 - 2016-03-13   <ul> <li>5.5X speedup of FASTA file parsing by avoid using regular expression to remove spaces (detail ) and using slice indexing instead of map to validate letters (detail)</li> <li>Change default value of global flag <code>-- thread</code> to 1. Since most of the subcommands are I/O intensive,  For computation intensive jobs, like extract and locate, you may set a bigger value.</li> <li>Change default value of global flag <code>--chunk-size</code> to 100.</li> <li>Add subcommand <code>stat</code></li> <li>Fix bug of failing to automatically detect alphabet when only one record in file.</li> </ul> </li> <li>seqkit v0.1 - 2016-03-11   <ul> <li>first release of seqkit</li> </ul> </li> </ul> Please enable JavaScript to view the comments powered by Disqus."},{"location":"faq/","title":"FAQs on FASTA/Q manipulations","text":"<p>This page was originally written as the section FASTA/Q manipulations of The Biostar Handbook: A Beginner's Guide to Bioinformatics (discussion on Biostars.org).</p> <p>This page illustrates common FASTA/Q manipulations using SeqKit. Some other utilities, including csvtk (CSV/TSV toolkit) and shell commands were also used.</p> <p>Note: SeqKit seamlessly support FASTA and FASTQ formats both in their original form or in stored in gzipped compressed format. We list FASTA or FASTQ depending on the more common usage but you can always use it on the other type as well.</p> <ul> <li>Example data</li> <li>How to produce an overview of FASTQ files?</li> <li>How to get GC content of every sequence in FASTA/Q file?</li> <li>How to extract sequences subset from FASTA/Q file with name/ID list file?</li> <li>How to find FASTA/Q sequences containing degenerate bases and locate them?</li> <li>How to remove duplicated FASTA/Q records with same sequences?</li> <li>How to locate motif/subsequence/enzyme digest site in FASTA/Q sequence?</li> <li>How to sort huge number of FASTA sequences by length?</li> <li>How to split FASTA sequences according to information in header?</li> <li>How to search and replace FASTA header with known character strings from a text file?</li> <li>How to extract paired reads from two paired-end reads file?</li> <li>How to concatenate two FASTA sequences in to one?</li> </ul>"},{"location":"faq/#example-data","title":"Example data","text":"<p>One FASTQ file (sample reads, 1M) and two FASTA files (Virus DNA and protein sequences from NCBI RefSeq database, 60+40M) are used.</p> <pre><code>wget http://data.biostarhandbook.com/reads/duplicated-reads.fq.gz\nwget ftp://ftp.ncbi.nih.gov/refseq/release/viral/viral.1.1.genomic.fna.gz\nwget ftp://ftp.ncbi.nih.gov/refseq/release/viral/viral.1.protein.faa.gz\n</code></pre>"},{"location":"faq/#how-to-produce-an-overview-of-fastq-files","title":"How to produce an overview of FASTQ files?","text":"<p>Sequence format and type are automatically detected.</p> <pre><code>$ seqkit stat *.gz\nfile                      format  type     num_seqs      sum_len  min_len   avg_len    max_len\nduplicated-reads.fq.gz    FASTQ   DNA        15,000    1,515,000      101       101        101\nviral.1.1.genomic.fna.gz  FASTA   DNA         7,048  203,325,120      200  28,848.6  2,473,870\nviral.1.protein.faa.gz    FASTA   Protein   252,611   62,024,702        5     245.5      8,960\n</code></pre>"},{"location":"faq/#how-to-get-gc-content-of-every-sequence-in-fastaq-file","title":"How to get GC content of every sequence in FASTA/Q file?","text":"<p><code>seqkit fx2tab</code> converts FASTA/Q to 3-column tabular format (1th: name/ID, 2nd: sequence, 3rd: quality), and can also provides various information in new columns, including sequence length, GC content/GC skew, alphabet.</p> <p>GC content:</p> <pre><code>$ seqkit fx2tab --name --only-id --gc viral*.fna.gz\ngi|526245010|ref|NC_021865.1|                   40.94\ngi|460042095|ref|NC_020479.1|                   41.82\ngi|526244636|ref|NC_021857.1|                   49.17\n</code></pre> <p>Custom bases (A, C and A+C):</p> <pre><code>$ seqkit fx2tab -H -n -i -B a -B c -B ac viral.1.1.genomic.fna.gz\n#name   seq     qual    a       c       ac\ngi|526245010|ref|NC_021865.1|                   33.20   18.24   51.44\ngi|460042095|ref|NC_020479.1|                   32.57   19.63   52.20\ngi|526244636|ref|NC_021857.1|                   25.52   23.06   48.59\n</code></pre>"},{"location":"faq/#how-to-extract-sequences-subset-from-fastaq-file-with-nameid-list-file","title":"How to extract sequences subset from FASTA/Q file with name/ID list file?","text":"<p>This is a frequently used manipulation. Let's create a sample ID list file, which may also come from other way like mapping result.</p> <pre><code>$ seqkit sample --proportion 0.001  duplicated-reads.fq.gz \\\n    | seqkit seq --name --only-id &gt; id.txt\n</code></pre> <p>ID list file:</p> <pre><code>$ head id.txt\nSRR1972739.2996\nSRR1972739.3044\nSRR1972739.3562\n</code></pre> <p>Searching by ID list file:</p> <pre><code>$ seqkit grep --pattern-file id.txt duplicated-reads.fq.gz \\\n    &gt; duplicated-reads.subset.fq.gz\n</code></pre>"},{"location":"faq/#how-to-find-fastaq-sequences-containing-degenerate-bases-and-locate-them","title":"How to find FASTA/Q sequences containing degenerate bases and locate them?","text":"<p><code>seqkit fx2tab</code> converts FASTA/Q to tabular format and can output the sequence alphabet in a new column. And then text searching tools can be used to filter the table.</p> <pre><code>$ seqkit fx2tab -n -i -a viral.1.1.genomic.fna.gz \\\n    | csvtk -H -t grep -f 4 -r -i -p \"[^ACGT]\"\ngi|446730228|ref|NC_019782.1|                   ACGNT\ngi|557940284|ref|NC_022800.1|                   ACGKT\ngi|564292828|ref|NC_023009.1|                   ACGNT\n</code></pre> <p>Long-option version of the command:</p> <pre><code>$ seqkit fx2tab --name --only-id --alphabet  viral.1.1.genomic.fna.gz \\\n    | csvtk --no-header-row --tabs grep --fields 4 --use-regexp --ignore-case --pattern \"[^ACGT]\"\n</code></pre> <p>You can then exclude these sequences with <code>seqkit grep</code>:</p> <pre><code># save the sequenece IDs.\n$ seqkit fx2tab -n -i -a viral.1.1.genomic.fna.gz \\\n    | csvtk -H -t grep -f 4 -r -i -p \"[^ACGT]\" | csvtk -H -t cut -f 1 &gt; id2.txt\n\n# search and exclude.\n$ seqkit grep --pattern-file id2.txt --invert-match viral.1.1.genomic.fna.gz &gt; clean.fa\n</code></pre> <p>Or locate the degenerate bases, e.g, <code>N</code> and <code>K</code></p> <pre><code>$ seqkit grep --pattern-file id2.txt viral.1.1.genomic.fna.gz \\\n    | seqkit locate --ignore-case --only-positive-strand --pattern K+ --pattern N+\nseqID   patternName     pattern strand  start   end     matched\ngi|564292828|ref|NC_023009.1|   N+      N+      +       87972   87972   N\ngi|564292828|ref|NC_023009.1|   N+      N+      +       100983  100983  N\ngi|557307918|ref|NC_022755.1|   K+      K+      +       1788    1788    K\ngi|557307918|ref|NC_022755.1|   K+      K+      +       4044    4044    K\ngi|589287065|ref|NC_023585.1|   K+      K+      +       28296   28296   K\ngi|590911929|ref|NC_023639.1|   N+      N+      +       741654  741753  NNNNNNNNNNNNNNNNNNNNNNNNNNN\n</code></pre>"},{"location":"faq/#how-to-remove-duplicated-fastaq-records-with-same-sequences","title":"How to remove duplicated FASTA/Q records with same sequences?","text":"<pre><code>$ seqkit rmdup --by-seq --ignore-case duplicated-reads.fq.gz &gt; duplicated-reads.uniq.fq.gz\n</code></pre> <p>If the FASTA/Q file is very large, please switch on flag <code>-m/--md5</code>, which use MD5 instead of original seqs to reduce memory usage when comparing by sequences.</p> <p>You can also deduplicate according to sequence ID (default) or full name (<code>--by-name</code>).</p>"},{"location":"faq/#how-to-locate-motifsubsequenceenzyme-digest-site-in-fastaq-sequence","title":"How to locate motif/subsequence/enzyme digest site in FASTA/Q sequence?","text":"<p>Related posts: Question: Count and location of strings in fastq file reads, Question: Finding TATAWAA in sequence .</p> <p>Assuming a list of motifs (enzyme digest sites) in FASTA format to be located:</p> <pre><code>$ cat enzymes.fa\n&gt;EcoRI\nGAATTC\n&gt;MmeI\nTCCRAC\n&gt;SacI\nGAGCTC\n&gt;XcmI\nCCANNNNNNNNNTGG\n</code></pre> <p>Flag <code>--degenerate</code> is on because patterns contain degenerate bases. Command:</p> <pre><code>$ seqkit locate --degenerate --ignore-case --pattern-file enzymes.fa viral.1.1.genomic.fna.gz\n</code></pre> <p>Sample output (simplified and reformated by <code>csvtk -t uniq -f 3 | csvtk -t pretty</code>)</p> <pre><code>seqID                           patternName   pattern           strand   start   end     matched\ngi|526245010|ref|NC_021865.1|   MmeI          TCCRAC            +        1816    1821    TCCGAC\ngi|526245010|ref|NC_021865.1|   SacI          GAGCTC            +        19506   19511   GAGCTC\ngi|526245010|ref|NC_021865.1|   XcmI          CCANNNNNNNNNTGG   +        2221    2235    CCATATTTAGTGTGG\n</code></pre>"},{"location":"faq/#how-to-sort-huge-number-of-fasta-sequences-by-length","title":"How to sort huge number of FASTA sequences by length?","text":"<p>Sorting FASTA file in order of sequence size (small to large).</p> <pre><code>$ seqkit sort --by-length viral.1.1.genomic.fna.gz &gt; viral.1.1.genomic.sorted.fa\n</code></pre> <p>If the files are too big, use flag <code>--two-pass</code> which consumes lesser memory.</p> <pre><code>$ seqkit sort --by-length --two-pass viral.1.1.genomic.fna.gz &gt; viral.1.1.genomic.sorted.fa\n</code></pre> <p>You can also sort by sequence ID (default), full header (<code>--by-name</code>) or sequence content (<code>--by-seq</code>).</p>"},{"location":"faq/#how-to-split-fasta-sequences-according-to-information-in-header","title":"How to split FASTA sequences according to information in header?","text":"<p>Related posts: Question: extract same all similar sequences in FASTA based on the header .</p> <p>For example, FASTA header line of <code>viral.1.protein.faa.gz</code> contain species name in square brackets.</p> <p>Overview of FASTA Headers:</p> <pre><code>$ seqkit head -n 3 viral.1.protein.faa.gz | seqkit seq --name\ngi|526245011|ref|YP_008320337.1| terminase small subunit [Paenibacillus phage phiIBB_Pl23]\ngi|526245012|ref|YP_008320338.1| terminase large subunit [Paenibacillus phage phiIBB_Pl23]\ngi|526245013|ref|YP_008320339.1| portal protein [Paenibacillus phage phiIBB_Pl23]\n</code></pre> <p><code>seqkit split</code> can split FASTA/Q files according to ID, number of parts, size of every parts, and sequence region. In this case, we'll split according to sequence ID (species names) which can be specified by flag <code>--id-regexp</code>.</p> <p>Default ID:</p> <pre><code>$ seqkit head -n 3 viral.1.protein.faa.gz \\\n    | seqkit seq --name --only-id\ngi|526245011|ref|YP_008320337.1|\ngi|526245012|ref|YP_008320338.1|\ngi|526245013|ref|YP_008320339.1|\n</code></pre> <p>New ID:</p> <pre><code>$ seqkit head -n 3 viral.1.protein.faa.gz \\\n    | seqkit seq --name --only-id --id-regexp \"\\[(.+)\\]\"\nPaenibacillus phage phiIBB_Pl23\nPaenibacillus phage phiIBB_Pl23\nPaenibacillus phage phiIBB_Pl23\n</code></pre> <p>Split:</p> <pre><code>$ seqkit split --by-id --id-regexp \"\\[(.+)\\]\" viral.1.protein.faa.gz\n</code></pre>"},{"location":"faq/#how-to-search-and-replace-fasta-header-with-known-character-strings-from-a-text-file","title":"How to search and replace FASTA header with known character strings from a text file?","text":"<p>Related posts: Question: Replace names in FASTA file with a known character string from a text file, Question: Fasta header, search and replace...? .</p> <p><code>seqKit replace</code> can find substrings in FASTA/Q header with regular expression and replace them with strings or corresponding values of found substrings provided by the tab-delimited key-value file.</p> <p>For example, to unify names of protein with unknown functions, we want to rename \"hypothetical\" to \"putative\" in lower case. The replacing rules are listed below in tab-delimited file:</p> <pre><code>$ cat changes.tsv\nHypothetical    putative\nhypothetical    putative\nPutative        putative\n</code></pre> <p>Overview the FASTA headers containing \"hypothetical\":</p> <pre><code>$ seqkit grep --by-name --use-regexp --ignore-case --pattern hypothetical viral.1.protein.faa.gz \\\n    | seqkit head -n 3 | seqkit seq --name\ngi|526245016|ref|YP_008320342.1| hypothetical protein IBBPl23_06 [Paenibacillus phage phiIBB_Pl23]\ngi|526245019|ref|YP_008320345.1| hypothetical protein IBBPl23_09 [Paenibacillus phage phiIBB_Pl23]\ngi|526245020|ref|YP_008320346.1| hypothetical protein IBBPl23_10 [Paenibacillus phage phiIBB_Pl23]\n</code></pre> <p>A regular expression, <code>^([^ ]+ )(\\w+)</code>, was used to specify the key to be replaced, which is the first word after sequence ID in this case. Note that we also capture the ID (<code>^([^ ]+ )</code>) so we can restore it in \"replacement\" with capture variable  <code>${1}</code> (robuster than <code>$1</code>). And flag <code>-I/--key-capt-idx</code> (default: 1) is set to 2 because the key <code>(\\w+)</code> is the second captured match. Command:</p> <pre><code>$ seqkit replace --kv-file changes.tsv --pattern \"^([^ ]+ )(\\w+) \" \\\n    --replacement \"\\${1}{kv} \" --key-capt-idx 2 --keep-key viral.1.protein.faa.gz &gt; renamed.fa\n</code></pre>"},{"location":"faq/#how-to-extract-paired-reads-from-two-paired-end-reads-file","title":"How to extract paired reads from two paired-end reads file?","text":"<p>Use seqkit pair.</p>"},{"location":"faq/#how-to-concatenate-two-fasta-sequences-in-to-one","title":"How to concatenate two FASTA sequences in to one?","text":"<p>Related posts: Combining two fasta sequences into one</p> <p>Data (not in same order):</p> <pre><code>$ cat 1.fa\n&gt;seq1\naaaaa\n&gt;seq2\nccccc\n&gt;seq3\nggggg\n\n$ cat 2.fa\n&gt;seq3\nTTTTT\n&gt;seq2\nGGGGG\n&gt;seq1\nCCCCC\n</code></pre> <p>Just one command:</p> <pre><code>$ seqkit concat 1.fa 2.fa\n&gt;seq1\naaaaaCCCCC\n&gt;seq2\ncccccGGGGG\n&gt;seq3\ngggggTTTTT\n</code></pre>"},{"location":"note/","title":"Notes","text":""},{"location":"note/#sequence-parsing-strategies","title":"Sequence Parsing Strategies","text":"<p>In general, most FASTA/Q file processing softwares read one sequence record and process it immediately, and then read the next one (Fig 1A). The early version of SeqKit adopted this strategy too; however, our sequence parsing algorithm was not efficient. Therefore, we tried another strategy by using a separated thread to non-blocking read and parse sequences (Fig 1B): The sequence parsing thread parsed fixed number of sequences and packaged them into Chunks, then one or more chunks were buffered in memory so they could be paralleled processed by one or more processing threads. This strategy saved time by reducing the waiting time between parsing sequences and by paralleled processing of sequences in some cases. However, this strategy consumes more memories especially for long sequences like human genome and it is hard to find an optimum combination of buffer and chunk size for different scales of sequences. </p> <p>During the major revision, we kept on optimizing the sequence parsing algorithm and several optimizations had been made (Seqkit v0.2.7). The most important one is using custom buffered file reading algorithm instead of using standard library \u201cbufio\u201d which is not efficient in memory for large sequences. This increased the speed and lowered memory usage significantly. Note that the \u201cbuffer\u201d here is different from the previously mentioned one in Fig 1B, this \u201cbuffer\u201d evolved in the low level processing of file reading. At last, the FASTA/Q format parsing algorithm was significant improved by factor of about 3\u00d7 in speed and the memory usage was only 1/2 compared to SeqKit version 0.2.4 in the first manuscript (this was not descripted in the manuscript). Therefore, we discarded the strategy of \u201cbuffer and chunk\u201d and adopt the serially reading strategy again (Fig 1C). The only difference between Fig C and Fig A is that the time of parsing one sequence is much shorter.</p> <p></p> <p>Figure 1 Illustration of FASTA/Q file parsing strategies. (A) and (C) Main thread parses one sequence, waits (blocked) it to be processed and then parses next one. (B) Sequence parsing thread continuously (non-blocked) parses sequences and passes them to main thread. The width of rectangles representing sequence parsing and sequence processing is proportional with running time. Sequence parsing speeds in (A) and (B) are the same, which are both much slower than that in (C). The speeds of sequence processing are identical in (A), (B) and (C). In (B), chunks of sequences in buffer can be processed in parallel, but most of the time the main thread needs to serially manipulate the sequences.</p>"},{"location":"note/#effect-of-random-seed-on-results-of-seqkit-sample","title":"Effect of random seed on results of <code>seqkit sample</code>","text":"<p><code>seqkit sample</code> supports FASTA/Q sampling by proportion or amount.</p> <ul> <li>For sampling by proportion (<code>P</code>), SeqKit returns a record if a random number (<code>[0, 1</code>])   is less than <code>P</code>.</li> <li>For sampling by amount (<code>N</code>), Seqkit firstly gets the total amount of records,   and compute the proportion (<code>P</code>), and sampling by proportion (<code>P</code>).   Cause the generated random number is pseudorandom   and affected by the random seed (<code>-s/--rand-seed</code>), the number of sampled records   may not be equal to <code>N</code>.</li> </ul> <p>Here we evaluate the effect of random seed on <code>seqkit sample</code> results. Used softwares</p> <ul> <li>csvtk: https://github.com/shenwei356/csvtk</li> <li>rush: https://github.com/shenwei356/rush</li> </ul>"},{"location":"note/#amount-distribution-of-sampled-records","title":"Amount distribution of sampled records","text":"<p>1000 FASTA records with IDs of <code>1</code>, <code>2</code>, ..., <code>1000</code> were sampled by proportion of <code>0.1</code> with random seeds from <code>1</code> to <code>1000</code>, and the distribution of number of sampled records was plotted in boxplot.</p> <pre><code>seq 1000 \\\n    | rush 'seq 1000 | csvtk -t -H mutate | seqkit tab2fx \\\n        | seqkit sample -p 0.1 -s {} \\\n        | seqkit fx2tab | wc -l' \\\n    &gt; ns.txt\n\ncat ns.txt | csvtk -H -t plot box -f 1 --horiz --height 2 \\\n    --xlab \"# of sampled records\" \\\n    &gt; ns.png\n</code></pre> <p></p>"},{"location":"note/#location-distribution-of-sampled-records","title":"Location distribution of sampled records","text":"<p>1000 FASTA records with IDs of <code>1</code>, <code>2</code>, ..., <code>1000</code> were sampled by proportion of <code>0.1</code> with random seeds from <code>1</code> to <code>10</code>. The record IDs (<code>x axis</code>) was used to plot a scatter plot, which show the location distribution of sampled records. The ideal distribution would produces a straight line in the plot.</p> <pre><code>seq 10 \\\n    | rush 'seq 1000 | csvtk -t -H mutate | seqkit tab2fx \\\n        | seqkit sample -p 0.1 -s {} \\\n        | seqkit fx2tab | csvtk -H -t mutate \\\n        | csvtk -H -t replace -f 2 -p '.+' -r '{nr}' \\\n        | csvtk -H -t replace -f 4 -p '.+' -r {}' \\\n    | csvtk -H -t plot line -x 1 -y 2 -g 4 \\\n        --xlab \"sampled locations\" --ylab \"new ID of sampled records\" \\\n        --title \"location distribution of sampled records\" \\\n        --width 8 --height 6 --point-size 2 \\\n    &gt; pos.png\n</code></pre> <p></p> Please enable JavaScript to view the comments powered by Disqus."},{"location":"tutorial/","title":"Tutorial","text":""},{"location":"tutorial/#real-time-monitoring-mapping-results-with-seqkit-bam-in-time-critical-situations","title":"Real-time monitoring mapping results with seqkit bam in time-critical situations","text":"<p>Some of SeqKit subcommands, including <code>watch</code>, <code>fish</code>, <code>scat</code>, <code>bam</code>, aid the real-time, streaming processing of data in FASTQ/FASTA and BAM formats, enabling the development of analysis pipelines in time-critical situations. To illustrate the possibilities opened up by the these features, we present a small pipeline which maps a dataset of SARS-CoV-2 amplicon reads to the reference sequence using <code>minimap2</code> as they are being downloaded. The stream of SAM alignments is then converted into a stream of BAM records and passed to <code>seqkit bam</code>, which filters for records with aligned reference lengths between <code>310</code> and <code>410</code> and displays the distribution of aligned reference lengths after every <code>5000</code> records also saving it to a PDF file. The next element of the pipeline is another instance of the <code>seqkit bam</code> subcommand, which displays the distribution of alignment accuracies after every 10000 records. The stream of BAM records is then passed on to samtools sort to produce the final sorted BAM file. Finally, <code>seqkit bam</code> is used to display detailed alignment statistics from the final sorted BAM file in a pretty format.</p> <pre><code>#!/bin/bash\nset -o nounset\n\n# Define reference and data URL:\nREF_URL=\"https://www.ncbi.nlm.nih.gov/sviewer/viewer.cgi?tool=portal&amp;save=file&amp;log$=seqview&amp;db=nuccore&amp;report=fasta&amp;id=1798174254&amp;extrafeat=null&amp;conwithfeat=on&amp;hide-cdd=on&amp;ncbi_phid=CE8B108356DDCF110000000005B10489\"\nDATA_URL=\"http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR145/055/SRR14560555/SRR14560555_1.fastq.gz\"\n\n# Map SARS-CoV-2 amplicon reads to reference followed by these steps:\n# - keep only primary reads with mapping quality greater than 1.\n# - Keep only alignments which align to a reference segments with lengths between 310 and 420.\n#   Show an approximate histogram of aligned reference lengths after every 5000 records.\n# - Show an approximate histogram of alignment accuracy after every 10000 records.\n# - Pipe the BAM into samtools for sorting.\nminimap2 -K 10M -t 8 -ax map-ont \\\n        &lt;(wget -q -O - \"$REF_URL\") \\\n        &lt;(wget -q -O - \"$DATA_URL\") 2&gt;/dev/null \\\n    | samtools view -F 2304 -q 1 -b - \\\n    | seqkit bam -O aln_len.pdf -f RefAln -m 310 -M 410 -p 5000 -x - \\\n    | seqkit bam -O aln_acc.pdf -f Acc  -p 10000 -x - \\\n    | samtools sort -o sars_artic_sorted.bam -\n\n# Show statistics from the sorted BAM:\nseqkit bam -s -k sars_artic_sorted.bam\n</code></pre>"},{"location":"tutorial/#removing-duplicated-and-nested-sequences","title":"Removing duplicated and nested sequences","text":"<p>https://twitter.com/bentemperton/status/1673999868933599232</p> <p>sample data</p> <pre><code>$ cat contigs.fa\n&gt;big\nACTGACGATCGATACGCAGCACAGCAG\n&gt;small_in_big_rc\nTGCTGCGTATCG\n&gt;small2\nACTACGACTACGACT\n&gt;small2_alias\nACTACGACTACGACT\n&gt;small2_rc\nAGTCGTAGTCGTAGT\n&gt;another\nACTAACGA\n\n$ seqkit fx2tab -Q contigs.fa  | csvtk pretty -Ht -W 40 --clip\nbig               ACTGACGATCGATACGCAGCACAGCAG\nsmall_in_big_rc   TGCTGCGTATCG\nsmall2            ACTACGACTACGACT\nsmall2_alias      ACTACGACTACGACT\nsmall2_rc         AGTCGTAGTCGTAGT\nanother           ACTAACGA\n</code></pre> <p>Step 1. remove exactly duplicated sequences.</p> <pre><code>$ seqkit rmdup -s -i contigs.fa -o contigs.uniq1.fa\n[INFO] 2 duplicated records removed\n</code></pre> <p>Step 2. remove nested seqs.</p> <pre><code># pair-wise exactly searching\nseqkit locate -M -f contigs.uniq1.fa contigs.uniq1.fa -o match.tsv\n\n$ csvtk pretty  -W 40 --clip -t match.tsv\nseqID             patternName       pattern                       strand   start   end\n---------------   ---------------   ---------------------------   ------   -----   ---\nbig               small_in_big_rc   TGCTGCGTATCG                  -        10      21\nbig               big               ACTGACGATCGATACGCAGCACAGCAG   +        1       27\nsmall_in_big_rc   small_in_big_rc   TGCTGCGTATCG                  +        1       12\nsmall2            small2            ACTACGACTACGACT               +        1       15\nanother           another           ACTAACGA                      +        1       8\n\n# IDs of embeded/nested sequences\n$ sed 1d match.tsv \\\n    | awk '$2 != $1' \\\n    | cut -f 2 \\\n    | tee nested.txt\nsmall_in_big_rc\n\n# remove nested sequences\n$ seqkit grep -v -f nested.txt contigs.uniq1.fa \\\n    -o contigs.uniq2.fa\n</code></pre> <p>Result</p> <pre><code>$ seqkit fx2tab -Q contigs.uniq2.fa | csvtk pretty -Ht -W 40 --clip\nbig       ACTGACGATCGATACGCAGCACAGCAG\nsmall2    ACTACGACTACGACT\nanother   ACTAACGA\n</code></pre>"},{"location":"tutorial/#some-manipulations-on-big-genomes","title":"Some manipulations on big genomes","text":"<p>A script memusg is used to check the peek memory usage of seqkit. Usage: <code>memusg [-t] command</code>.</p> <ol> <li> <p>Human genome</p> <pre><code>$ seqkit stat hsa.fa\nfile    format  type  num_seqs        sum_len  min_len       avg_len      max_len\nhsa.fa  FASTA   DNA        194  3,099,750,718      970  15,978,096.5  248,956,422\n</code></pre> </li> <li> <p>Build FASTA index (optional, when using flag <code>-2</code> (<code>--two-pass</code>),    some commands will automaticlly build it).    For some commands, including <code>subseq</code>, <code>split</code>, <code>sort</code> and <code>shuffle</code>,    when input files are (plain or gzipped) FASTA files or stdin,    FASTA index would be optional used for    rapid acccess of sequences and reducing memory occupation.    ATTENTION: the <code>.seqkit.fai</code> file created by SeqKit is a little different from .fai file    created by samtools. SeqKit uses full sequence head instead of just ID as key.</p> <pre><code>$ memusg -t seqkit faidx --id-regexp \"^(.+)$\"  hsa.fa -o hsa.fa.seqkit.fai\n\nelapsed time: 10.011s\npeak rss: 177.21 MB\n</code></pre> <p>Create common .fai file:</p> <pre><code>$ memusg -t seqkit faidx hsa.fa -o hsa.fa.fai2\n\nelapsed time: 10.454s\npeak rss: 172.82 MB\n</code></pre> <p>Performance of samtools:</p> <pre><code>$ memusg -t samtools faidx hsa.fa\n\nelapsed time: 9.574s\npeak rss: 1.45 MB\n</code></pre> <p>Exactly same content:</p> <pre><code>$ md5sum hsa.fa.fai*\n21e0c25b4d817d1c19ee8107191b9b31  hsa.fa.fai\n21e0c25b4d817d1c19ee8107191b9b31  hsa.fa.fai2\n</code></pre> </li> <li> <p>Sorting by sequence length</p> <pre><code>$ memusg -t seqkit sort --by-length --reverse --two-pass hsa.fa &gt; hsa.sorted.fa\n[INFO] create and read FASTA index ...\n[INFO] read sequence IDs and lengths from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] sorting ...\n[INFO] output ...\n\nelapsed time: 4.892s\npeak rss: 500.15 MB\n</code></pre> <p>Detail:</p> <pre><code>$ seqkit fx2tab --length hsa.sorted.fa --name --only-id | cut -f 1,4 | more\n1       248956422\n2       242193529\n3       198295559\n4       190214555\n5       181538259\n6       170805979\n7       159345973\nX       156040895\n8       145138636\n9       138394717\n11      135086622\n10      133797422\n12      133275309\n13      114364328\n14      107043718\n15      101991189\n16      90338345\n17      83257441\n18      80373285\n20      64444167\n19      58617616\nY       57227415\n22      50818468\n21      46709983\nKI270728.1      1872759\nKI270727.1      448248\n...\n\nreal    0m10.697s\nuser    0m11.153s\nsys     0m0.917s\n</code></pre> </li> <li> <p>Shuffling sequences</p> <pre><code>$ memusg -t seqkit shuffle hsa.fa --two-pass &gt; hsa.shuffled.fa\n[INFO] create and read FASTA index ...\n[INFO] read sequence IDs from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] shuffle ...\n[INFO] output ...\n\nelapsed time: 6.632s\npeak rss: 528.3 MB\n</code></pre> </li> <li> <p>Spliting into files with single sequence</p> <pre><code>$ memusg -t seqkit split --by-id hsa.fa --two-pass\n[INFO] split by ID. idRegexp: ^([^\\s]+)\\s?\n[INFO] create and read FASTA index ...\n[INFO] read sequence IDs from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] write 1 sequences to file: hsa.id_KI270743.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270706.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270717.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270718.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270468.1.fa\n...\n\nelapsed time: 18.807s\npeak rss: 1.36 GB\n</code></pre> </li> <li> <p>Geting subsequence of some chromesomes</p> <pre><code>$ memusg -t seqkit subseq -r 1:10 --chr X --chr Y  hsa.fa\n&gt;X_1-10 X dna_sm:chromosome chromosome:GRCh38:X:1:156040895:1 REF\nnnnnnnnnnn\n&gt;Y_1-10 Y dna_sm:chromosome chromosome:GRCh38:Y:2781480:56887902:1 REF\nNNNNNNNNNN\n\nelapsed time: 1.276s\npeak rss: 640.92 MB\n</code></pre> </li> <li> <p>Geting CDS sequence of chr 1 by GTF files</p> <pre><code>$ memusg -t seqkit subseq --gtf Homo_sapiens.GRCh38.84.gtf.gz --chr X --feature cds  hsa.fa &gt; chrX.gtf.cds.fa\n[INFO] read GTF file ...\n[INFO] 22420 GTF features loaded\n\nelapsed time: 8.643s\npeak rss: 846.14 MB\n</code></pre> </li> </ol>"},{"location":"tutorial/#remove-contaminated-reads","title":"Remove contaminated reads","text":"<ol> <li> <p>Mapping with reads on some potential contaminate genomes, and get the reads IDs list.</p> <pre><code>$ wc -l contaminate.list\n244 contaminate.list\n\n$ head -n 2 contaminate.list\nHWI-D00523:240:HF3WGBCXX:1:1101:2574:2226\nHWI-D00523:240:HF3WGBCXX:1:1101:12616:2205\n</code></pre> </li> <li> <p>Remove contaminated reads</p> <pre><code>$ seqkit grep -f contaminate.list -v reads_1.fq.gz -o reads_1.clean.fq.gz\n$ seqkit grep -f contaminate.list -v reads_2.fq.gz -o reads_2.clean.fq.gz\n\n$ seqkit stat *.fq.gz\nfile                  seq_format   seq_type   num_seqs   min_len   avg_len   max_len\nreads_1.clean.fq.gz   FASTQ        DNA           2,256       226       227       229\nreads_1.fq.gz         FASTQ        DNA           2,500       226       227       229\nreads_2.clean.fq.gz   FASTQ        DNA           2,256       223       224       225\nreads_2.fq.gz         FASTQ        DNA           2,500       223       224       225\n</code></pre> </li> </ol>"},{"location":"tutorial/#handling-of-aligned-sequences","title":"Handling of aligned sequences","text":"<ol> <li> <p>Some mock sequences (usually they will be much longer)</p> <pre><code>$ cat seqs.fa\n&gt;seq1\nACAACGTCTACTTACGTTGCATCGTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq2\nACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq3\nACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG\n&gt;seq4\nACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> </li> <li> <p>Run multiple sequence alignment (clustalo)</p> <pre><code>clustalo -i seqs.fa -o seqs.msa.fa --force --outfmt fasta --threads=4\n</code></pre> </li> <li> <p>Convert FASTA format to tabular format.</p> <pre><code>$ seqkit fx2tab seqs.msa.fa\nseq1    ACAACGTCTACTTACGTTGCAT----CGTCATGCTGCATTACGTAGTCTGATGATG\nseq2    ---------------ACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\nseq3    ACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG-------------\nseq4    ACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> <p>or</p> <pre><code>$ seqkit fx2tab seqs.msa.fa | cut -f 2\nACAACGTCTACTTACGTTGCAT----CGTCATGCTGCATTACGTAGTCTGATGATG\n---------------ACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\nACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG-------------\nACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> <p>For me, it's useful when 1) manually assembling Sanger sequencing result, 2) designing site specific PCR primers.</p> </li> <li> <p>Remove gaps</p> <pre><code>$ seqkit seq seqs.msa.fa -g\n&gt;seq1\nACAACGTCTACTTACGTTGCATCGTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq2\nACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq3\nACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG\n&gt;seq4\nACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> </li> </ol>"},{"location":"tutorial/#play-with-mirna-hairpins","title":"Play with miRNA hairpins","text":""},{"location":"tutorial/#dataset","title":"Dataset","text":"<p><code>hairpin.fa.gz</code> from The miRBase Sequence Database -- Release 21</p>"},{"location":"tutorial/#quick-glance","title":"Quick glance","text":"<ol> <li> <p>Sequence number</p> <pre><code>$ seqkit stat hairpin.fa.gz\nfile           format  type  num_seqs    sum_len  min_len  avg_len  max_len\nhairpin.fa.gz  FASTA   RNA     28,645  2,949,871       39      103    2,354\n</code></pre> </li> <li> <p>First 10 bases</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit subseq -r 1:10 \\\n    | seqkit sort -s\n    | seqkit seq -s \\\n    | head -n 10\nAAAAAAAAAA\nAAAAAAAAAA\nAAAAAAAAAG\nAAAAAAAAAG\nAAAAAAAAAG\nAAAAAAAAAU\nAAAAAAAAGG\nAAAAAAACAU\nAAAAAAACGA\nAAAAAAAUUA\n</code></pre> <p>hmm, nothing special, non-coding RNA~</p> </li> </ol>"},{"location":"tutorial/#repeated-hairpin-sequences","title":"Repeated hairpin sequences","text":"<p>We may want to check how may identical hairpins among different species there are. <code>seqkit rmdup</code> could remove duplicated sequences by sequence content, and save the replicates to another file (here is <code>duplicated.fa.gz</code>), as well as replicating details (<code>duplicated.detail.txt</code>, 1th column is the repeated number, 2nd column contains sequence IDs seperated by comma).</p> <pre><code>$ seqkit rmdup -s -i hairpin.fa.gz -o clean.fa.gz -d duplicated.fa.gz -D duplicated.detail.txt\n\n$ head -n 5 duplicated.detail.txt\n18      dre-mir-430c-1, dre-mir-430c-2, dre-mir-430c-3, dre-mir-430c-4, dre-mir-430c-5, dre-mir-430c-6, dre-mir-430c-7, dre-mir-430c-8, dre-mir-430c-9, dre-mir-430c-10, dre-mir-430c-11, dre-mir-430c-12, dre-mir-430c-13, dre-mir-430c-14, dre-mir-430c-15, dre-mir-430c-16, dre-mir-430c-17, dre-mir-430c-18\n16      hsa-mir-29b-2, mmu-mir-29b-2, rno-mir-29b-2, ptr-mir-29b-2, ggo-mir-29b-2, ppy-mir-29b-2, sla-mir-29b, mne-mir-29b, ppa-mir-29b-2, bta-mir-29b-2, mml-mir-29b-2, eca-mir-29b-2, aja-mir-29b, oar-mir-29b-1, oar-mir-29b-2, rno-mir-29b-3\n15      dme-mir-125, dps-mir-125, dan-mir-125, der-mir-125, dgr-mir-125-1, dgr-mir-125-2, dmo-mir-125, dpe-mir-125-2, dpe-mir-125-1, dpe-mir-125-3, dse-mir-125, dsi-mir-125, dvi-mir-125, dwi-mir-125, dya-mir-125\n13      hsa-mir-19b-1, ggo-mir-19b-1, age-mir-19b-1, ppa-mir-19b-1, ppy-mir-19b-1, ptr-mir-19b-1, mml-mir-19b-1, sla-mir-19b-1, lla-mir-19b-1, mne-mir-19b-1, bta-mir-19b, oar-mir-19b, chi-mir-19b\n13      hsa-mir-20a, ssc-mir-20a, ggo-mir-20a, age-mir-20, ppa-mir-20, ppy-mir-20a, ptr-mir-20a, mml-mir-20a, sla-mir-20, lla-mir-20, mne-mir-20, bta-mir-20a, eca-mir-20a\n</code></pre> <p>The result shows the most conserved miRNAs among different species, <code>mir-29b</code>, <code>mir-125</code>, <code>mir-19b-1</code> and <code>mir-20a</code>. And the <code>dre-miR-430c</code> has the most multicopies in Danio rerio.</p>"},{"location":"tutorial/#hairpins-in-different-species","title":"Hairpins in different species","text":"<ol> <li> <p>Before spliting by species, let's take a look at the sequence names.</p> <pre><code>$ seqkit seq hairpin.fa.gz -n | head -n 3\ncel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\ncel-lin-4 MI0000002 Caenorhabditis elegans lin-4 stem-loop\ncel-mir-1 MI0000003 Caenorhabditis elegans miR-1 stem-loop\n</code></pre> <p>The first three letters (e.g. <code>cel</code>) are the abbreviation of species names. So we could split hairpins by the first letters by defining custom sequence ID parsing regular expression <code>^([\\w]+)\\-</code>.</p> <p>By default, <code>seqkit</code> takes the first non-space letters as sequence ID. For example,</p> FASTA head ID &gt;123456 gene name 123456 &gt;longname longname &gt;gi|110645304|ref|NC_002516.2| Pseudomona gi|110645304|ref|NC_002516.2| <p>But for some sequences from NCBI, e.g. <code>&gt;gi|110645304|ref|NC_002516.2| Pseudomona</code>, the ID is <code>NC_002516.2</code>. In this case, we could set sequence ID parsing regular expression by flag <code>--id-regexp \"\\|([^\\|]+)\\| \"</code> or just use flag <code>--id-ncbi</code>. If you want the <code>gi</code> number, then use <code>--id-regexp \"^gi\\|([^\\|]+)\\|\"</code>.</p> </li> <li> <p>Split sequences by species. A custom ID parsing regular expression is used, <code>^([\\w]+)\\-</code>.</p> <pre><code>$ seqkit split hairpin.fa.gz -i --id-regexp \"^([\\w]+)\\-\" --two-pass\n</code></pre> <p>To reduce memory usage when splitting big file, we should always use flag <code>--two-pass</code></p> </li> <li> <p>Species with most miRNA hairpins. Third column is the sequences number.</p> <pre><code>$ cd hairpin.fa.gz.split/;\n$ seqkit stat hairpin.id_* \\\n    | csvtk space2tab \\\n    | csvtk -t sort -k num_seqs:nr \\\n    | csvtk -t pretty \\\n    | more\nfile                     format   type   num_seqs   sum_len   min_len   avg_len   max_len\nhairpin.id_hsa.fasta     FASTA    RNA    1,881      154,242   82        82        82\nhairpin.id_mmu.fasta     FASTA    RNA    1,193      107,370   90        90        90\nhairpin.id_bta.fasta     FASTA    RNA    808        61,408    76        76        76\nhairpin.id_gga.fasta     FASTA    RNA    740        42,180    57        57        57\nhairpin.id_eca.fasta     FASTA    RNA    715        89,375    125       125       125\nhairpin.id_mtr.fasta     FASTA    RNA    672        231,840   345       345       345\n</code></pre> <p>Here, a CSV/TSV tool csvtk is used to sort and view the result.</p> </li> </ol> <p>For human miRNA hairpins</p> <ol> <li> <p>Length distribution.  <code>seqkit fx2tab</code> could show extra information like sequence length, GC content.  <code>csvtk</code> is used to plot.</p> <pre><code>$ seqkit grep -r -p '^hsa' hairpin.fa.gz  \\\n    | seqkit fx2tab -l \\\n    | cut -f 4  \\\n    | csvtk -H plot hist --xlab Length --title \"Human pre-miRNA length distribution\"\n</code></pre> <p></p> <pre><code>$ seqkit grep -r -p '^hsa' hairpin.fa.gz \\\n    | seqkit fx2tab -l \\\n    | cut -f 4 \\\n    | csvtk -H plot box --xlab Length --horiz --height 1.5\n</code></pre> <p></p> </li> </ol>"},{"location":"tutorial/#bacteria-genomes","title":"Bacteria genomes","text":""},{"location":"tutorial/#dataset_1","title":"Dataset","text":"<p>Pseudomonas aeruginosa PAO1, files:</p> <ul> <li>Genbank file <code>PAO1.gb</code></li> <li>Genome FASTA file <code>PAO1.fasta</code></li> <li>GTF file <code>PAO1.gtf</code> was created with <code>extract_features_from_genbank_file.py</code>, by<pre><code>extract_features_from_genbank_file.py  PAO1.gb -t . -f gtf &gt; PAO1.gtf\n</code></pre> </li> </ul>"},{"location":"tutorial/#motif-distribution","title":"Motif distribution","text":"<p>Motifs</p> <pre><code>$ cat motifs.fa\n&gt;GTAGCGS\nGTAGCGS\n&gt;GGWGKTCG\nGGWGKTCG\n</code></pre> <ol> <li> <p>Sliding. Remember flag <code>--id-ncbi</code>, do you?   By the way, do not be scared by the long flag <code>--circle-genome</code>, <code>--step</code>   and so on. They have short ones, <code>-c</code>, <code>-s</code></p> <pre><code>$ seqkit sliding --id-ncbi --circular-genome \\\n    --step 20000 --window 200000 PAO1.fasta -o PAO1.fasta.sliding.fa\n\n$ seqkit stat PAO1.fasta.sliding.fa\nfile                   format  type  num_seqs     sum_len  min_len  avg_len  max_len\nPAO1.fasta.sliding.fa  FASTA   DNA        314  62,800,000  200,000  200,000  200,000\n</code></pre> </li> <li> <p>Locating motifs</p> <pre><code>$ seqkit locate --id-ncbi --ignore-case --degenerate \\\n    --pattern-file motifs.fa  PAO1.fasta.sliding.fa -o  PAO1.fasta.sliding.fa.motifs.tsv\n</code></pre> </li> <li> <p>Ploting distribution (plot_motif_distribution.R)</p> <pre><code># preproccess\n$ perl -ne 'if (/_sliding:(\\d+)-(\\d+)\\t(.+)/) {$loc= $1 + 100000; print \"$loc\\t$3\\n\";} else {print}' PAO1.fasta.sliding.fa.motifs.tsv  &gt; PAO1.fasta.sliding.fa.motifs.tsv2\n\n# plot\n$ ./plot_motif_distribution.R\n</code></pre> <p>Result</p> <p></p> </li> </ol>"},{"location":"tutorial/#find-multicopy-genes","title":"Find multicopy genes","text":"<ol> <li> <p>Get all CDS sequences</p> <pre><code>$ seqkit subseq --id-ncbi --gtf PAO1.gtf --feature cds PAO1.fasta -o PAO1.cds.fasta\n\n$ seqkit stat *.fasta\nfile            format  type  num_seqs    sum_len    min_len    avg_len    max_len\nPAO1.cds.fasta  FASTA   DNA      5,572  5,593,306         72    1,003.8     16,884\nPAO1.fasta      FASTA   DNA          1  6,264,404  6,264,404  6,264,404  6,264,404\n</code></pre> </li> <li> <p>Get duplicated sequences</p> <pre><code>$ seqkit rmdup --by-seq --ignore-case PAO1.cds.fasta -o PAO1.cds.uniq.fasta \\\n    --dup-seqs-file PAO1.cds.dup.fasta --dup-num-file PAO1.cds.dup.text\n\n$ cat PAO1.cds.dup.text\n6       NC_002516.2_500104:501120:-, NC_002516.2_2556948:2557964:+, NC_002516.2_3043750:3044766:-, NC_002516.2_3842274:3843290:-, NC_002516.2_4473623:4474639:+, NC_002516.2_5382796:5383812:-\n2       NC_002516.2_2073555:2075438:+, NC_002516.2_4716660:4718543:+\n2       NC_002516.2_2072935:2073558:+, NC_002516.2_4716040:4716663:+\n2       NC_002516.2_2075452:2076288:+, NC_002516.2_4718557:4719393:+\n</code></pre> </li> </ol>"},{"location":"tutorial/#flanking-sequences","title":"Flanking sequences","text":"<ol> <li> <p>Get CDS and 1000 bp upstream sequence</p> <pre><code>$ seqkit subseq --id-ncbi --gtf PAO1.gtf \\\n    --feature cds PAO1.fasta --up-stream 1000\n</code></pre> </li> <li> <p>Get 1000 bp upstream sequence of CDS, NOT including CDS.</p> <pre><code>$ seqkit subseq --id-ncbi --gtf PAO1.gtf \\\n    --feature cds PAO1.fasta --up-stream 1000 --only-flank\n</code></pre> </li> </ol> Please enable JavaScript to view the comments powered by Disqus."},{"location":"usage/","title":"Usage and Examples","text":""},{"location":"usage/#quick-guide","title":"Quick Guide","text":"<ul> <li>Basic: seq, stats, subseq, sliding,   faidx, translate, watch, sana, scat</li> <li>Format conversion: fq2fa, fa2fq, fx2tab, tab2fx,   convert</li> <li>Searching: grep, locate, amplicon, fish</li> <li>Set operation: sample, rmdup, common,   duplicate, split, split2, head,   head-genome, range, pair</li> <li>Edit: concat, replace, restart, mutate,   rename</li> <li>Ordering: sort, shuffle</li> <li>BAM processing: bam</li> <li>Others: sum, merge-slides</li> </ul>"},{"location":"usage/#technical-details-and-guides-for-use","title":"Technical details and guides for use","text":""},{"location":"usage/#fastaq-format-parsing-and-writing","title":"FASTA/Q format parsing and writing","text":"<p>Seqkit also supports reading and writing xz (.xz) and zstd (.zst) formats since v2.2.0. Bzip2 format is supported since v2.4.0.</p> <p>SeqKit uses the author's lightweight and high-performance bioinformatics package bio for FASTA/Q parsing, which has high performance close to the famous C lib klib (kseq.h).</p> <p></p> <p>Notes:</p> <ul> <li><code>seqkit</code> uses 4 threads by default.</li> <li><code>seqkit_t1</code> uses 1 thread.</li> <li><code>seqtk</code> is single-threaded.</li> <li><code>seqtk+gzip</code>: <code>seqtk</code> pipes data to the single-threaded <code>gzip</code>.</li> <li><code>seqtk+pigz</code>: <code>seqtk</code> pipes data to the multithreaded <code>pigz</code> which uses 4 threads here.</li> </ul>"},{"location":"usage/#input-and-output-files","title":"Input and output files","text":"<p>Seqkit accepts input data from standard input (STDIN) and plain or gzip-compressed files. Files can be given via positional arguments or the flag <code>--infile-list</code>. For example:</p> <pre><code>seqkit seq   a.fasta b.fasta\n\nseqkit seq --infile-list file-list.txt\n\nseqkit seq --infile-list &lt;(find -name \"*.fq.gz\" directory)\n</code></pre> <p>Result are printed to standard ouput (STDOUT) by default, you can also specify the output file via the flag <code>-o/--out-file</code>. The file name extension <code>.gz</code> is automatically recognized. For example:</p> <pre><code>seqkit seq a.fasta -o a.fasta.gz\n\nseqkit grep -f IDs.txt read_1.fq.gz -o dir/read_1.fq.gz\n</code></pre> <p>Seqkit utlizies the pgzip package to read and write gzip file, and the outputted gzip file would be slighty larger than files generated by GNU <code>gzip</code>.</p> <p>Seqkit writes gzip files very fast, much faster than the multi-threaded <code>pigz</code>, so there's no need to pipe the result to <code>gzip</code>/<code>pigz</code>.</p>"},{"location":"usage/#sequence-formats-and-types","title":"Sequence formats and types","text":"<p>SeqKit seamlessly support FASTA and FASTQ format. Sequence format is automatically detected. All subcommands except for <code>faidx</code> and <code>bam</code> can handle both formats. And only when some commands (<code>subseq</code>, <code>split</code>, <code>sort</code> and <code>shuffle</code>) which utilise FASTA index to improve perfrmance for large files in two pass mode (by flag <code>--two-pass</code>), only FASTA format is supported.</p> <p>Sequence type (DNA/RNA/Protein) is automatically detected by leading subsequences of the first sequences in file or STDIN. The length of the leading subsequences is configurable by global flag <code>--alphabet-guess-seq-length</code> with default value of 10000. If length of the sequences is less than that, whole sequences will be checked.</p>"},{"location":"usage/#sequence-id","title":"Sequence ID","text":"<p>By default, most softwares, including <code>seqkit</code>, take the leading non-space letters as sequence identifier (ID). For example,</p> FASTA header ID &gt;123456 gene name 123456 &gt;longname longname &gt;gi|110645304|ref|NC_002516.2| Pseudomona gi|110645304|ref|NC_002516.2| <p>But for some sequences from NCBI, e.g. <code>&gt;gi|110645304|ref|NC_002516.2| Pseudomona</code>, the ID is <code>NC_002516.2</code>. In this case, we could set sequence ID parsing regular expression by global flag <code>--id-regexp \"\\|([^\\|]+)\\| \"</code> or just use flag <code>--id-ncbi</code>. If you want the <code>gi</code> number, then use <code>--id-regexp \"^gi\\|([^\\|]+)\\|\"</code>.</p>"},{"location":"usage/#fasta-index","title":"FASTA index","text":"<p>For some commands, including <code>subseq</code>, <code>split</code>, <code>sort</code> and <code>shuffle</code>, when input files are (plain or gzipped) FASTA files, FASTA index would be optional used for rapid access of sequences and reducing memory occupation.</p> <p>ATTENTION: the <code>.seqkit.fai</code> file created by SeqKit is a little different from <code>.fai</code> file created by <code>samtools</code>. SeqKit uses full sequence head instead of just ID as key.</p>"},{"location":"usage/#parallelization-of-cpu-intensive-jobs","title":"Parallelization of CPU intensive jobs","text":"<p>The validation of sequences bases and complement process of sequences are parallelized for large sequences.</p> <p>Parsing of line-based files, including BED/GFF file and ID list file are also parallelized.</p> <p>The <code>pgzip</code> package reads and write gzip files in parallel.</p> <p>The Parallelization is implemented by multiple goroutines in golang  which are similar to but much lighter weight than threads. The concurrency number is configurable with global flag <code>-j</code> or <code>--threads</code> (default value: 4).</p> <p>Using four threads is fast enough for most commands where FASTA/Q reading and writing is the performance bottleneck, and using more threads will not increase the speed. </p> <p>Few commands could benefit from multiple (&gt;4) threads:</p> <ul> <li><code>stats</code></li> <li><code>scat</code></li> <li><code>grep -s -m</code></li> <li><code>locate -s -m</code></li> <li><code>amplicon -s -m</code></li> </ul>"},{"location":"usage/#memory-occupation","title":"Memory occupation","text":"<p>Most of the subcommands do not read whole FASTA/Q records in to memory, including <code>stat</code>, <code>fq2fa</code>, <code>fx2tab</code>, <code>tab2fx</code>, <code>grep</code>, <code>locate</code>, <code>replace</code>,  <code>seq</code>, <code>sliding</code>, <code>subseq</code>.</p> <p>Note that when using <code>subseq --gtf | --bed</code>, if the GTF/BED files are too big, the memory usage will increase. You could use <code>--chr</code> to specify chromesomes and <code>--feature</code> to limit features.</p> <p>Some subcommands could either read all records or read the files twice by flag <code>-2</code> (<code>--two-pass</code>), including <code>sample</code>, <code>split</code>, <code>shuffle</code> and <code>sort</code>. They use FASTA index for rapid acccess of sequences and reducing memory occupation.</p>"},{"location":"usage/#reproducibility","title":"Reproducibility","text":"<p>Subcommands <code>sample</code> and <code>shuffle</code> use random function, random seed could be given by flag <code>-s</code> (<code>--rand-seed</code>). This makes sure that sampling result could be reproduced in different environments with same random seed.</p>"},{"location":"usage/#seqkit","title":"seqkit","text":"<pre><code>SeqKit -- a cross-platform and ultrafast toolkit for FASTA/Q file manipulation\n\nVersion: 2.8.0\n\nAuthor: Wei Shen &lt;shenwei356@gmail.com&gt;\n\nDocuments  : http://bioinf.shenwei.me/seqkit\nSource code: https://github.com/shenwei356/seqkit\nPlease cite: https://doi.org/10.1371/journal.pone.0163962\n\n\nSeqkit utlizies the pgzip (https://github.com/klauspost/pgzip) package to\nread and write gzip file, and the outputted gzip file would be slighty\nlarger than files generated by GNU gzip.\n\nSeqkit writes gzip files very fast, much faster than the multi-threaded pigz,\ntherefore there's no need to pipe the result to gzip/pigz.\n\nSeqkit also supports reading and writing xz (.xz) and zstd (.zst) formats since v2.2.0.\nBzip2 format is supported since v2.4.0.\n\nCompression level:\n  format   range   default  comment\n  gzip     1-9     5        https://github.com/klauspost/pgzip sets 5 as the default value.\n  xz       NA      NA       https://github.com/ulikunitz/xz does not support.\n  zstd     1-4     2        roughly equals to zstd 1, 3, 7, 11, respectively.\n  bzip     1-9     6        https://github.com/dsnet/compress\n\nUsage:\n  seqkit [command]\n\nCommands for Basic Operation:\n  faidx           create the FASTA index file and extract subsequences\n  scat            real time recursive concatenation and streaming of fastx files\n  seq             transform sequences (extract ID, filter by length, remove gaps, reverse complement...)\n  sliding         extract subsequences in sliding windows\n  stats           simple statistics of FASTA/Q files\n  subseq          get subsequences by region/gtf/bed, including flanking sequences\n  translate       translate DNA/RNA to protein sequence (supporting ambiguous bases)\n  watch           monitoring and online histograms of sequence features\n\nCommands for Format Conversion:\n  convert         convert FASTQ quality encoding between Sanger, Solexa and Illumina\n  fa2fq           retrieve corresponding FASTQ records by a FASTA file\n  fq2fa           convert FASTQ to FASTA\n  fx2tab          convert FASTA/Q to tabular format (and length, GC content, average quality...)\n  tab2fx          convert tabular format to FASTA/Q format\n\nCommands for Searching:\n  amplicon        extract amplicon (or specific region around it) via primer(s)\n  fish            look for short sequences in larger sequences using local alignment\n  grep            search sequences by ID/name/sequence/sequence motifs, mismatch allowed\n  locate          locate subsequences/motifs, mismatch allowed\n\nCommands for Set Operation:\n  common          find common/shared sequences of multiple files by id/name/sequence\n  duplicate       duplicate sequences N times\n  head            print first N FASTA/Q records\n  head-genome     print sequences of the first genome with common prefixes in name\n  pair            match up paired-end reads from two fastq files\n  range           print FASTA/Q records in a range (start:end)\n  rmdup           remove duplicated sequences by ID/name/sequence\n  sample          sample sequences by number or proportion\n  split           split sequences into files by id/seq region/size/parts (mainly for FASTA)\n  split2          split sequences into files by size/parts (FASTA, PE/SE FASTQ)\n\nCommands for Edit:\n  concat          concatenate sequences with the same ID from multiple files\n  mutate          edit sequence (point mutation, insertion, deletion)\n  rename          rename duplicated IDs\n  replace         replace name/sequence by regular expression\n  restart         reset start position for circular genome\n  sana            sanitize broken single line FASTQ files\n\nCommands for Ordering:\n  shuffle         shuffle sequences\n  sort            sort sequences by id/name/sequence/length\n\nCommands for BAM Processing:\n  bam             monitoring and online histograms of BAM record features\n\nCommands for Miscellaneous:\n  merge-slides    merge sliding windows generated from seqkit sliding\n  sum             compute message digest for all sequences in FASTA/Q files\n\nAdditional Commands:\n  genautocomplete generate shell autocompletion script (bash|zsh|fish|powershell)\n  version         print version information and check for update\n\nFlags:\n      --alphabet-guess-seq-length int   length of sequence prefix of the first FASTA record based on\n                                        which seqkit guesses the sequence type (0 for whole seq)\n                                        (default 10000)\n      --compress-level int              compression level for gzip, zstd, xz and bzip2. type \"seqkit -h\"\n                                        for the range and default value for each format (default -1)\n  -h, --help                            help for seqkit\n      --id-ncbi                         FASTA head is NCBI-style, e.g. &gt;gi|110645304|ref|NC_002516.2|\n                                        Pseud...\n      --id-regexp string                regular expression for parsing ID (default \"^(\\\\S+)\\\\s?\")\n  -X, --infile-list string              file of input files list (one file per line), if given, they are\n                                        appended to files from cli arguments\n  -w, --line-width int                  line width when outputting FASTA format (0 for no wrap) (default 60)\n  -o, --out-file string                 out file (\"-\" for stdout, suffix .gz for gzipped out) (default \"-\")\n      --quiet                           be quiet and do not show extra information\n  -t, --seq-type string                 sequence type (dna|rna|protein|unlimit|auto) (for auto, it\n                                        automatically detect by the first sequence) (default \"auto\")\n  -j, --threads int                     number of CPUs. can also set with environment variable\n                                        SEQKIT_THREADS) (default 4)\n\nUse \"seqkit [command] --help\" for more information about a command.\n</code></pre>"},{"location":"usage/#datasets","title":"Datasets","text":"<p>Datasets from The miRBase Sequence Database -- Release 21</p> <ul> <li><code>hairpin.fa.gz</code></li> <li><code>mature.fa.gz</code></li> <li><code>miRNA.diff.gz</code></li> </ul> <p>Human genome from ensembl (For <code>seqkit subseq</code>)</p> <ul> <li><code>Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz</code></li> <li><code>Homo_sapiens.GRCh38.84.gtf.gz</code></li> <li><code>Homo_sapiens.GRCh38.84.bed.gz</code> is converted from <code>Homo_sapiens.GRCh38.84.gtf.gz</code> by <code>gtf2bed</code> with the command<pre><code>zcat Homo_sapiens.GRCh38.84.gtf.gz \\\n    | gtf2bed --do-not-sort \\\n    | gzip -c &gt; Homo_sapiens.GRCh38.84.bed.gz\n</code></pre> </li> </ul> <p>Only DNA and gtf/bed data of Chr1 were used:</p> <ul> <li> <p><code>chr1.fa.gz</code></p> <pre><code>seqkit grep -p 1 Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz -o chr1.fa.gz\n</code></pre> </li> <li> <p><code>chr1.gtf.gz</code></p> <pre><code>zcat Homo_sapiens.GRCh38.84.gtf.gz | grep -w '^1' | gzip -c &gt; chr1.gtf.gz\n</code></pre> </li> <li> <p><code>chr1.bed.gz</code></p> <pre><code>zcat Homo_sapiens.GRCh38.84.bed.gz | grep -w '^1' | gzip -c &gt; chr1.bed.gz\n</code></pre> </li> </ul>"},{"location":"usage/#seq","title":"seq","text":"<p>Usage</p> <pre><code>transform sequences (extract ID, filter by length, remove gaps, reverse complement...)\n\nUsage:\n  seqkit seq [flags] \n\nFlags:\n  -k, --color                 colorize sequences - to be piped into \"less -R\"\n  -p, --complement            complement sequence, flag '-v' is recommended to switch on\n      --dna2rna               DNA to RNA\n  -G, --gap-letters string    gap letters to be removed with -g/--remove-gaps (default \"- \\t.\")\n  -h, --help                  help for seq\n  -l, --lower-case            print sequences in lower case\n  -M, --max-len int           only print sequences shorter than or equal to the maximum length (-1 for\n                              no limit) (default -1)\n  -R, --max-qual float        only print sequences with average quality less than this limit (-1 for no\n                              limit) (default -1)\n  -m, --min-len int           only print sequences longer than or equal to the minimum length (-1 for no\n                              limit) (default -1)\n  -Q, --min-qual float        only print sequences with average quality greater or equal than this limit\n                              (-1 for no limit) (default -1)\n  -n, --name                  only print names/sequence headers\n  -i, --only-id               print IDs instead of full headers\n  -q, --qual                  only print qualities\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n  -g, --remove-gaps           remove gaps letters set by -G/--gap-letters, e.g., spaces, tabs, and\n                              dashes (gaps \"-\" in aligned sequences)\n  -r, --reverse               reverse sequence\n      --rna2dna               RNA to DNA\n  -s, --seq                   only print sequences\n  -u, --upper-case            print sequences in upper case\n  -v, --validate-seq          validate bases according to the alphabet\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Read and print</p> <ul> <li> <p>From file:</p> <pre><code>$ seqkit seq hairpin.fa.gz\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n\n$ seqkit seq reads_1.fq.gz\n@HWI-D00523:240:HF3WGBCXX:1:1101:2574:2226 1:N:0:CTGTAG\nTGAGGAATATTGGTCAATGGGCGCGAGCCTGAACCAGCCAAGTAGCGTGAAGGATGACTGCCCTACGGG\n+\nHIHIIIIIHIIHGHHIHHIIIIIIIIIIIIIIIHHIIIIIHHIHIIIIIGIHIIIIHHHHHHGHIHIII\n</code></pre> </li> <li> <p>From stdin:</p> <pre><code>zcat hairpin.fa.gz | seqkit seq\n</code></pre> </li> </ul> </li> <li> <p>Sequence types</p> <ul> <li> <p>By default, <code>seqkit seq</code> automatically detect the sequence type</p> <pre><code>$ echo -e \"&gt;seq\\nacgtryswkmbdhvACGTRYSWKMBDHV\" | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   DNA          1       28       28       28       28\n\n$ echo -e \"&gt;seq\\nACGUN ACGUN\" | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA          1       11       11       11       11\n\n$ echo -e \"&gt;seq\\nabcdefghijklmnpqrstvwyz\" | seqkit stats\nfile  format  type     num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   Protein         1       23       23       23       23\n\n$ echo -e \"@read\\nACTGCN\\n+\\n@IICCG\" | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTQ   DNA          1        6        6        6        6\n</code></pre> </li> <li> <p>You can also set sequence type by flag <code>-t</code> (<code>--seq-type</code>).   But this only take effect on subcommands <code>seq</code> and <code>locate</code>.</p> <pre><code>$ echo -e \"&gt;seq\\nabcdefghijklmnpqrstvwyz\" | seqkit seq -t dna\n[INFO] when flag -t (--seq-type) given, flag -v (--validate-seq) is automatically switched on\n[ERRO] error when parsing seq: seq (invalid DNAredundant letter: e)\n</code></pre> </li> </ul> </li> <li> <p>Only print names</p> <ul> <li> <p>Full name:</p> <pre><code>$ seqkit seq hairpin.fa.gz -n\ncel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\ncel-lin-4 MI0000002 Caenorhabditis elegans lin-4 stem-loop\ncel-mir-1 MI0000003 Caenorhabditis elegans miR-1 stem-loop\n</code></pre> </li> <li> <p>Only ID:</p> <pre><code>$ seqkit seq hairpin.fa.gz -n -i\ncel-let-7\ncel-lin-4\ncel-mir-1\n</code></pre> </li> <li> <p>Custom ID region by regular expression (this could be applied to all subcommands):</p> <pre><code>$ seqkit seq hairpin.fa.gz -n -i --id-regexp \"^[^\\s]+\\s([^\\s]+)\\s\"\nMI0000001\nMI0000002\nMI0000003\n</code></pre> </li> </ul> </li> <li> <p>Only print seq (global flag <code>-w</code> defines the output line width, <code>0</code> for no wrap)</p> <pre><code>$ seqkit seq hairpin.fa.gz -s -w 0\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAACUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\nAUGCUUCCGGCCUGUUCCCUGAGACCUCAAGUGUGAGUGUACUAUUGAUGCUUCACACCUGGGCUCUCCGGGUACCAGGACGGUUUGAGCAGAU\nAAAGUGACCGUACCGAGCUGCAUACUUCCUUACAUGCCCAUACUAUAUCAUAAAUGGAUAUGGAAUGUAAAGAAGUAUGUAGAACGGGGUGGUAGU\n</code></pre> </li> <li> <p>Convert multi-line FASTQ to 4-line FASTQ</p> <pre><code>$ seqkit seq reads_1.fq.gz -w 0\n</code></pre> </li> <li> <p>Reverse comlement sequence</p> <pre><code>$ seqkit seq hairpin.fa.gz -r -p\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUCGAAGAGUUCUGUCUCCGGUAAGGUAGAAAAUUGCAUAGUUCACCGGUGGUAAUAUUCC\nAAACUAUACAACCUACUACCUCACCGGAUCCACAGUGUA\n</code></pre> </li> <li> <p>Remove gaps and to lower/upper case</p> <pre><code>$ echo -e \"&gt;seq\\nACGT-ACTGC-ACC\" | seqkit seq -g -u\n&gt;seq\nACGTACTGCACC\n</code></pre> </li> <li> <p>RNA to DNA</p> <pre><code>$ echo -e \"&gt;seq\\nUCAUAUGCUUGUCUCAAAGAUUA\" | seqkit seq --rna2dna\n&gt;seq\nTCATATGCTTGTCTCAAAGATTA\n</code></pre> </li> <li> <p>Filter by sequence length</p> <pre><code>$ cat hairpin.fa | seqkit seq | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     28,645  2,949,871       39      103    2,354\n\n$ cat hairpin.fa | seqkit seq -m 100 | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     10,975  1,565,486      100    142.6    2,354\n\n$ cat hairpin.fa | seqkit seq -m 100 -M 1000 | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     10,972  1,560,270      100    142.2      938\n</code></pre> </li> </ol>"},{"location":"usage/#subseq","title":"subseq","text":"<p>Usage</p> <pre><code>get subsequences by region/gtf/bed, including flanking sequences.\n\nAttention:\n  1. When extracting with BED/GTF from plain text FASTA files, the order of output sequences\n     are random. To keep the order, just compress the FASTA file (input.fasta) and use the\n     compressed one (input.fasta.gz) as the input.\n  2. Use \"seqkit grep\" for extracting subsets of sequences.\n     \"seqtk subseq seqs.fasta id.txt\" equals to\n     \"seqkit grep -f id.txt seqs.fasta\"\n\nRecommendation:\n  1. Use plain FASTA file, so seqkit could utilize FASTA index.\n  2. The flag -U/--update-faidx is recommended to ensure the .fai file matches the FASTA file.\n\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit subseq [flags] \n\nFlags:\n      --bed string        by tab-delimited BED file\n      --chr strings       select limited sequence with sequence IDs when using --gtf or --bed (multiple\n                          value supported, case ignored)\n  -d, --down-stream int   down stream length\n      --feature strings   select limited feature types (multiple value supported, case ignored, only\n                          works with GTF)\n      --gtf string        by GTF (version 2.2) file\n      --gtf-tag string    output this tag as sequence comment (default \"gene_id\")\n  -h, --help              help for subseq\n  -f, --only-flank        only return up/down stream sequence\n  -r, --region string     by region. e.g 1:12 for first 12 bases, -12:-1 for last 12 bases, 13:-1 for\n                          cutting first 12 bases. type \"seqkit subseq -h\" for more examples\n  -R, --region-coord      append coordinates to sequence ID for -r/--region\n  -u, --up-stream int     up stream length\n  -U, --update-faidx      update the fasta index file if it exists. Use this if you are not sure whether\n                          the fasta file changed\n\n</code></pre> <p>Examples</p> <p>Recommendation: use plain FASTA file, so seqkit could utilize FASTA index.</p> <ol> <li> <p>First 12 bases</p> <pre><code>$ zcat hairpin.fa.gz | seqkit subseq -r 1:12\n</code></pre> </li> <li> <p>Last 12 bases</p> <pre><code>$ zcat hairpin.fa.gz | seqkit subseq -r -12:-1\n</code></pre> </li> <li> <p>Subsequences without first and last 12 bases</p> <pre><code>$ zcat hairpin.fa.gz | seqkit subseq -r 13:-13\n</code></pre> </li> <li> <p>Get subsequence by GTF file</p> <pre><code>$ cat t.fa\n&gt;seq\nactgACTGactgn\n\n$ cat t.gtf\nseq     test    CDS     5       8       .       .       .       gene_id \"A\"; transcript_id \"\";\nseq     test    CDS     5       8       .       -       .       gene_id \"B\"; transcript_id \"\";\n\n$ seqkit subseq --gtf t.gtf t.fa\n&gt;seq_5:8:. A\nACTG\n&gt;seq_5:8:- B\nCAGT\n</code></pre> <p>Human genome example:</p> <p>AVOID loading all data from Homo_sapiens.GRCh38.84.gtf.gz, the uncompressed data are so big and may exhaust your RAM.</p> <p>We could specify chromesomes and features.</p> <pre><code>$ seqkit subseq --gtf Homo_sapiens.GRCh38.84.gtf.gz --chr 1 --feature cds  hsa.fa &gt; chr1.gtf.cds.fa\n\n$ seqkit stats chr1.gtf.cds.fa\nfile             format  type  num_seqs    sum_len  min_len  avg_len  max_len\nchr1.gtf.cds.fa  FASTA   DNA     65,012  9,842,274        1    151.4   12,045\n</code></pre> </li> <li> <p>Get CDS and 3bp up-stream sequences</p> <pre><code>$ seqkit subseq --gtf t.gtf t.fa -u 3\n&gt;seq_5:8:._us:3 A\nctgACTG\n&gt;seq_5:8:-_us:3 B\nagtCAGT\n</code></pre> </li> <li> <p>Get 3bp up-stream sequences of CDS, not including CDS</p> <pre><code>$ seqkit subseq --gtf t.gtf t.fa -u 3 -f\n&gt;seq_5:8:._usf:3 A\nctg\n&gt;seq_5:8:-_usf:3 B\nagt\n</code></pre> </li> <li> <p>Get subsequences by BED file.</p> <p>AVOID loading all data from Homo_sapiens.GRCh38.84.gtf.gz, the uncompressed data are so big and may exhaust your RAM.</p> <pre><code>$ seqkit subseq --bed Homo_sapiens.GRCh38.84.bed.gz --chr 1 hsa.fa \\\n    &gt;  chr1.bed.gz.fa\n</code></pre> <p>We may need to remove duplicated sequences</p> <pre><code>$ seqkit subseq --bed Homo_sapiens.GRCh38.84.bed.gz --chr 1 hsa.fa \\\n    | seqkit rmdup &gt; chr1.bed.rmdup.fa\n[INFO] 141060 duplicated records removed\n</code></pre> <p>Summary:</p> <pre><code>$ seqkit stats chr1.gz.*.gz\nfile               seq_format   seq_type   num_seqs   min_len   avg_len     max_len\nchr1.gz.fa         FASTA        DNA         231,974         1   3,089.5   1,551,957\nchr1.gz.rmdup.fa   FASTA        DNA          90,914         1   6,455.8   1,551,957\n</code></pre> </li> </ol>"},{"location":"usage/#sliding","title":"sliding","text":"<p>Usage</p> <pre><code>extract subsequences in sliding windows\n\nUsage:\n  seqkit sliding [flags]\n\nFlags:\n  -c, --circular          circular genome (same to -C/--circular-genome)\n  -C, --circular-genome   circular genome (same to -c/--circular)\n  -g, --greedy            greedy mode, i.e., exporting last subsequences even shorter than the windows size\n  -h, --help              help for sliding\n  -s, --step int          step size\n  -S, --sufix string      suffix added to the sequence ID\n  -W, --window int        window size\n\n</code></pre> <p>Examples</p> <ol> <li> <p>General use</p> <pre><code>$ echo -e \"&gt;seq\\nACGTacgtNN\" | seqkit sliding -s 3 -W 6\n&gt;seq_sliding:1-6\nACGTac\n&gt;seq_sliding:4-9\nTacgtN\n</code></pre> </li> <li> <p>Greedy mode</p> <pre><code>$ echo -e \"&gt;seq\\nACGTacgtNN\" | seqkit sliding -s 3 -W 6 -g\n&gt;seq_sliding:1-6\nACGTac\n&gt;seq_sliding:4-9\nTacgtN\n&gt;seq_sliding:7-12\ngtNN\n&gt;seq_sliding:10-15\nN\n</code></pre> </li> <li> <p>Circular genome</p> <pre><code>$ echo -e \"&gt;seq\\nACGTacgtNN\" | seqkit sliding -s 3 -W 6 -C\n&gt;seq_sliding:1-6\nACGTac\n&gt;seq_sliding:4-9\nTacgtN\n&gt;seq_sliding:7-2\ngtNNAC\n&gt;seq_sliding:10-5\nNACGTa\n</code></pre> </li> <li> <p>Generate GC content for ploting</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sliding -s 5 -W 30 \\\n    | seqkit fx2tab -n -g\ncel-let-7_sliding:1-30          50.00\ncel-let-7_sliding:6-35          46.67\ncel-let-7_sliding:11-40         43.33\ncel-let-7_sliding:16-45         36.67\ncel-let-7_sliding:21-50         33.33\ncel-let-7_sliding:26-55         40.00\n...\n</code></pre> </li> </ol>"},{"location":"usage/#stats","title":"stats","text":"<p>Usage</p> <pre><code>simple statistics of FASTA/Q files\n\nColumns:\n\n  1.  file      input file, \"-\" for STDIN\n  2.  format    FASTA or FASTQ\n  3.  type      DNA, RNA, Protein or Unlimit\n  4.  num_seqs  number of sequences\n  5.  sum_len   number of bases or residues       , with gaps or spaces counted\n  6.  min_len   minimal sequence length           , with gaps or spaces counted\n  7.  avg_len   average sequence length           , with gaps or spaces counted\n  8.  max_len   miximal sequence length           , with gaps or spaces counted\n  9.  Q1        first quartile of sequence length , with gaps or spaces counted\n  10. Q2        median of sequence length         , with gaps or spaces counted\n  11. Q3        third quartile of sequence length , with gaps or spaces counted\n  12. sum_gap   number of gaps\n  13. N50       N50. https://en.wikipedia.org/wiki/N50,_L50,_and_related_statistics#N50\n  14. N50_num   N50_num or L50. https://en.wikipedia.org/wiki/N50,_L50,_and_related_statistics#L50\n  15. Q20(%)    percentage of bases with the quality score greater than 20\n  16. Q30(%)    percentage of bases with the quality score greater than 30\n  17. AvgQual   average quality\n  18. GC(%)     percentage of GC content\n\nAttention:\n  1. Sequence length metrics (sum_len, min_len, avg_len, max_len, Q1, Q2, Q3)\n     count the number of gaps or spaces. You can remove them with \"seqkit seq -g\":\n         seqkit seq -g input.fasta | seqkit stats\n\nTips:\n  1. For lots of small files (especially on SDD), use big value of '-j' to\n     parallelize counting.\n  2. Extract one metric with csvtk (https://github.com/shenwei356/csvtk):\n         seqkit stats -Ta input.fastq.gz | csvtk cut -t -f \"Q30(%)\" | csvtk del-header\n\nUsage:\n  seqkit stats [flags]\n\nAliases:\n  stats, stat\n\nFlags:\n  -N, --N strings            other N50-like stats. value range [0, 100], multiple values supported,\n                             e.g., -N 50,90 or -N 50 -N 90\n  -a, --all                  all statistics, including quartiles of seq length, sum_gap, N50\n  -b, --basename             only output basename of files\n  -E, --fq-encoding string   fastq quality encoding. available values: 'sanger', 'solexa',\n                             'illumina-1.3+', 'illumina-1.5+', 'illumina-1.8+'. (default \"sanger\")\n  -G, --gap-letters string   gap letters (default \"- .\")\n  -h, --help                 help for stats\n  -e, --skip-err             skip error, only show warning message\n  -S, --skip-file-check      skip input file checking when given files or a file list.\n  -i, --stdin-label string   label for replacing default \"-\" for stdin (default \"-\")\n  -T, --tabular              output in machine-friendly tabular format\n\n</code></pre> <p>Eexamples</p> <ol> <li> <p>General use</p> <pre><code>$ seqkit stats *.f{a,q}.gz\nfile           format  type  num_seqs    sum_len  min_len  avg_len  max_len\nhairpin.fa.gz  FASTA   RNA     28,645  2,949,871       39      103    2,354\nmature.fa.gz   FASTA   RNA     35,828    781,222       15     21.8       34\nreads_1.fq.gz  FASTQ   DNA      2,500    567,516      226      227      229\nreads_2.fq.gz  FASTQ   DNA      2,500    560,002      223      224      225\n</code></pre> </li> <li> <p>Machine-friendly tabular format</p> <pre><code>$ seqkit stats *.f{a,q}.gz -T\nfile    format  type    num_seqs        sum_len min_len avg_len max_len\nhairpin.fa.gz   FASTA   RNA     28645   2949871 39      103.0   2354\nmature.fa.gz    FASTA   RNA     35828   781222  15      21.8    34\nIllimina1.8.fq.gz       FASTQ   DNA     10000   1500000 150     150.0   150\nreads_1.fq.gz   FASTQ   DNA     2500    567516  226     227.0   229\nreads_2.fq.gz   FASTQ   DNA     2500    560002  223     224.0   225\n\n# So you can process the result with tools like csvtk: http://bioinf.shenwei.me/csvtk\n\n$ seqkit stats *.f{a,q}.gz -T | csvtk pretty -t\nfile                format   type   num_seqs   sum_len   min_len   avg_len   max_len\n-----------------   ------   ----   --------   -------   -------   -------   -------\nhairpin.fa.gz       FASTA    RNA    28645      2949871   39        103.0     2354\nmature.fa.gz        FASTA    RNA    35828      781222    15        21.8      34\nIllimina1.8.fq.gz   FASTQ    DNA    10000      1500000   150       150.0     150\nreads_1.fq.gz       FASTQ    DNA    2500       567516    226       227.0     229\nreads_2.fq.gz       FASTQ    DNA    2500       560002    223       224.0     225\n\n# To markdown\n\n$ seqkit stats *.f{a,q}.gz -T | csvtk csv2md -t\n|file             |format|type|num_seqs|sum_len|min_len|avg_len|max_len|\n|:----------------|:-----|:---|:-------|:------|:------|:------|:------|\n|hairpin.fa.gz    |FASTA |RNA |28645   |2949871|39     |103.0  |2354   |\n|mature.fa.gz     |FASTA |RNA |35828   |781222 |15     |21.8   |34     |\n|Illimina1.8.fq.gz|FASTQ |DNA |10000   |1500000|150    |150.0  |150    |\n|reads_1.fq.gz    |FASTQ |DNA |2500    |567516 |226    |227.0  |229    |\n|reads_2.fq.gz    |FASTQ |DNA |2500    |560002 |223    |224.0  |225    |\n</code></pre> file format type num_seqs sum_len min_len avg_len max_len hairpin.fa.gz FASTA RNA 28645 2949871 39 103.0 2354 mature.fa.gz FASTA RNA 35828 781222 15 21.8 34 Illimina1.8.fq.gz FASTQ DNA 10000 1500000 150 150.0 150 reads_1.fq.gz FASTQ DNA 2500 567516 226 227.0 229 reads_2.fq.gz FASTQ DNA 2500 560002 223 224.0 225 </li> <li> <p>Extra information</p> <pre><code>$ seqkit stats *.f{a,q}.gz -a\nfile               format  type  num_seqs    sum_len  min_len  avg_len  max_len   Q1   Q2   Q3  sum_gap  N50  N50_num  Q20(%)  Q30(%)  AvgQual  GC(%)\nhairpin.fa.gz      FASTA   RNA     28,645  2,949,871       39      103    2,354   76   91  111        0  101      380       0       0        0  45.77\nmature.fa.gz       FASTA   RNA     35,828    781,222       15     21.8       34   21   22   22        0   22       12       0       0        0   47.6\nIllimina1.8.fq.gz  FASTQ   DNA     10,000  1,500,000      150      150      150  150  150  150        0  150        1   96.16   89.71    24.82  49.91\nnanopore.fq.gz     FASTQ   DNA      4,000  1,798,723      153    449.7    6,006  271  318  391        0  395      585   40.79   12.63     9.48  46.66\nreads_1.fq.gz      FASTQ   DNA      2,500    567,516      226      227      229  227  227  227        0  227        3   91.24   86.62    15.45  53.63\nreads_2.fq.gz      FASTQ   DNA      2,500    560,002      223      224      225  224  224  224        0  224        2   91.06   87.66    14.62  54.77\n</code></pre> </li> <li> <p>Parallelize counting files, it's much faster for lots of small files, especially for files on SSD</p> <pre><code>seqkit stats -j 10 refseq/virual/*.fna.gz\n</code></pre> </li> <li> <p>Skip error</p> <pre><code>$ seqkit stats tests/*\n[ERRO] tests/hairpin.fa.fai: fastx: invalid FASTA/Q format\n\n$ seqkit stats tests/* -e\n[WARN] tests/hairpin.fa.fai: fastx: invalid FASTA/Q format\n[WARN] tests/hairpin.fa.seqkit.fai: fastx: invalid FASTA/Q format\n[WARN] tests/miRNA.diff.gz: fastx: invalid FASTA/Q format\n[WARN] tests/test.sh: fastx: invalid FASTA/Q format\nfile                     format  type  num_seqs    sum_len  min_len  avg_len  max_len\ntests/contigs.fa         FASTA   DNA          9         54        2        6       10\ntests/hairpin.fa         FASTA   RNA     28,645  2,949,871       39      103    2,354\ntests/Illimina1.5.fq     FASTQ   DNA          1        100      100      100      100\ntests/Illimina1.8.fq.gz  FASTQ   DNA     10,000  1,500,000      150      150      150\ntests/hairpin.fa.gz      FASTA   RNA     28,645  2,949,871       39      103    2,354\ntests/reads_1.fq.gz      FASTQ   DNA      2,500    567,516      226      227      229\ntests/mature.fa.gz       FASTA   RNA     35,828    781,222       15     21.8       34\ntests/reads_2.fq.gz      FASTQ   DNA      2,500    560,002      223      224      225\n</code></pre> </li> <li> <p>Output basename instead of full path (<code>-b/--basename</code>)</p> </li> </ol>"},{"location":"usage/#sum","title":"sum","text":"<p>Usage</p> <pre><code>compute message digest for all sequences in FASTA/Q files\n\nAttention:\n  1. Sequence headers and qualities are skipped, only sequences matter.\n  2. The order of sequences records does not matter.\n  3. Circular complete genomes are supported with the flag -c/--circular.\n     - The same double-stranded genomes with different start positions or\n       in reverse complement strand will not affect the result.\n     - For single-stranded genomes like ssRNA genomes, use -s/--single-strand.\n     - The message digest would change with different values of k-mer size.\n  4. Multiple files are processed in parallel (-j/--threads).\n\nMethod:\n  1. Converting the sequences to low cases, optionally removing gaps (-g).\n  2. Computing the hash (xxhash) for all sequences or k-mers of a circular\n     complete genome (-c/--circular).\n  3. Sorting all hash values, for ignoring the order of sequences.\n  4. Computing MD5 digest from the hash values, sequences length, and\n     the number of sequences.\n\nFollowing the seqhash in Poly (https://github.com/TimothyStiles/poly/),\nWe add meta information to the message digest, with the format of:\n\n    seqkit.&lt;version&gt;_&lt;seq type&gt;&lt;seq structure&gt;&lt;strand&gt;_&lt;kmer size&gt;_&lt;seq digest&gt;\n\n    &lt;version&gt;:       digest version\n    &lt;seq type&gt;:      'D' for DNA, 'R' for RNA, 'P' for protein, 'N' for others\n    &lt;seq structure&gt;: 'L' for linear sequence, 'C' for circular genome\n    &lt;strand&gt;:        'D' for double-stranded, 'S' for single-stranded\n    &lt;kmer size&gt;:     0 for linear sequence, other values for circular genome\n\nExamples:\n\n    seqkit.v0.1_DLS_k0_176250c8d1cde6c385397df525aa1a94    DNA.fq.gz\n    seqkit.v0.1_PLS_k0_c244954e4960dd2a1409cd8ee53d92b9    Protein.fasta\n    seqkit.v0.1_RLS_k0_0f1fb263f0c05a259ae179a61a80578d    single-stranded RNA.fasta\n\n    seqkit.v0.1_DCD_k31_e59dad6d561f1f1f28ebf185c6f4c183   double-stranded-circular DNA.fasta\n    seqkit.v0.1_DCS_k31_dd050490cd62ea5f94d73d4d636b7d60   single-stranded-circular DNA.fasta\n\nUsage:\n  seqkit sum [flags]\n\nFlags:\n  -a, --all                  show all information, including the sequences length and the number of sequences\n  -b, --basename             only output basename of files\n  -c, --circular             the file contains a single cicular genome sequence\n  -G, --gap-letters string   gap letters to delete with the flag -g/--remove-gaps (default \"- \\t.*\")\n  -h, --help                 help for sum\n  -k, --kmer-size int        k-mer size for processing circular genomes (default 1000)\n  -g, --remove-gaps          remove gap characters set in the option -G/gap-letters\n      --rna2dna              convert RNA to DNA\n  -s, --single-strand        only consider the positive strand of a circular genome, e.g., ssRNA virus\n                             genomes\n\n</code></pre> <p>Examples:</p> <p>A, B, C, D are the same vircular genomes with different starting positions or strands:</p> <pre><code>$ cat virus-{A,B,C,D}.fasta\n&gt;seq\nTGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAA\n&gt;seq.revcom\nTTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCA\n&gt;seq.new-start\nGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAAT\n&gt;seq.revcom.new-start\nTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCAT\n\n# cat to one file\n$ cat virus-{A,B,C,D}.fasta &gt; virues.fasta\n\n# shuffle and rename\n$ cat virus-{A,B,C,D}.fasta \\\n    | seqkit shuffle \\\n    | seqkit replace -p '.*' -r '{nr}' \\\n    | tee virues.shuffled.fasta\n&gt;1\nTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCAT\n&gt;2\nTGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAA\n&gt;3\nGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAAT\n&gt;4\nTTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCA\n</code></pre> <p>Sum of all files (the sequences order does not matter):</p> <pre><code>$ seqkit sum viru*.fasta\nseqkit.v0.1_DLS_k0_9bbe0abefc26013dffdde952a6725b17    virues.fasta\nseqkit.v0.1_DLS_k0_9bbe0abefc26013dffdde952a6725b17    virues.shuffled.fasta\nseqkit.v0.1_DLS_k0_176250c8d1cde6c385397df525aa1a94    virus-A.fasta\nseqkit.v0.1_DLS_k0_7a813339f9ae686b376b1df55cd596ca    virus-B.fasta\nseqkit.v0.1_DLS_k0_0fd51028bfbfa85ddbdd2b86ef7bd1c1    virus-C.fasta\nseqkit.v0.1_DLS_k0_88b1d20dd0fe0dbf41c00b075fee4e4e    virus-D.fasta\n</code></pre> <p>Circular genomes (the same genomes with different start positions or in reverse complement strand will not affect the result):</p> <pre><code>$ seqkit sum -c -k 21  virus-*.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-A.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-B.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-C.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-D.fasta\n\n$ seqkit sum -c -k 51  virus-*.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-A.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-B.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-C.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-D.fasta\n\n# collect files with the same genomes\n$ seqkit sum -c -k 51  virus-*.fasta | csvtk fold -Ht -f 1 -v 2 \nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-A.fasta; virus-B.fasta; virus-C.fasta; virus-D.fasta\n</code></pre>"},{"location":"usage/#faidx","title":"faidx","text":"<p>Usage</p> <pre><code>create the FASTA index file and extract subsequences\n\nThis command is similar with \"samtools faidx\" but has some extra features:\n\n  1. output full header line with the flag -f\n  2. support regular expression as sequence ID with the flag -r\n  3. if you have large number of IDs, you can use:\n        seqkit faidx seqs.fasta -l IDs.txt\n\nAttention:\n  1. The flag -U/--update-faidx is recommended to ensure the .fai file matches the FASTA file.\n\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit faidx [flags] &lt;fasta-file&gt; [regions...]\n\nFlags:\n  -f, --full-head            print full header line instead of just ID. New fasta index file ending with\n                             .seqkit.fai will be created\n  -h, --help                 help for faidx\n  -i, --ignore-case          ignore case\n  -I, --immediate-output     print output immediately, do not use write buffer\n  -l, --region-file string   file containing a list of regions\n  -U, --update-faidx         update the fasta index file if it exists. Use this if you are not sure\n                             whether the fasta file changed\n  -r, --use-regexp           IDs are regular expression. But subseq region is not supported here.\n\n</code></pre> <p>Example</p> <ol> <li> <p>common usage like <code>samtools faidx</code></p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1 hsa-let-7a-2\n&gt;hsa-let-7a-1\nUGGGAUGAGGUAGUAGGUUGUAUAGUUUUAGGGUCACACCCACCACUGGGAGAUAACUAU\nACAAUCUACUGUCUUUCCUA\n&gt;hsa-let-7a-2\nAGGUUGAGGUAGUAGGUUGUAUAGUUUAGAAUUACAUCAAGGGAGAUAACUGUACAGCCU\nCCUAGCUUUCCU\n</code></pre> </li> <li> <p>output full header, not supported by <code>samtools faidx</code></p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1 hsa-let-7a-2 -f\n&gt;hsa-let-7a-1 MI0000060 Homo sapiens let-7a-1 stem-loop\nUGGGAUGAGGUAGUAGGUUGUAUAGUUUUAGGGUCACACCCACCACUGGGAGAUAACUAU\nACAAUCUACUGUCUUUCCUA\n&gt;hsa-let-7a-2 MI0000061 Homo sapiens let-7a-2 stem-loop\nAGGUUGAGGUAGUAGGUUGUAUAGUUUAGAAUUACAUCAAGGGAGAUAACUGUACAGCCU\nCCUAGCUUUCCU\n</code></pre> </li> <li> <p>extract subsequence of specific region</p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:1-10\n&gt;hsa-let-7a-1:1-10\nUGGGAUGAGG\n\n$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:-10--1\n&gt;hsa-let-7a-1:-10--1\nGUCUUUCCUA\n\n$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:1\n&gt;hsa-let-7a-1:1-1\nU\n</code></pre> </li> <li> <p>supporting <code>begin</code> &gt; <code>start</code>, i.e., returning reverse complement sequence, not supported by <code>samtools faidx</code></p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:10-1\n&gt;hsa-let-7a-1:10-1\nCCUCAUCCCA\n</code></pre> </li> <li> <p>use regular expression</p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa -r | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA      1,881  154,002       41     81.9      180\n</code></pre> </li> </ol>"},{"location":"usage/#watch","title":"watch","text":"<p>Usage</p> <pre><code>monitoring and online histograms of sequence features\n\nUsage:\n  seqkit watch [flags]\n\nFlags:\n  -B, --bins int              number of histogram bins (default -1)\n  -W, --delay int             sleep this many seconds after online plotting (default 1)\n  -y, --dump                  print histogram data to stderr instead of plotting\n  -f, --fields string         target fields, available values: ReadLen, MeanQual, GC, GCSkew (default\n                              \"ReadLen\")\n  -h, --help                  help for watch\n  -O, --img string            save histogram to this PDF/image file\n  -H, --list-fields           print out a list of available fields\n  -L, --log                   log10(x+1) transform numeric values\n  -x, --pass                  pass through mode (write input to stdout)\n  -p, --print-freq int        print/report after this many records (-1 for print after EOF) (default -1)\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n  -Q, --quiet-mode            supress all plotting to stderr\n  -R, --reset                 reset histogram after every report\n  -v, --validate-seq          validate bases according to the alphabet\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Histogram of sequence length</p> <pre><code>seqkit watch --fields ReadLen nanopore.fq.gz -O len.png\n</code></pre> <p></p> </li> <li> <p>Dynamic histogram of log sequence length</p> <pre><code>seqkit watch --log --fields ReadLen nanopore.fq.gz\n</code></pre> <p></p> </li> <li> <p>Histogram of mean base qualities every 500 record, also saved as png</p> <pre><code>seqkit watch -p 500 -O qhist.png -f MeanQual nanopore.fq.gz\n</code></pre> </li> </ol>"},{"location":"usage/#sana","title":"sana","text":"<p>Usage</p> <pre><code>sanitize broken single line FASTQ files\n\nSana is a resilient FASTQ/FASTA parser. Unlike many parsers,\nit won't stop at the first error. Instead, it skips malformed records\nand continues processing the file.\n\nSana currently supports this FASTQ dialect:\n\n  - One line for each sequence and quality value\n\nUsage:\n  seqkit sana [flags]\n\nFlags:\n  -A, --allow-gaps            allow gap character (-) in sequences\n  -i, --format string         input and output format: fastq or fasta (default \"fastq\")\n  -h, --help                  help for sana\n  -I, --in-format string      input format: fastq or fasta\n  -O, --out-format string     output format: fastq or fasta\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n</code></pre> <p>Examples</p> <ol> <li> <p>Rescue usable reads from fastq file with malformed records.</p> <pre><code>seqkit sana broken.fq.gz -o rescued.fq.gz\n</code></pre> </li> <li> <p>For paired reads, you can process both files and pair them with <code>pair</code>.</p> <pre><code>seqkit sana broken_1.fq.gz -o rescued_1.fq.gz\nseqkit sana broken_2.fq.gz -o rescued_2.fq.gz\n\nseqkit pair -1 rescued_1.fq.gz -2 rescued_2.fq.gz -O rescued/\n</code></pre> </li> </ol>"},{"location":"usage/#scat","title":"scat","text":"<p>Usage</p> <pre><code>real time recursive concatenation and streaming of fastx files\n\nUsage:\n  seqkit scat [flags]\n\nFlags:\n  -A, --allow-gaps            allow gap character (-) in sequences\n  -d, --delta int             minimum size increase in kilobytes to trigger parsing (default 5)\n  -D, --drop-time string      Notification drop interval (default \"500ms\")\n  -f, --find-only             concatenate exisiting files and quit\n  -i, --format string         input and output format: fastq or fasta (fastq) (default \"fastq\")\n  -g, --gz-only               only look for gzipped files (.gz suffix)\n  -h, --help                  help for scat\n  -I, --in-format string      input format: fastq or fasta (fastq)\n  -O, --out-format string     output format: fastq or fasta\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n  -r, --regexp string         regexp for watched files, by default guessed from the input format\n  -T, --time-limit string     quit after inactive for this time period\n  -p, --wait-pid int          after process with this PID exited (default -1)\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Concatenate all fastq files recursively under a directory</p> <pre><code>seqkit scat -j 4 -f fastq_dir &gt; all_records.fq\n</code></pre> </li> <li> <p>Watch a directory and stream fastq records in real time until interrupt is recieved and plot read lengths using <code>seqkit watch</code>:</p> <pre><code>seqkit scat -j 4 fastq_dir | seqkit watch -f ReadLen -\n</code></pre> </li> <li> <p>Watch a directory and stream fastq records in real time until there is no write activity under the directory for 5 seconds:</p> <pre><code>seqkit scat -j 4 -T \"5s\" fastq_dir &gt; all_records.fq\n</code></pre> </li> <li> <p>Watch a directory and stream fastq records in real time until a process with a specified PID is alive:</p> <pre><code>seqkit scat -j 4 -p $PID fastq_dir &gt; all_records.fq\n</code></pre> </li> </ol> <p>Notes: You might need to increase the <code>ulimit</code> allowance on open files if you intend to stream fastx records from a large number of files.</p>"},{"location":"usage/#fq2fa","title":"fq2fa","text":"<p>Usage</p> <pre><code>convert FASTQ to FASTA\n\nUsage:\n  seqkit fq2fa [flags]\n\n</code></pre> <p>Examples</p> <pre><code>seqkit fq2fa reads_1.fq.gz -o reads_1.fa.gz\n</code></pre>"},{"location":"usage/#fa2fq","title":"fa2fq","text":"<p>Usage</p> <pre><code>retrieve corresponding FASTQ records by a FASTA file\n\nAttention:\n  1. We assume the FASTA file comes from the FASTQ file,\n     so they share sequence IDs, and sequences in FASTA\n     should be subseq of sequences in FASTQ file.\n\nUsage:\n  seqkit fa2fq [flags]\n\nFlags:\n  -f, --fasta-file string      FASTA file)\n  -h, --help                   help for fa2fq\n  -P, --only-positive-strand   only search on positive strand\n</code></pre>"},{"location":"usage/#fx2tab-tab2fx","title":"fx2tab &amp; tab2fx","text":"<p>Usage (fx2tab)</p> <pre><code>convert FASTA/Q to tabular format, and provide various information,\nlike sequence length, GC content/GC skew.\n\nAttention:\n  1. Fixed three columns (ID, sequence, quality) are outputted for either FASTA\n     or FASTQ, except when flag -n/--name is on. This is for format compatibility.\n\nUsage:\n  seqkit fx2tab [flags]\n\nFlags:\n  -a, --alphabet               print alphabet letters\n  -q, --avg-qual               print average quality of a read\n  -B, --base-content strings   print base content. (case ignored, multiple values supported) e.g. -B AT -B N\n  -C, --base-count strings     print base count. (case ignored, multiple values supported) e.g. -C AT -C N\n  -I, --case-sensitive         calculate case sensitive base content/sequence hash\n  -g, --gc                     print GC content\n  -G, --gc-skew                print GC-Skew\n  -H, --header-line            print header line\n  -h, --help                   help for fx2tab\n  -l, --length                 print sequence length\n  -n, --name                   only print names (no sequences and qualities)\n  -Q, --no-qual                only output two column even for FASTQ file\n  -i, --only-id                print ID instead of full head\n  -b, --qual-ascii-base int    ASCII BASE, 33 for Phred+33 (default 33)\n  -s, --seq-hash               print hash (MD5) of sequence\n\n</code></pre> <p>Usage (tab2fx)</p> <pre><code>convert tabular format (first two/three columns) to FASTA/Q format\n\nUsage:\n  seqkit tab2fx [flags] \n\nFlags:\n  -b, --buffer-size string            size of buffer, supported unit: K, M, G. You need increase the\n                                      value when \"bufio.Scanner: token too long\" error reported (default\n                                      \"1G\")\n  -p, --comment-line-prefix strings   comment line prefix (default [#,//])\n  -h, --help                          help for tab2fx\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Default output</p> <pre><code>$ seqkit fx2tab hairpin.fa.gz | head -n 2\ncel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop      UACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAACUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\ncel-lin-4 MI0000002 Caenorhabditis elegans lin-4 stem-loop      AUGCUUCCGGCCUGUUCCCUGAGACCUCAAGUGUGAGUGUACUAUUGAUGCUUCACACCUGGGCUCUCCGGGUACCAGGACGGUUUGAGCAGAU\n</code></pre> </li> <li> <p>Print sequence length, GC content, and only print names (no sequences), we could also print title line by flag <code>-H</code>.</p> <pre><code>$ seqkit fx2tab hairpin.fa.gz -l -g -n -i -H | head -n 4 | csvtk -t -C '&amp;' pretty\n#name       seq   qual   length   GC\ncel-let-7                99       43.43\ncel-lin-4                94       54.26\ncel-mir-1                96       40.62\n</code></pre> </li> <li> <p>Use fx2tab and tab2fx in pipe</p> <pre><code>$ zcat hairpin.fa.gz | seqkit fx2tab | seqkit tab2fx\n\n$ zcat reads_1.fq.gz | seqkit fx2tab | seqkit tab2fx\n</code></pre> </li> <li> <p>Sort sequences by length (use <code>seqkit sort -l</code>)</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit fx2tab -l \\\n    | sort -t\"`echo -e '\\t'`\" -n -k4,4 \\\n    | seqkit tab2fx\n&gt;cin-mir-4129 MI0015684 Ciona intestinalis miR-4129 stem-loop\nUUCGUUAUUGGAAGACCUUAGUCCGUUAAUAAAGGCAUC\n&gt;mmu-mir-7228 MI0023723 Mus musculus miR-7228 stem-loop\nUGGCGACCUGAACAGAUGUCGCAGUGUUCGGUCUCCAGU\n&gt;cin-mir-4103 MI0015657 Ciona intestinalis miR-4103 stem-loop\nACCACGGGUCUGUGACGUAGCAGCGCUGCGGGUCCGCUGU\n\n$ seqkit sort -l hairpin.fa.gz\n</code></pre> <p>Sorting or filtering by GC (or other base by -flag <code>-B</code>) content could also achieved in similar way.</p> </li> <li> <p>Get first 1000 sequences (use <code>seqkit head -n 1000</code>)</p> <pre><code>$ seqkit fx2tab hairpin.fa.gz | head -n 1000 | seqkit tab2fx\n\n$ seqkit fx2tab reads_1.fq.gz | head -n 1000 | seqkit tab2fx\n</code></pre> </li> </ol> <p>Extension</p> <p>After converting FASTA to tabular format with <code>seqkit fx2tab</code>, it could be handled with CSV/TSV tools,  e.g. csvtk, a cross-platform, efficient and practical CSV/TSV toolkit</p> <ul> <li><code>csvtk grep</code> could be used to filter sequences (similar with <code>seqkit grep</code>)</li> <li><code>csvtk inter</code> computates intersection of multiple files. It could achieve similar function as <code>seqkit common -n</code> along with shell.</li> <li><code>csvtk join</code> joins multiple CSV/TSV files by multiple IDs.</li> </ul>"},{"location":"usage/#convert","title":"convert","text":"<p>Usage</p> <pre><code>convert FASTQ quality encoding between Sanger, Solexa and Illumina\n\nUsage:\n  seqkit convert [flags]\n\nFlags:\n  -d, --dry-run                         dry run\n  -f, --force                           for Illumina-1.8+ -&gt; Sanger, truncate scores &gt; 40 to 40\n      --from string                     source quality encoding. if not given, we'll guess it\n  -h, --help                            help for convert\n  -n, --nrecords int                    number of records for guessing quality encoding (default 1000)\n  -N, --thresh-B-in-n-most-common int   threshold of 'B' in top N most common quality for guessing\n                                        Illumina 1.5. (default 2)\n  -F, --thresh-illumina1.5-frac float   threshold of faction of Illumina 1.5 in the leading N records\n                                        (default 0.1)\n      --to string                       target quality encoding (default \"Sanger\")\n</code></pre> <p>Examples:</p> <p>Note that <code>seqkit convert</code> always output sequences.</p> <p>The test dataset contains score 41 (<code>J</code>):</p> <pre><code>$ seqkit head -n 1 tests/Illimina1.8.fq.gz\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n#AAAFAAJFFFJJJ&lt;JJJJJFFFJFJJJJJFJJAJJJFJJFJFJJJJFAFJ&lt;JA&lt;FFJ7FJJFJJAAJJJJ&lt;JJJJJJJFJJJAJJJJJFJJ77&lt;JJJJ-F7A-FJFFJJJJJJ&lt;FFJ-&lt;7FJJJFJJ)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>By default, nothing changes when converting Illumina 1.8 to Sanger. A warning message show that source and target quality encoding match.</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz  | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Sanger\n[WARN] source and target quality encoding match.\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n#AAAFAAJFFFJJJ&lt;JJJJJFFFJFJJJJJFJJAJJJFJJFJFJJJJFAFJ&lt;JA&lt;FFJ7FJJFJJAAJJJJ&lt;JJJJJJJFJJJAJJJJJFJJ77&lt;JJJJ-F7A-FJFFJJJJJJ&lt;FFJ-&lt;7FJJJFJJ)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>When switching flag <code>--force</code> on,  <code>J</code> (41) was converted to <code>I</code> (40).</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz -f | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Sanger\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n#AAAFAAIFFFIII&lt;IIIIIFFFIFIIIIIFIIAIIIFIIFIFIIIIFAFI&lt;IA&lt;FFI7FIIFIIAAIIII&lt;IIIIIIIFIIIAIIIIIFII77&lt;IIII-F7A-FIFFIIIIII&lt;FFI-&lt;7FIIIFII)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>Other cases:</p> <p>To Illumina-1.5.</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz --to Illumina-1.5+ | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Illumina-1.5+\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\nB```e``ieeeiii[iiiiieeeieiiiiieii`iiieiieieiiiie`ei[i`[eeiVeiieii``iiii[iiiiiiieiii`iiiiieiiVV[iiiiLeV`Leieeiiiiii[eeiL[VeiiieiiH`VHV``[VLLH[LVeL`Ve`[\n</code></pre> <p>To Illumina-1.5 and back to Sanger.</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz --to Illumina-1.5+ | seqkit convert | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Illumina-1.5+\n[INFO] possible quality encodings: [Illumina-1.5+]\n[INFO] guessed quality encoding: Illumina-1.5+\n[INFO] converting Illumina-1.5+ -&gt; Sanger\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n!AAAFAAJFFFJJJ&lt;JJJJJFFFJFJJJJJFJJAJJJFJJFJFJJJJFAFJ&lt;JA&lt;FFJ7FJJFJJAAJJJJ&lt;JJJJJJJFJJJAJJJJJFJJ77&lt;JJJJ-F7A-FJFFJJJJJJ&lt;FFJ-&lt;7FJJJFJJ)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>Checking encoding</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz --from Solexa\n[INFO] converting Solexa -&gt; Sanger\n[ERRO] seq: invalid Solexa quality\n</code></pre> <p>Real Illumina 1.5+ data</p> <pre><code>$ seqkit seq tests/Illimina1.5.fq\n@HWI-EAS209_0006_FC706VJ:5:58:5894:21141#ATCACG/1\nTTAATTGGTAAATAAATCTCCTAATAGCTTAGATNTTACCTTNNNNNNNNNNTAGTTTCTTGAGATTTGTTGGGGGAGACATTTTTGTGATTGCCTTGAT\n+\nefcfffffcfeefffcffffffddf`feed]`]_Ba_^__[YBBBBBBBBBBRTT\\]][]dddd`ddd^dddadd^BBBBBBBBBBBBBBBBBBBBBBBB\n\n$ seqkit convert tests/Illimina1.5.fq | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.5+]\n[INFO] guessed quality encoding: Illumina-1.5+\n[INFO] converting Illumina-1.5+ -&gt; Sanger\n@HWI-EAS209_0006_FC706VJ:5:58:5894:21141#ATCACG/1\nTTAATTGGTAAATAAATCTCCTAATAGCTTAGATNTTACCTTNNNNNNNNNNTAGTTTCTTGAGATTTGTTGGGGGAGACATTTTTGTGATTGCCTTGAT\n+\nFGDGGGGGDGFFGGGDGGGGGGEEGAGFFE&gt;A&gt;@!B@?@@&lt;:!!!!!!!!!!355=&gt;&gt;&lt;&gt;EEEEAEEE?EEEBEE?!!!!!!!!!!!!!!!!!!!!!!!!\n</code></pre>"},{"location":"usage/#translate","title":"translate","text":"<p>Usage</p> <pre><code>translate DNA/RNA to protein sequence (supporting ambiguous bases)\n\nNote:\n\n  1. This command supports codons containing any ambiguous base.\n     Please switch on flag -L INT for details. e.g., for standard table:\n\n        ACN -&gt; T\n        CCN -&gt; P\n        CGN -&gt; R\n        CTN -&gt; L\n        GCN -&gt; A\n        GGN -&gt; G\n        GTN -&gt; V\n        TCN -&gt; S\n\n        MGR -&gt; R\n        YTR -&gt; L\n\nTranslate Tables/Genetic Codes:\n\n    # https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html/index.cgi?chapter=tgencodes\n\n     1: The Standard Code\n     2: The Vertebrate Mitochondrial Code\n     3: The Yeast Mitochondrial Code\n     4: The Mold, Protozoan, and Coelenterate Mitochondrial Code and the Mycoplasma/Spiroplasma Code\n     5: The Invertebrate Mitochondrial Code\n     6: The Ciliate, Dasycladacean and Hexamita Nuclear Code\n     9: The Echinoderm and Flatworm Mitochondrial Code\n    10: The Euplotid Nuclear Code\n    11: The Bacterial, Archaeal and Plant Plastid Code\n    12: The Alternative Yeast Nuclear Code\n    13: The Ascidian Mitochondrial Code\n    14: The Alternative Flatworm Mitochondrial Code\n    16: Chlorophycean Mitochondrial Code\n    21: Trematode Mitochondrial Code\n    22: Scenedesmus obliquus Mitochondrial Code\n    23: Thraustochytrium Mitochondrial Code\n    24: Pterobranchia Mitochondrial Code\n    25: Candidate Division SR1 and Gracilibacteria Code\n    26: Pachysolen tannophilus Nuclear Code\n    27: Karyorelict Nuclear\n    28: Condylostoma Nuclear\n    29: Mesodinium Nuclear\n    30: Peritrich Nuclear\n    31: Blastocrithidia Nuclear\n\nUsage:\n  seqkit translate [flags]\n\nFlags:\n  -x, --allow-unknown-codon                     translate unknown code to 'X'. And you may not use flag\n                                                --trim which removes 'X'\n  -F, --append-frame                            append frame information to sequence ID\n      --clean                                   change all STOP codon positions from the '*' character\n                                                to 'X' (an unknown residue)\n  -f, --frame strings                           frame(s) to translate, available value: 1, 2, 3, -1, -2,\n                                                -3, and 6 for all six frames (default [1])\n  -h, --help                                    help for translate\n  -M, --init-codon-as-M                         translate initial codon at beginning to 'M'\n  -l, --list-transl-table int                   show details of translate table N, 0 for all (default -1)\n  -L, --list-transl-table-with-amb-codons int   show details of translate table N (including ambigugous\n                                                codons), 0 for all.  (default -1)\n  -m, --min-len int                             the minimum length of amino acid sequence\n  -s, --out-subseqs                             output individual amino acid subsequences seperated by\n                                                the stop symbol \"*\"\n  -T, --transl-table int                        translate table/genetic code, type 'seqkit translate\n                                                --help' for more details (default 1)\n      --trim                                    remove all 'X' and '*' characters from the right end of\n                                                the translation\n\n</code></pre> <p>Examples</p> <ol> <li> <p>common usage</p> <pre><code>$ seqkit translate tests/mouse-p53-cds.fna\n&gt;lcl|AB021961.1_cds_BAA82344.1_1 [gene=p53] [protein=P53] [protein_id=BAA82344.1] [location=101..1273] [gbkey=CDS]\nMTAMEESQSDISLELPLSQETFSGLWKLLPPEDILPSPHCMDDLLLPQDVEEFFEGPSEA\nLRVSGAPAAQDPVTETPGPVAPAPATPWPLSSFVPSQKTYQGNYGFHLGFLQSGTAKSVM\nCTYSPPLNKLFCQLAKTCPVQLWVSATPPAGSRVRAMAIYKKSQHMTEVVRRCPHHERCS\nDGDGLAPPQHRIRVEGNLYPEYLEDRQTFRHSVVVPYEPPEAGSEYTTIHYKYMCNSSCM\nGGMNRRPILTIITLEDSSGNLLGRDSFEVRVCACPGRDRRTEEENFRKKEVLCPELPPGS\nAKRALPTCTSASPPQKKKPLDGEYFTLKIRGRKRFEMFRELNEALELKDAHATEESGDSR\nAHSSYLKTKKGQSTSRHKKTMVKKVGPDSD*\n</code></pre> </li> <li> <p>trim the <code>*</code></p> <pre><code>$ seqkit translate tests/mouse-p53-cds.fna --trim\n&gt;lcl|AB021961.1_cds_BAA82344.1_1 [gene=p53] [protein=P53] [protein_id=BAA82344.1] [location=101..1273] [gbkey=CDS]\nMTAMEESQSDISLELPLSQETFSGLWKLLPPEDILPSPHCMDDLLLPQDVEEFFEGPSEA\nLRVSGAPAAQDPVTETPGPVAPAPATPWPLSSFVPSQKTYQGNYGFHLGFLQSGTAKSVM\nCTYSPPLNKLFCQLAKTCPVQLWVSATPPAGSRVRAMAIYKKSQHMTEVVRRCPHHERCS\nDGDGLAPPQHRIRVEGNLYPEYLEDRQTFRHSVVVPYEPPEAGSEYTTIHYKYMCNSSCM\nGGMNRRPILTIITLEDSSGNLLGRDSFEVRVCACPGRDRRTEEENFRKKEVLCPELPPGS\nAKRALPTCTSASPPQKKKPLDGEYFTLKIRGRKRFEMFRELNEALELKDAHATEESGDSR\nAHSSYLKTKKGQSTSRHKKTMVKKVGPDSD\n</code></pre> </li> <li> <p>different translate table</p> <pre><code>$ cat tests/Lactococcus-lactis-phage-BK5-T-ORF25.fasta \\\n    | seqkit translate -T 11 --trim\n&gt;CAC80166.1 hypothetical protein [Lactococcus phage BK5-T]\nMEEQAWREVLERLARIETKLDNYETVRDKAERALLIAQSNAKLIEKMEANNKWAWGFMLT\nLAVTVIGYLFTKIRF\n</code></pre> </li> <li> <p>different frame</p> <pre><code>$ cat tests/Lactococcus-lactis-phage-BK5-T-ORF25.fasta \\\n    | seqkit translate -T 11 --frame -1\n&gt;CAC80166.1 hypothetical protein [Lactococcus phage BK5-T]\nSESNFSE*ITNNSYGKSKHKAPSPLIISFHFFYKFRI*LSY*ERSFCFISNCFIVI*LCF\nNSS*TFEDFSPCLFLH\n\n$ cat tests/Lactococcus-lactis-phage-BK5-T-ORF25.fasta \\\n    | seqkit seq -r -p \\\n    | seqkit translate -T 11 --frame -1\n&gt;CAC80166.1 hypothetical protein [Lactococcus phage BK5-T]\nMEEQAWREVLERLARIETKLDNYETVRDKAERALLIAQSNAKLIEKMEANNKWAWGFMLT\nLAVTVIGYLFTKIRF*\n</code></pre> </li> <li> <p>show details of translate table 1</p> <pre><code>$ seqkit translate -l 1\nThe Standard Code (transl_table=1)\nSource: https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html/index.cgi?chapter=tgencodes#SG1\n\nInitiation Codons:\n    ATG, CTG, TTG\n\nStop Codons:\n    TAA, TAG, TGA\n\nStranslate Table:\n    AAA: K, AAC: N, AAG: K, AAT: N\n    ACA: T, ACC: T, ACG: T, ACT: T\n    AGA: R, AGC: S, AGG: R, AGT: S\n    ATA: I, ATC: I, ATG: M, ATT: I\n\n    CAA: Q, CAC: H, CAG: Q, CAT: H\n    CCA: P, CCC: P, CCG: P, CCT: P\n    CGA: R, CGC: R, CGG: R, CGT: R\n    CTA: L, CTC: L, CTG: L, CTT: L\n\n    GAA: E, GAC: D, GAG: E, GAT: D\n    GCA: A, GCC: A, GCG: A, GCT: A\n    GGA: G, GGC: G, GGG: G, GGT: G\n    GTA: V, GTC: V, GTG: V, GTT: V\n\n    TAA: *, TAC: Y, TAG: *, TAT: Y\n    TCA: S, TCC: S, TCG: S, TCT: S\n    TGA: *, TGC: C, TGG: W, TGT: C\n    TTA: L, TTC: F, TTG: L, TTT: F\n</code></pre> </li> <li> <p>show details of translate table 1, including ambigugous codons</p> <pre><code>$ seqkit translate -L 1\nThe Standard Code (transl_table=1)\nSource: https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html/index.cgi?chapter=tgencodes#SG1\n\nInitiation Codons:\n    ATG, CTG, TTG\n\nStop Codons:\n    TAA, TAG, TGA\n\nStranslate Table:\n    AAA: K, AAC: N, AAG: K, AAR: K, AAT: N, AAY: N\n    ACA: T, ACC: T, ACM: T, ACG: T, ACR: T, ACS: T, ACV: T, ACT: T, ACW: T, ACY: T, ACH: T, ACK: T, ACD: T, ACB: T, ACN: T\n    AGA: R, AGC: S, AGG: R, AGR: R, AGT: S, AGY: S\n    ATA: I, ATC: I, ATM: I, ATG: M, ATT: I, ATW: I, ATY: I, ATH: I\n\n    CAA: Q, CAC: H, CAG: Q, CAR: Q, CAT: H, CAY: H\n    CCA: P, CCC: P, CCM: P, CCG: P, CCR: P, CCS: P, CCV: P, CCT: P, CCW: P, CCY: P, CCH: P, CCK: P, CCD: P, CCB: P, CCN: P\n    CGA: R, CGC: R, CGM: R, CGG: R, CGR: R, CGS: R, CGV: R, CGT: R, CGW: R, CGY: R, CGH: R, CGK: R, CGD: R, CGB: R, CGN: R\n    CTA: L, CTC: L, CTM: L, CTG: L, CTR: L, CTS: L, CTV: L, CTT: L, CTW: L, CTY: L, CTH: L, CTK: L, CTD: L, CTB: L, CTN: L\n\n    MGA: R, MGG: R, MGR: R\n\n    GAA: E, GAC: D, GAG: E, GAR: E, GAT: D, GAY: D\n    GCA: A, GCC: A, GCM: A, GCG: A, GCR: A, GCS: A, GCV: A, GCT: A, GCW: A, GCY: A, GCH: A, GCK: A, GCD: A, GCB: A, GCN: A\n    GGA: G, GGC: G, GGM: G, GGG: G, GGR: G, GGS: G, GGV: G, GGT: G, GGW: G, GGY: G, GGH: G, GGK: G, GGD: G, GGB: G, GGN: G\n    GTA: V, GTC: V, GTM: V, GTG: V, GTR: V, GTS: V, GTV: V, GTT: V, GTW: V, GTY: V, GTH: V, GTK: V, GTD: V, GTB: V, GTN: V\n\n    TAA: *, TAC: Y, TAG: *, TAR: *, TAT: Y, TAY: Y\n    TCA: S, TCC: S, TCM: S, TCG: S, TCR: S, TCS: S, TCV: S, TCT: S, TCW: S, TCY: S, TCH: S, TCK: S, TCD: S, TCB: S, TCN: S\n    TGA: *, TGC: C, TGG: W, TGT: C, TGY: C\n    TRA: *\n    TTA: L, TTC: F, TTG: L, TTR: L, TTT: F, TTY: F\n\n    YTA: L, YTG: L, YTR: L\n</code></pre> </li> </ol>"},{"location":"usage/#grep","title":"grep","text":"<p>Usage</p> <pre><code>search sequences by ID/name/sequence/sequence motifs, mismatch allowed\n\nAttention:\n\n  0. By default, we match sequence ID with patterns, use \"-n/--by-name\"\n     for matching full name instead of just ID.\n  1. Unlike POSIX/GNU grep, we compare the pattern to the whole target\n     (ID/full header) by default. Please switch \"-r/--use-regexp\" on\n     for partly matching.\n  2. When searching by sequences, it's partly matching, and both positive\n     and negative strands are searched.\n     Please switch on \"-P/--only-positive-strand\" if you would like to\n     search only on the positive strand.\n     Mismatch is allowed using flag \"-m/--max-mismatch\", you can increase\n     the value of \"-j/--threads\" to accelerate processing.\n  3. Degenerate bases/residues like \"RYMM..\" are also supported by flag -d.\n     But do not use degenerate bases/residues in regular expression, you need\n     convert them to regular expression, e.g., change \"N\" or \"X\"  to \".\".\n  4. When providing search patterns (motifs) via flag '-p',\n     please use double quotation marks for patterns containing comma,\n     e.g., -p '\"A{2,}\"' or -p \"\\\"A{2,}\\\"\". Because the command line argument\n     parser accepts comma-separated-values (CSV) for multiple values (motifs).\n     Patterns in file do not follow this rule.\n  5. The order of sequences in result is consistent with that in original\n     file, not the order of the query patterns.\n     But for FASTA file, you can use:\n        seqkit faidx seqs.fasta --infile-list IDs.txt\n  6. For multiple patterns, you can either set \"-p\" multiple times, i.e.,\n     -p pattern1 -p pattern2, or give a file of patterns via \"-f/--pattern-file\".\n\nYou can specify the sequence region for searching with the flag -R (--region).\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit grep [flags]\n\nFlags:\n  -D, --allow-duplicated-patterns   output records multiple times when duplicated patterns are given\n  -n, --by-name                     match by full name instead of just ID\n  -s, --by-seq                      search subseq on seq. Both positive and negative strand are searched\n                                    by default, you might use -P/--only-positive-strand. Mismatch\n                                    allowed using flag -m/--max-mismatch\n  -c, --circular                    circular genome\n  -C, --count                       just print a count of matching records. with the -v/--invert-match\n                                    flag, count non-matching records\n  -d, --degenerate                  pattern/motif contains degenerate base\n      --delete-matched              delete a pattern right after being matched, this keeps the firstly\n                                    matched data and speedups when using regular expressions\n  -h, --help                        help for grep\n  -i, --ignore-case                 ignore case\n  -I, --immediate-output            print output immediately, do not use write buffer\n  -v, --invert-match                invert the sense of matching, to select non-matching records\n  -m, --max-mismatch int            max mismatch when matching by seq. For large genomes like human\n                                    genome, using mapping/alignment tools would be faster\n  -P, --only-positive-strand        only search on the positive strand\n  -p, --pattern strings             search pattern (multiple values supported. Attention: use double\n                                    quotation marks for patterns containing comma, e.g., -p '\"A{2,}\"')\n  -f, --pattern-file string         pattern file (one record per line)\n  -R, --region string               specify sequence region for searching. e.g 1:12 for first 12 bases,\n                                    -12:-1 for last 12 bases\n  -r, --use-regexp                  patterns are regular expression\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Searching with list of sequence IDs (do not contain whitespace)</p> <pre><code>$ seqkit grep -f id.txt seqs.fq.gz -o result.fq.gz\n\n# ignore case\n$ seqkit grep -i -f id.txt seqs.fq.gz -o result.fq.gz\n</code></pre> </li> <li> <p>Just print the matched number, like GNU grep (<code>grep -c</code>) </p> <pre><code>$ seqkit grep -f id.txt seqs.fq.gz -C\n</code></pre> </li> <li> <p>Serching non-canonical sequence IDs, Using <code>--id-regexp</code> to capture IDs.     Refer to section Sequence ID and seqkit seq for examples.</p> </li> <li> <p>Searching with list of sequence names (they may contain whitespace).</p> <pre><code>$ seqkit grep -n -f name.txt seqs.fa.gz -o result.fa.gz\n</code></pre> </li> <li> <p>Useq <code>-r/--use-regexp</code> for partly matching, but this may produce \"false positive\" matches.    For example, <code>seq_1</code> matches <code>seq_10</code> with <code>-nri</code>.</p> </li> <li> <p>Extract human hairpins (i.e. sequences with name starting with <code>hsa</code>)</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -r -p ^hsa\n&gt;hsa-let-7a-1 MI0000060 Homo sapiens let-7a-1 stem-loop\nUGGGAUGAGGUAGUAGGUUGUAUAGUUUUAGGGUCACACCCACCACUGGGAGAUAACUAU\nACAAUCUACUGUCUUUCCUA\n&gt;hsa-let-7a-2 MI0000061 Homo sapiens let-7a-2 stem-loop\nAGGUUGAGGUAGUAGGUUGUAUAGUUUAGAAUUACAUCAAGGGAGAUAACUGUACAGCCU\nCCUAGCUUUCCU\n</code></pre> </li> <li> <p>Remove human and mice hairpins (invert match with <code>-v</code>)</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -r -p ^hsa -p ^mmu -v\n</code></pre> </li> <li> <p>Extract new entries by information from miRNA.diff.gz</p> <ol> <li> <p>Get IDs of new entries.</p> <pre><code>$ zcat miRNA.diff.gz | grep ^# -v | grep NEW | cut -f 2 &gt; list\n$ more list\ncfa-mir-486\ncfa-mir-339-1\npmi-let-7\n</code></pre> </li> <li> <p>Extract by ID list file</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -f list &gt; new.fa\n</code></pre> </li> </ol> </li> <li> <p>Extract sequences containing AGGCG</p> <pre><code>$ cat hairpin.fa.gz | seqkit grep -s -i -p aggcg\n</code></pre> </li> <li> <p>Circular genome</p> <pre><code>$ echo -e \"&gt;seq\\nACGTTGCA\" \n&gt;seq\nACGTTGCA\n\n$ echo -e \"&gt;seq\\nACGTTGCA\"  | seqkit grep -s -i -p AA\n\n$ echo -e \"&gt;seq\\nACGTTGCA\"  | seqkit grep -s -i -p AA -c\n&gt;seq\nACGTTGCA\n</code></pre> </li> <li> <p>Extract sequences containing AGGCG (allow mismatch)</p> <pre><code>$ time cat hairpin.fa.gz | seqkit grep -s -i -p aggcg | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA      1,181  145,789       49    123.4    2,354\n\nreal    0m0.058s\nuser    0m0.100s\nsys     0m0.017s\n\n$ time zcat hairpin.fa.gz | seqkit grep -s -i -p aggcg -m 1 | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     22,290  2,375,819       39    106.6    2,354\n\nreal    0m1.081s\nuser    0m1.305s\nsys     0m0.158s\n</code></pre> </li> <li> <p>Extract sequences starting with AGGCG</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -s -r -i -p ^aggcg\n</code></pre> </li> <li> <p>Extract sequences with TTSAA (AgsI digest site) in SEQUENCE. Base S stands for C or G.</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -s -d -i -p TTSAA\n</code></pre> <p>It's equal to but simpler than:</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -s -r -i -p TT[CG]AA\n</code></pre> </li> <li> <p>Specify sequence regions for searching. e.g., leading 30 bases.</p> <pre><code>$ seqkit grep -s -R 1:30 -i -r -p GCTGG\n</code></pre> </li> </ol>"},{"location":"usage/#locate","title":"locate","text":"<p>Usage</p> <pre><code>locate subsequences/motifs, mismatch allowed\n\nAttention:\n\n  1. Motifs could be EITHER plain sequence containing \"ACTGN\" OR regular\n     expression like \"A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)\" for ORFs.     \n  2. Degenerate bases/residues like \"RYMM..\" are also supported by flag -d.\n     But do not use degenerate bases/residues in regular expression, you need\n     convert them to regular expression, e.g., change \"N\" or \"X\"  to \".\".\n  3. When providing search patterns (motifs) via flag '-p',\n     please use double quotation marks for patterns containing comma, \n     e.g., -p '\"A{2,}\"' or -p \"\\\"A{2,}\\\"\". Because the command line argument\n     parser accepts comma-separated-values (CSV) for multiple values (motifs).\n     Patterns in file do not follow this rule.     \n  4. Mismatch is allowed using flag \"-m/--max-mismatch\",\n     you can increase the value of \"-j/--threads\" to accelerate processing.\n  5. When using flag --circular, end position of matched subsequence that \n     crossing genome sequence end would be greater than sequence length.\n\nUsage:\n  seqkit locate [flags]\n\nFlags:\n      --bed                    output in BED6 format\n  -c, --circular               circular genome. type \"seqkit locate -h\" for details\n  -d, --degenerate             pattern/motif contains degenerate base\n      --gtf                    output in GTF format\n  -h, --help                   help for locate\n  -M, --hide-matched           do not show matched sequences\n  -i, --ignore-case            ignore case\n  -I, --immediate-output       print output immediately, do not use write buffer\n  -s, --max-len-to-show int    show at most X characters for the search pattern or matched sequences\n  -m, --max-mismatch int       max mismatch when matching by seq. For large genomes like human genome,\n                               using mapping/alignment tools would be faster\n  -G, --non-greedy             non-greedy mode, faster but may miss motifs overlapping with others\n  -P, --only-positive-strand   only search on positive strand\n  -p, --pattern strings        pattern/motif (multiple values supported. Attention: use double quotation\n                               marks for patterns containing comma, e.g., -p '\"A{2,}\"')\n  -f, --pattern-file string    pattern/motif file (FASTA format)\n  -F, --use-fmi                use FM-index for much faster search of lots of sequence patterns\n  -r, --use-regexp             patterns/motifs are regular expression\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Locating subsequences (mismatch allowed)</p> <pre><code>$ cat t.fa\n&gt;seq\nagctggagctacc\n\n$ cat t.fa \\\n  | seqkit locate -p agc \\\n  | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end   matched\nseq     agc           agc       +        1       3     agc\nseq     agc           agc       +        7       9     agc\nseq     agc           agc       -        8       10    agc\nseq     agc           agc       -        2       4     agc\n\n# do not show matched sequences\n$ cat t.fa \\\n  | seqkit locate -p agc -M \\\n  | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end\nseq     agc           agc       +        1       3\nseq     agc           agc       +        7       9\nseq     agc           agc       -        8       10\nseq     agc           agc       -        2       4\n\n# max mismatch: 1\n$ cat t.fa \\\n  | seqkit locate -p agc -m 1 \\\n  | csvtk pretty -t\nseqID   patternName     pattern strand  start   end    matched\nseq     agc           agc       +        1       3     agc\nseq     agc           agc       +        7       9     agc\nseq     agc           agc       +        11      13    acc\nseq     agc           agc       -        8       10    agc\nseq     agc           agc       -        2       4     agc\n\n# max mismatch: 2\n$ cat t.fa \\\n  | seqkit locate -p agc -m 2 \\\n  | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end   matched\nseq     agc           agc       +        1       3     agc\nseq     agc           agc       +        4       6     tgg\nseq     agc           agc       +        5       7     gga\nseq     agc           agc       +        7       9     agc\nseq     agc           agc       +        10      12    tac\nseq     agc           agc       +        11      13    acc\nseq     agc           agc       -        11      13    ggt\nseq     agc           agc       -        8       10    agc\nseq     agc           agc       -        6       8     ctc\nseq     agc           agc       -        5       7     tcc\nseq     agc           agc       -        2       4     agc\n</code></pre> </li> <li> <p>Locate ORFs.</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit locate -i -p \"A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)\" -r \\\n    | head -n 4 \\\n    | csvtk pretty -t\nseqID       patternName                        pattern                            strand   start   end   matched\ncel-lin-4   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   +        1       36    AUGCUUCCGGCCUGUUCCCUGAGACCUCAAGUGUGA\ncel-mir-1   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   +        54      95    AUGGAUAUGGAAUGUAAAGAAGUAUGUAGAACGGGGUGGUAG\ncel-mir-1   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   -        43      51    AUGAUAUAG\n</code></pre> </li> <li> <p>Locate Motif.</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit locate -i -d -p AUGGACUN \\\n    | head -n 4 \\\n    | csvtk pretty -t \nseqID         patternName   pattern    strand   start   end   matched\ncel-mir-58a   AUGGACUN      AUGGACUN   +        81      88    AUGGACUG\nath-MIR163    AUGGACUN      AUGGACUN   -        122     129   AUGGACUC\ncel-mir-270   AUGGACUN      AUGGACUN   +        84      91    AUGGACUG\n</code></pre> </li> <li> <p>Output in <code>GTF</code> or <code>BED6</code> format, which you can use in <code>seqkit subseq</code></p> <pre><code>$ zcat hairpin.fa.gz | seqkit locate -i -d -p AUGGACUN --bed\ncel-mir-58a     80      88      AUGGACUN        0       +\nath-MIR163      121     129     AUGGACUN        0       -\n\n$ zcat hairpin.fa.gz | seqkit locate -i -d -p AUGGACUN --gtf\ncel-mir-58a     SeqKit  location        81      88      0       +       .       gene_id \"AUGGACUN\";\nath-MIR163      SeqKit  location        122     129     0       -       .       gene_id \"AUGGACUN\";\n</code></pre> </li> <li> <p>Greedy mode (default)</p> <pre><code> $ echo -e '&gt;seq\\nACGACGACGA' | seqkit locate -p ACGA | csvtk -t pretty\n seqID   patternName   pattern   strand   start   end   matched\n seq     ACGA          ACGA      +        1       4     ACGA\n seq     ACGA          ACGA      +        4       7     ACGA\n seq     ACGA          ACGA      +        7       10    ACGA\n</code></pre> </li> <li> <p>Non-greedy mode (<code>-G</code>)</p> <pre><code>$ echo -e '&gt;seq\\nACGACGACGA' | seqkit locate -p ACGA -G | csvtk -t pretty\nseqID   patternName   pattern   strand   start   end   matched\nseq     ACGA          ACGA      +        1       4     ACGA\nseq     ACGA          ACGA      +        7       10    ACGA\n</code></pre> </li> <li> <p>Circular genome. Note that end position of matched subsequence that  crossing genome sequence end would be greater than sequence length.</p> <pre><code>$ echo -e \"&gt;seq\\nACGTTGCA\"\n&gt;seq\nACGTTGCA\n\n$ echo -e \"&gt;seq\\nACGTTGCA\" \\\n    | seqkit locate -i -p aa\nseqID   patternName     pattern strand  start   end     matched\nseq     aa      aa      -       4       5       aa\n\n$ echo -e \"&gt;seq\\nACGTTGCA\" \\\n    | seqkit locate -i -p aa -c \\\n    | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end   matched\nseq     aa            aa        +        8       9     aa\nseq     aa            aa        -        4       5     aa\n</code></pre> </li> </ol>"},{"location":"usage/#fish","title":"fish","text":"<p>Usage</p> <pre><code>look for short sequences in larger sequences using local alignment\n\nAttention:\n  1. output coordinates are BED-like 0-based, left-close and right-open.\n  2. alignment information are printed to STDERR.\n\nUsage:\n  seqkit fish [flags]\n\nFlags:\n  -a, --all                      search all\n  -p, --aln-params string        alignment parameters in format\n                                 \"&lt;match&gt;,&lt;mismatch&gt;,&lt;gap_open&gt;,&lt;gap_extend&gt;\" (default \"4,-4,-2,-1\")\n  -h, --help                     help for fish\n  -i, --invert                   print out references not matching with any query\n  -q, --min-qual float           minimum mapping quality (default 5)\n  -b, --out-bam string           save aligmnets to this BAM file (memory intensive)\n  -x, --pass                     pass through mode (write input to stdout)\n  -g, --print-aln                print sequence alignments\n  -D, --print-desc               print full sequence header\n  -f, --query-fastx string       query fasta\n  -F, --query-sequences string   query sequences\n  -r, --ranges string            target ranges, for example: \":10,30:40,-20:\"\n  -s, --stranded                 search + strand only\n  -v, --validate-seq             validate bases according to the alphabet\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Find best local alignment of a short sequence in reads in a fasta file, print results as tabular</p> <pre><code>$ seqkit fish -q 4.7 -F GGCGGCTGTGACC -g mouse-p53-cds.fna\n</code></pre> </li> <li> <p>Compare to <code>seqkit locate</code>:</p> <pre><code>$ echo -e '&gt;seq\\nACGACGACGA' \\\n    | seqkit locate -p ACGA -G | csvtk -t pretty\nseqID   patternName   pattern   strand   start   end   matched\nseq     ACGA          ACGA      +        1       4     ACGA\nseq     ACGA          ACGA      +        7       10    ACGA\n\n$ echo -e '&gt;seq\\nACGACGACGA' \\\n    | seqkit fish -F ACGA -a 2&gt;&amp;1 | csvtk -t pretty \nRef   RefStart   RefEnd   Query   QueryStart   QueryEnd   Strand   MapQual   RawScore   Acc      ClipAcc   QueryCov\nseq   6          10       q0      0            4          +        60.00     16         100.00   100.00    100.00\nseq   0          4        q0      0            4          +        60.00     16         100.00   100.00    100.00\n</code></pre> </li> <li> <p>Find all local alignment of a short sequences in reads in a fasta file, print results as tabular and save as BAM</p> <pre><code>seqkit fish -a -q 4.67 -f query.fas -b alignments.bam -g mouse-p53-cds.fna\n</code></pre> </li> </ol>"},{"location":"usage/#amplicon","title":"amplicon","text":"<p>Usage</p> <pre><code>extract amplicon (or specific region around it) via primer(s).\n\nAttention:\n  1. Only one (the longest) matching location is returned for every primer pair.\n  2. Mismatch is allowed, but the mismatch location (5' or 3') is not controled. \n     You can increase the value of \"-j/--threads\" to accelerate processing.\n  3. Degenerate bases/residues like \"RYMM..\" are also supported.\n     But do not use degenerate bases/residues in regular expression, you need\n     convert them to regular expression, e.g., change \"N\" or \"X\"  to \".\".\n\nExamples:\n  0. no region given.\n\n                    F\n        -----===============-----\n             F             R\n        -----=====-----=====-----\n\n             ===============         amplicon\n\n  1. inner region (-r x:y).\n\n                    F\n        -----===============-----\n             1 3 5                    x/y\n                      -5-3-1          x/y\n             F             R\n        -----=====-----=====-----     x:y\n\n             ===============          1:-1\n             =======                  1:7\n               =====                  3:7\n                  =====               6:10\n                  =====             -10:-6\n                     =====           -7:-3\n                                     -x:y (invalid)\n\n  2. flanking region (-r x:y -f)\n\n                    F\n        -----===============-----\n         -3-1                        x/y\n                            1 3 5    x/y\n             F             R\n        -----=====-----=====-----\n\n        =====                        -5:-1\n        ===                          -5:-3\n                            =====     1:5\n                              ===     3:5\n            =================        -1:1\n        =========================    -5:5\n                                      x:-y (invalid)\n\nUsage:\n  seqkit amplicon [flags]\n\nFlags:\n      --bed                    output in BED6+1 format with amplicon as the 7th column\n  -f, --flanking-region        region is flanking region\n  -F, --forward string         forward primer (5'-primer-3'), degenerate bases allowed\n  -h, --help                   help for amplicon\n  -I, --immediate-output       print output immediately, do not use write buffer\n  -m, --max-mismatch int       max mismatch when matching primers, no degenerate bases allowed\n  -P, --only-positive-strand   only search on positive strand\n  -M, --output-mismatches      append the total mismatches and mismatches of 5' end and 3' end\n  -p, --primer-file string     3- or 2-column tabular primer file, with first column as primer name\n  -r, --region string          specify region to return. type \"seqkit amplicon -h\" for detail\n  -R, --reverse string         reverse primer (5'-primer-3'), degenerate bases allowed\n  -u, --save-unmatched         also save records that do not match any primer\n  -s, --strict-mode            strict mode, i.e., discarding seqs not fully matching (shorter) given\n                               region range\n\n</code></pre> <p>Examples</p> <ol> <li> <p>No region given.</p> <pre><code>$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \n&gt;seq\nacgcccactgaaatga\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt\n&gt;seq\ncccactgaaa\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt\n\n# BED6+1\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt --bed\nseq     3       13      .       0       +       cccactgaaa\n\n# supporting degenerate bases.\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccR -R ttt --bed\nseq     4       13      .       0       +       ccactgaaa\n</code></pre> </li> <li> <p>Output mismatches:</p> <pre><code>$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -M\n&gt;seq mismatches=0(0+0)\ncccactgaaa\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -m 1 -M\n&gt;seq mismatches=2(1+1)\ncgcccactgaaat\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -m 1 -M --bed\nseq     1       14      .       0       +       cgcccactgaaat   2       1       1\n</code></pre> </li> <li> <p>Load primers from 3- or 2-column tabular primer file, with first column as primer name.</p> <pre><code>$ cat seqs4amplicon.fa \n&gt;seq1\nACGCCCACTGAAATGA\n&gt;seq2\nACGTACGGTCAGATCCA\n\n$ cat primers.tsv \np1      ccc     ttt\np2      ttt     ccc\np3      ttt\n\np4      CG      TG\nP5      CG      GA\n\n# containing degenerate bases\np6      TRC     WGG\n\n$ cat seqs4amplicon.fa | seqkit amplicon -p primers.tsv --bed\nseq1    3       13      p1      0       +       CCCACTGAAA\nseq1    1       7       p4      0       +       CGCCCA\nseq1    3       13      p2      0       -       TTTCAGTGGG\nseq1    10      13      p3      0       -       TTT\nseq2    1       17      p4      0       +       CGTACGGTCAGATCCA\nseq2    1       15      P5      0       +       CGTACGGTCAGATC\nseq2    3       17      p6      0       +       TACGGTCAGATCCA\n</code></pre> </li> <li> <p>Inner region</p> <pre><code># region right behind forward primer\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -r 4:7\n&gt;seq\nactg\n\n# BED\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -r 4:7 --bed\nseq     6       10      .       0       +       actg\n\n# more common case is triming primers\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -r 4:-4\n&gt;seq\nactg\n</code></pre> </li> <li> <p>flanking region</p> <pre><code># in one of my sequencing data, I only care about \n# region downstream of forward primer\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -f -r 3:6\n&gt;seq\ntgaa\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -f -r 3:6 --bed\nseq     8       12      .       0       +       tgaa\n\n# if given region if out scope of sequence. e.g,\n# 2-5bp downstream of aaa, we can get part of region (2-4) by default\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F aaa -f -r 2:5\n&gt;seq\nga\n\n# you can also use strict mode to discard those cases\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F aaa -f -r 2:5 -s\n</code></pre> </li> </ol>"},{"location":"usage/#duplicate","title":"duplicate","text":"<p>Usage</p> <pre><code>duplicate sequences N times\n\nYou may need \"seqkit rename\" to make the the sequence IDs unique.\n\nUsage:\n  seqkit duplicate [flags]\n\nAliases:\n  duplicate, dup\n\nFlags:\n  -h, --help        help for duplicate\n  -n, --times int   duplication number (default 1)\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Data</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 1\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> <li> <p>Duplicate 2 times</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 1 \\\n    | seqkit duplicate -n 2\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> <li> <p>use <code>seqkit rename</code> to make the the sequence IDs unique.</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 1 \\\n    | seqkit duplicate -n 2 | seqkit rename\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n&gt;cel-let-7_2 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> </ol>"},{"location":"usage/#rmdup","title":"rmdup","text":"<p>Usage</p> <pre><code>remove duplicated sequences by ID/name/sequence\n\nAttention:\n  1. When comparing by sequences, both positive and negative strands are\n     compared. Switch on -P/--only-positive-strand for considering the\n     positive strand only.\n  2. Only the first record is saved for duplicates.\n\nUsage:\n  seqkit rmdup [flags]\n\nFlags:\n  -n, --by-name                by full name instead of just id\n  -s, --by-seq                 by seq\n  -D, --dup-num-file string    file to save numbers and ID lists of duplicated seqs\n  -d, --dup-seqs-file string   file to save duplicated seqs\n  -h, --help                   help for rmdup\n  -i, --ignore-case            ignore case\n  -P, --only-positive-strand   only considering positive strand when comparing by sequence\n\n</code></pre> <p>Examples</p> <p>Similar to <code>common</code>.</p> <ol> <li> <p>General use</p> <pre><code>$ zcat hairpin.fa.gz | seqkit rmdup -s -o clean.fa.gz\n[INFO] 2226 duplicated records removed\n\n$ zcat reads_1.fq.gz | seqkit rmdup -s -o clean.fa.gz\n[INFO] 1086 duplicated records removed\n</code></pre> </li> <li> <p>Save duplicated sequences to file</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit rmdup -s -i -o clean.fa.gz -d duplicated.fa.gz -D duplicated.detail.txt\n\n$ cat duplicated.detail.txt   # here is not the entire list\n3   hsa-mir-424, mml-mir-424, ppy-mir-424\n3   hsa-mir-342, mml-mir-342, ppy-mir-342\n2   ngi-mir-932, nlo-mir-932\n2   ssc-mir-9784-1, ssc-mir-9784-2\n</code></pre> </li> </ol>"},{"location":"usage/#common","title":"common","text":"<p>Usage</p> <pre><code>find common sequences of multiple files by id/name/sequence\n\nNote:\n  1. 'seqkit common' is designed to support 2 and MORE files.\n  2. When comparing by sequences,\n     a) Both positive and negative strands are compared. You can switch on\n        -P/--only-positive-strand for considering the positive strand only.\n     b) You can switch on -e/--check-embedded-seqs to check embedded sequences.\n          e.g, for file A and B, the reverse complement sequence of CCCC from file B\n          is a part of TTGGGGTT from file A, we will extract and output GGGG from file A.\n          If sequences CCC exist in other files except file A, we will skip it,\n          as it is an embedded subsequence of GGGG.\n        It is recommended to put the smallest file as the first file, for saving\n        memory usage.\n  3. For 2 files, 'seqkit grep' is much faster and consumes lesser memory:\n       seqkit grep -f &lt;(seqkit seq -n -i small.fq.gz) big.fq.gz # by seq ID\n     But note that searching by sequence would be much slower, as it's partly\n     string matching.\n       seqkit grep -s -f &lt;(seqkit seq -s small.fq.gz) big.fq.gz # much slower!!!!\n  4. Some records in one file may have same sequences/IDs. They will ALL be\n     retrieved if the sequence/ID was shared in multiple files.\n     So the records number may be larger than that of the smallest file.\n\nUsage:\n  seqkit common [flags]\n\nFlags:\n  -n, --by-name                match by full name instead of just id\n  -s, --by-seq                 match by sequence\n  -e, --check-embedded-seqs    check embedded sequences, e.g., if a sequence is part of another one,\n                               we'll keep the shorter one\n  -h, --help                   help for common\n  -i, --ignore-case            ignore case\n  -P, --only-positive-strand   only considering the positive strand when comparing by sequence\n\n</code></pre> <p>Examples</p> <ol> <li> <p>By ID (default)</p> <pre><code>seqkit common file*.fa -o common.fasta\n</code></pre> </li> <li> <p>By full name</p> <pre><code>seqkit common file*.fa -n -o common.fasta\n</code></pre> </li> <li> <p>By sequence</p> <pre><code>seqkit common file*.fa -s -i -o common.fasta\n\n# seqkit common file*.fa -s -i -o common.fasta\nseqkit common file*.fa -s -i -o common.fasta -P\n</code></pre> </li> <li> <p>Since v2.6.0, we can use the new flag <code>-e/--check-embedded-seqs</code> to detect embedded sequences.</p> <pre><code>$ cat common_a.fasta\n&gt;A_a\nACTTTGA\n&gt;A_b\nACC\n&gt;A_c\nGGG\n&gt;A_d\nCCCCCCCCCCCCC\n&gt;A_c1\nAGGGCCC\n&gt;A_x\ngggg\n&gt;A_y\nACGTAC\n\n$ cat common_b.fasta\n&gt;B_a\nAAA\n&gt;B_b\nGGT\n&gt;B_c\nAGGGCC\n&gt;B_e\nACCCCCGGGAACC\n&gt;B_z\nTTT\n\n$ cat common_c.fasta\n&gt;C_a\nACTTTGA\n&gt;C_b\nACC\n&gt;C_c\nGGG\n&gt;C_d\nCCCCCCCCCCCCC\n&gt;C_c1\nAGGGCCC\n&gt;C_x\ngggg\n&gt;C_y\nACGTAC\n\n# excactly same sequences\n\n$ seqkit common -s -i common_a.fasta common_b.fasta common_c.fasta\n[INFO] read file 1/3: common_a.fasta\n[INFO] read file 2/3: common_b.fasta\n[INFO] read file 3/3: common_c.fasta\n[INFO] find common seqs ...\n[INFO] 1 unique sequences found in 3 files, which belong to 1 records in the first file: common_a.fasta\n[INFO] retrieve 1 seqs from the first file: common_a.fasta\n&gt;A_b\nACC\n\n# with -e\n\n$ seqkit common -s -i common_a.fasta common_b.fasta common_c.fasta -e\n[INFO] read file 1/3: common_a.fasta\n[INFO]   7 seqs loaded\n[INFO] read file 2/3: common_b.fasta\n[INFO]   5 seqs left\n[INFO] read file 3/3: common_c.fasta\n[INFO]   5 seqs left\n[INFO] 5 unique sequences found in 3 files, which belong to 5 records in the first file: common_a.fasta\n[INFO] 5 common/shared sequences saved to: -\n&gt;A_a:3-5\nTTT\n&gt;A_b\nACC\n&gt;A_c\nGGG\n&gt;A_c1:1-6\nAGGGCC\n&gt;A_x\ngggg\n\n# change the order of file, the sequences are the same\n\n$ seqkit common -s -i common_c.fasta common_b.fasta common_a.fasta -e\n[INFO] read file 1/3: common_c.fasta\n[INFO]   7 seqs loaded\n[INFO] read file 2/3: common_b.fasta\n[INFO]   5 seqs left\n[INFO] read file 3/3: common_a.fasta\n[INFO]   5 seqs left\n[INFO] 5 unique sequences found in 3 files, which belong to 5 records in the first file: common_c.fasta\n[INFO] 5 common/shared sequences saved to: -\n&gt;C_a:3-5\nTTT\n&gt;C_b\nACC\n&gt;C_c\nGGG\n&gt;C_c1:1-6\nAGGGCC\n&gt;C_x\ngggg\n</code></pre> </li> </ol>"},{"location":"usage/#split","title":"split","text":"<p>Usage</p> <pre><code>split sequences into files by name ID, subsequence of given region,\npart size or number of parts.\n\nIf you just want to split by parts or sizes, please use \"seqkit split2\",\nwhich can apply to paired- and single-end FASTQ.\n\nIf you want to cut a sequence into multiple segments.\n  1. For cutting into even chunks, please use 'kmcp utils split-genomes'\n     (https://bioinf.shenwei.me/kmcp/usage/#split-genomes).\n     E.g., cutting into 4 segments of equal size, with no overlap between adjacent segments:\n        kmcp utils split-genomes -m 1 -k 1 --split-number 4 --split-overlap 0 input.fasta -O out\n  2. For cutting into multiple chunks of fixed size, please using 'seqkit sliding'.\n     E.g., cutting into segments of 40 bp and keeping the last segment which can be shorter than 40 bp.\n        seqkit sliding -g -s 40 -W 40 input.fasta -o out.fasta\n\nAttention:\n  1. For the two-pass mode (-2/--two-pass), The flag -U/--update-faidx is recommended to\n     ensure the .fai file matches the FASTA file.\n  2. For splitting by sequence IDs in Windows/MacOS, where the file systems might be case-insensitive,\n     output files might be overwritten if they are only different in cases, like Abc and ABC.\n     To avoid this, please switch one -I/--ignore-case.\n\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit split [flags]\n\nFlags:\n  -i, --by-id                     split squences according to sequence ID\n      --by-id-prefix string       file prefix for --by-id\n  -p, --by-part int               split sequences into N parts\n      --by-part-prefix string     file prefix for --by-part\n  -r, --by-region string          split squences according to subsequence of given region. e.g 1:12 for\n                                  first 12 bases, -12:-1 for last 12 bases. type \"seqkit split -h\" for\n                                  more examples\n      --by-region-prefix string   file prefix for --by-region\n  -s, --by-size int               split sequences into multi parts with N sequences\n      --by-size-prefix string     file prefix for --by-size\n  -d, --dry-run                   dry run, just print message and no files will be created.\n  -e, --extension string          set output file extension, e.g., \".gz\", \".xz\", or \".zst\"\n  -f, --force                     overwrite output directory\n  -h, --help                      help for split\n      --ignore-case               ignore case when using -i/--by-id\n  -k, --keep-temp                 keep temporary FASTA and .fai file when using 2-pass mode\n  -O, --out-dir string            output directory (default value is $infile.split)\n  -2, --two-pass                  two-pass mode read files twice to lower memory usage. (only for FASTA\n                                  format)\n  -U, --update-faidx              update the fasta index file if it exists. Use this if you are not sure\n                                  whether the fasta file changed\n</code></pre> <p>Examples</p> <ol> <li> <p>Split sequences into parts with at most 10000 sequences</p> <pre><code>$ seqkit split hairpin.fa.gz -s 10000\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: hairpin.fa.part_001.gz\n[INFO] write 10000 sequences to file: hairpin.fa.part_002.gz\n[INFO] write 8645 sequences to file: hairpin.fa.part_003.gz\n</code></pre> </li> <li> <p>Split sequences into 4 parts</p> <pre><code>$ seqkit split hairpin.fa.gz -p 4\n[INFO] split into 4 parts\n[INFO] read sequences ...\n[INFO] read 28645 sequences\n[INFO] write 7162 sequences to file: hairpin.fa.part_001.gz\n[INFO] write 7162 sequences to file: hairpin.fa.part_002.gz\n[INFO] write 7162 sequences to file: hairpin.fa.part_003.gz\n[INFO] write 7159 sequences to file: hairpin.fa.part_004.gz\n</code></pre> <p>To reduce memory usage when spliting big file, we should alwasy use flag <code>--two-pass</code></p> <pre><code>$ seqkit split hairpin.fa.gz -p 4 -2\n[INFO] split into 4 parts\n[INFO] read and write sequences to temporary file: hairpin.fa.gz.fastx ...\n[INFO] 28645 sequences saved\n[INFO] create or read FASTA index ...\n[INFO] create FASTA index for hairpin.fa.gz.fastx\n[INFO]   28645 records loaded from hairpin.fa.gz.fastx.seqkit.fai\n[INFO] write 7162 sequences to file: hairpin.part_001.fa.gz\n[INFO] write 7162 sequences to file: hairpin.part_002.fa.gz\n[INFO] write 7162 sequences to file: hairpin.part_003.fa.gz\n[INFO] write 7159 sequences to file: hairpin.part_004.fa.gz\n</code></pre> </li> <li> <p>Split sequences by species. i.e. by custom IDs (first three letters)</p> <pre><code>$ seqkit split hairpin.fa.gz -i --id-regexp \"^([\\w]+)\\-\" -2\n[INFO] split by ID. idRegexp: ^([\\w]+)\\-\n[INFO] read and write sequences to temporary file: hairpin.fa.gz.fastx ...\n[INFO] 28645 sequences saved\n[INFO] create or read FASTA index ...\n[INFO] create FASTA index for hairpin.fa.gz.fastx\n[INFO]   28645 records loaded from hairpin.fa.gz.fastx.seqkit.fai\n[INFO] write 48 sequences to file: hairpin.id_cca.fa.gz\n[INFO] write 3 sequences to file: hairpin.id_hci.fa.gz\n[INFO] write 106 sequences to file: hairpin.id_str.fa.gz\n[INFO] write 1 sequences to file: hairpin.id_bkv.fa.gz\n...\n</code></pre> </li> <li> <p>Split sequences by sequence region (for example, sequence barcode)</p> <pre><code>$ seqkit split hairpin.fa.gz -r 1:3 -2\n[INFO] split by region: 1:3\n[INFO] read and write sequences to temporary file: hairpin.fa.gz.fastx ...\n[INFO] 28645 sequences saved\n[INFO] read sequence IDs and sequence region from FASTA file ...\n[INFO] create or read FASTA index ...\n[INFO] create FASTA index for hairpin.fa.gz.fastx\n[INFO]   28645 records loaded from hairpin.fa.gz.fastx.seqkit.fai\n[INFO] write 463 sequences to file: hairpin.region_1:3_AUG.fa.gz\n[INFO] write 349 sequences to file: hairpin.region_1:3_ACU.fa.gz\n[INFO] write 311 sequences to file: hairpin.region_1:3_CGG.fa.gz\n</code></pre> <p>Sequence suffix could be defined as <code>-r -12:-1</code></p> </li> </ol>"},{"location":"usage/#split2","title":"split2","text":"<p>Usage</p> <pre><code>split sequences into files by part size or number of parts\n\nThis command supports FASTA and paired- or single-end FASTQ with low memory\noccupation and fast speed.\n\nThe prefix of output files:\n  1. For stdin: stdin\n  2. Others: same to the input file\n  3. Set via the options: --by-length-prefix, --by-part-prefix, or --by-size-prefix\n\nThe extension of output files:\n  1. For stdin: .fast[aq]\n  2. Others: same to the input file\n  3. Additional extension via the option -e/--extension, e.g.\uff0c outputting\n     gzipped files for plain text input:\n         seqkit split2 -p 2 -O test tests/hairpin.fa -e .gz\n\n\nIf you want to cut a sequence into multiple segments.\n  1. For cutting into even chunks, please use 'kmcp utils split-genomes'\n     (https://bioinf.shenwei.me/kmcp/usage/#split-genomes).\n     E.g., cutting into 4 segments of equal size, with no overlap between adjacent segments:\n        kmcp utils split-genomes -m 1 -k 1 --split-number 4 --split-overlap 0 input.fasta -O out\n  2. For cutting into multiple chunks of fixed size, please using 'seqkit sliding'.\n     E.g., cutting into segments of 40 bp and keeping the last segment which can be shorter than 40 bp.\n        seqkit sliding -g -s 40 -W 40 input.fasta -o out.fasta\n\nUsage:\n  seqkit split2 [flags]\n\nFlags:\n  -l, --by-length string          split sequences into chunks of &gt;=N bases, supports K/M/G suffix\n      --by-length-prefix string   file prefix for --by-length\n  -p, --by-part int               split sequences into N parts\n      --by-part-prefix string     file prefix for --by-part\n  -s, --by-size int               split sequences into multi parts with N sequences\n      --by-size-prefix string     file prefix for --by-size\n  -e, --extension string          set output file extension, e.g., \".gz\", \".xz\", or \".zst\"\n  -f, --force                     overwrite output directory\n  -h, --help                      help for split2\n  -O, --out-dir string            output directory (default value is $infile.split)\n  -1, --read1 string              (gzipped) read1 file\n  -2, --read2 string              (gzipped) read2 file\n</code></pre> <p>Examples</p> <ol> <li> <p>Split sequences into parts with at most 10000 sequences:</p> <pre><code>$ seqkit split2 hairpin.fa -s 10000\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: hairpin.fa.split/hairpin.part_001.fa\n[INFO] write 10000 sequences to file: hairpin.fa.split/hairpin.part_002.fa\n[INFO] write 8645 sequences to file: hairpin.fa.split/hairpin.part_003.fa\n</code></pre> </li> <li> <p>Force compression for plain text input by adding an extra extension:</p> <pre><code># gzip\n$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .gz\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/hairpin.part_001.fa.gz\n[INFO] write 10000 sequences to file: test/hairpin.part_002.fa.gz\n[INFO] write 8645 sequences to file: test/hairpin.part_003.fa.gz\n\n# xz\n$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .xz\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/hairpin.part_001.fa.xz\n[INFO] write 10000 sequences to file: test/hairpin.part_002.fa.xz\n[INFO] write 8645 sequences to file: test/hairpin.part_003.fa.xz\n\n# zstd\n$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .zst\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/hairpin.part_001.fa.zst\n[INFO] write 10000 sequences to file: test/hairpin.part_002.fa.zst\n[INFO] write 8645 sequences to file: test/hairpin.part_003.fa.zst\n</code></pre> </li> <li> <p>Change the prefix of output files:</p> <pre><code>$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .gz -o xxx\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/xxx.part_001.fa.gz\n[INFO] write 10000 sequences to file: test/xxx.part_002.fa.gz\n[INFO] write 8645 sequences to file: test/xxx.part_003.fa.gz\n\n# here, we also change the compression format from xz to zstd\n$ cat hairpin.fa.xz | seqkit split2 -O test -f -s 10000 -e .zst\n[INFO] split seqs from stdin\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/stdin.part_001.fasta.zst\n[INFO] write 10000 sequences to file: test/stdin.part_002.fasta.zst\n[INFO] write 8645 sequences to file: test/stdin.part_003.fasta.zst\n</code></pre> </li> <li> <p>Split sequences into 4 parts</p> <pre><code>$ seqkit split hairpin.fa.gz -p 4 -f\n[INFO] split into 4 parts\n[INFO] read sequences ...\n[INFO] read 28645 sequences\n[INFO] write 7162 sequences to file: hairpin.fa.gz.split/hairpin.part_001.fa.gz\n[INFO] write 7162 sequences to file: hairpin.fa.gz.split/hairpin.part_002.fa.gz\n[INFO] write 7162 sequences to file: hairpin.fa.gz.split/hairpin.part_003.fa.gz\n[INFO] write 7159 sequences to file: hairpin.fa.gz.split/hairpin.part_004.fa.gz\n</code></pre> </li> <li> <p>For FASTQ files (paired-end)</p> <pre><code>$ seqkit split2 -1 reads_1.fq.gz -2 reads_2.fq.gz -p 2 -O out -f\n[INFO] split seqs from reads_1.fq.gz and reads_2.fq.gz\n[INFO] split into 2 parts\n[INFO] write 1250 sequences to file: out/reads_2.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_2.part_002.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_002.fq.gz\n</code></pre> </li> <li> <p>For FASTA files (single-end)</p> <pre><code>$ seqkit split2 -1 reads_1.fq.gz reads_2.fq.gz -p 2 -O out -f\n[INFO] flag -1/--read1 given, ignore: reads_2.fq.gz\n[INFO] split seqs from reads_1.fq.gz\n[INFO] split into 2 parts\n[INFO] write 1250 sequences to file: out/reads_1.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_002.fq.gz\n\n$ seqkit split2 reads_1.fq.gz -p 2 -O out -f\n[INFO] split seqs from reads_1.fq.gz\n[INFO] split into 2 parts\n[INFO] write 1250 sequences to file: out/reads_1.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_002.fq.gz\n</code></pre> </li> </ol>"},{"location":"usage/#pair","title":"pair","text":"<p>Usage</p> <pre><code>match up paired-end reads from two fastq files\n\nAttention:\n1. Orders of headers in the two files should be the same (not shuffled),\n   otherwise, it consumes a huge number of memory for buffering reads in memory.\n2. Unpaired reads are optional outputted with the flag -u/--save-unpaired.\n3. If the flag -O/--out-dir is not given, the output will be saved in the same directory\n   of input, with the suffix \"paired\", e.g., read_1.paired.fq.gz.\n   Otherwise, names are kept untouched in the given output directory.\n4. Paired gzipped files may be slightly larger than original files, because\n   of using a different gzip package/library, don't worry.\n\nTips:\n1. Support for '/1 'and '/2' tags for paired read files generated by platforms like MGI.\n   You can simply specify the regular expression for extracting sequence IDs:\n     --id-regexp '^(\\S+)\\/[12]'\n\nUsage:\n  seqkit pair [flags]\n\nFlags:\n  -f, --force            overwrite output directory\n  -h, --help             help for pair\n  -O, --out-dir string   output directory\n  -1, --read1 string     (gzipped) read1 file\n  -2, --read2 string     (gzipped) read2 file\n  -u, --save-unpaired    save unpaired reads if there are\n</code></pre> <p>Examples</p> <ol> <li> <p>Simple one</p> <pre><code>$ seqkit pair -1 reads_1.fq.gz -2 reads_2.fq.gz\n\n# output\nreads_1.paired.fq.gz\nreads_2.paired.fq.gz\n</code></pre> </li> <li> <p>Set output directory, file names are kept untouched.</p> <pre><code>$ seqkit pair -1 reads_1.fq.gz -2 reads_2.fq.gz -O result\n\n$ tree result        \nresult/\n\u251c\u2500\u2500 reads_1.fq.gz\n\u2514\u2500\u2500 reads_2.fq.gz\n</code></pre> </li> <li> <p>Save unpaired reads if there are.</p> <pre><code>$ seqkit pair -1 reads_1.fq.gz -2 reads_2.fq.gz -O result -u\n\n$ tree result\nresult\n\u251c\u2500\u2500 reads_1.fq.gz\n\u251c\u2500\u2500 reads_1.unpaired.fq.gz\n\u251c\u2500\u2500 reads_2.fq.gz\n\u2514\u2500\u2500 reads_2.unpaired.fq.gz\n</code></pre> </li> <li> <p>Support for '/1 'and '/2' tags for paired read files generated by platforms like MGI.    You can simply specify the regular expression for extracting sequence IDs:</p> <pre><code>seqkit pair --id-regexp '^(\\S+)\\/[12]' -1 reads_1.fq.gz -2 reads_2.fq.gz\n</code></pre> </li> </ol>"},{"location":"usage/#sample","title":"sample","text":"<p>Usage</p> <pre><code>sample sequences by number or proportion.\n\nAttention:\n1. Do not use '-n' on large FASTQ files, it loads all seqs into memory!\n   use 'seqkit sample -p 0.1 seqs.fq.gz | seqkit head -n N' instead!\n\nUsage:\n  seqkit sample [flags]\n\nFlags:\n  -h, --help               help for sample\n  -n, --number int         sample by number (result may not exactly match), DO NOT use on large FASTQ files.\n  -p, --proportion float   sample by proportion\n  -s, --rand-seed int      random seed. For paired-end data, use the same seed across fastq files to\n                           sample the same read pairs (default 11)\n  -2, --two-pass           2-pass mode read files twice to lower memory usage. Not allowed when reading\n                           from stdin\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Sample by proportion</p> <pre><code>$ zcat hairpin.fa.gz | seqkit sample -p 0.1 -o sample.fa.gz\n[INFO] sample by proportion\n[INFO] 2814 sequences outputed\n</code></pre> </li> <li> <p>Sample by number</p> <pre><code>$ zcat hairpin.fa.gz | seqkit sample -n 1000 -o sample.fa.gz\n[INFO] sample by number\n[INFO] 949 sequences outputed\n</code></pre> <p>949 != 1000 ??? see Effect of random seed on results of <code>seqkit sample</code></p> <p>To reduce memory usage when spliting big file, we could use flag <code>--two-pass</code></p> <p>We can also use <code>seqkit sample -p</code> followed with <code>seqkit head -n</code>:</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sample -p 0.1 \\\n    | seqkit head -n 1000 -o sample.fa.gz\n</code></pre> </li> <li> <p>Set rand seed to reproduce the result</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sample -p 0.1 -s 11\n</code></pre> </li> <li> <p>Most of the time, we could shuffle after sampling</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sample -p 0.1 \\\n    | seqkit shuffle -o sample.fa.gz\n</code></pre> </li> </ol> <p>Note that when sampling on FASTQ files, make sure using same random seed by flag <code>-s</code> (<code>--rand-seed</code>)</p>"},{"location":"usage/#head","title":"head","text":"<p>Usage</p> <pre><code>print first N FASTA/Q records\n\nFor returning the last N records, use:\n    seqkit range -r -N:-1 seqs.fasta\n\nUsage:\n  seqkit head [flags]\n\nFlags:\n  -n, --number int   print first N FASTA/Q records (default 10)\n\n</code></pre> <p>Examples</p> <ol> <li> <p>FASTA</p> <pre><code>$ seqkit head -n 1 hairpin.fa.gz\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> <li> <p>FASTQ</p> <pre><code>$ seqkit head -n 1 reads_1.fq.gz\n@HWI-D00523:240:HF3WGBCXX:1:1101:2574:2226 1:N:0:CTGTAG\nTGAGGAATATTGGTCAATGGGCGCGAGCCTGAACCAGCCAAGTAGCGTGAAGGATGACTGCCCTACGGGTTGTAA\n+\nHIHIIIIIHIIHGHHIHHIIIIIIIIIIIIIIIHHIIIIIHHIHIIIIIGIHIIIIHHHHHHGHIHIIIIIIIII\n</code></pre> </li> </ol>"},{"location":"usage/#head-genome","title":"head-genome","text":"<p>Usage</p> <pre><code>\nprint sequences of the first genome with common prefixes in name\n\nFor a FASTA file containing multiple contigs of strains (see example below),\nthese's no list of IDs available for retrieving sequences of a certain strain,\nwhile descriptions of each strain share the same prefix.\n\nThis command is used to restrieve sequences of the first strain,\ni.e., \"Vibrio cholerae strain M29\".\n\n&gt;NZ_JFGR01000001.1 Vibrio cholerae strain M29 Contig_1, whole genome shotgun sequence\n&gt;NZ_JFGR01000002.1 Vibrio cholerae strain M29 Contig_2, whole genome shotgun sequence\n&gt;NZ_JFGR01000003.1 Vibrio cholerae strain M29 Contig_3, whole genome shotgun sequence\n&gt;NZ_JSTP01000001.1 Vibrio cholerae strain 2012HC-12 NODE_79, whole genome shotgun sequence\n&gt;NZ_JSTP01000002.1 Vibrio cholerae strain 2012HC-12 NODE_78, whole genome shotgun sequence\n\nAttention:\n\n  1. Sequences in file should be well organized.\n\nUsage:\n  seqkit head-genome [flags]\n\nFlags:\n  -h, --help                    help for head-genome\n  -m, --mini-common-words int   minimal shared prefix words (default 4)\n\n</code></pre>"},{"location":"usage/#range","title":"range","text":"<p>Usage</p> <pre><code>print FASTA/Q records in a range (start:end)\n\nExamples:\n  1. leading 100 records (head -n 100)\n      seqkit range -r 1:100\n  2. last 100 records (tail -n 100)\n      seqkit range -r -100:-1\n  3. remove leading 100 records (tail -n +101)\n      seqkit range -r 101:-1\n  4. other ranges:\n      seqkit range -r 10:100\n      seqkit range -r -100:-10\n\nUsage:\n  seqkit range [flags]\n\nFlags:\n  -h, --help           help for range\n  -r, --range string   range. e.g., 1:12 for first 12 records (head -n 12), -12:-1 for last 12 records (tail -n 12)\n</code></pre> <p>Examples</p> <ol> <li> <p>leading N records (head)</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 100 | md5sum\nf65116af7d9298d93ba4b3d19077bbf1  -\n$ cat tests/hairpin.fa | seqkit range -r 1:100 | md5sum\nf65116af7d9298d93ba4b3d19077bbf1  -\n</code></pre> </li> <li> <p>last N records (tail)</p> <pre><code>$ cat tests/hairpin.fa | seqkit range -r -100:-1 | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA        100    8,656       58     86.6      172\n</code></pre> </li> <li> <p>remove leading 100 records (tail -n +101)</p> <pre><code>$ seqkit range -r 101:-1 tests/hairpin.fa | seqkit sum -a\nseqkit.v0.1_RLS_k0_e1feced9bb0be653afa8205dec4987db     -       28545   2940929\n\n$ seqkit fx2tab tests/hairpin.fa | tail -n +101 | seqkit tab2fx | seqkit sum -a\nseqkit.v0.1_RLS_k0_e1feced9bb0be653afa8205dec4987db     -       28545   2940929\n</code></pre> </li> <li> <p>Other ranges</p> <pre><code>$ cat tests/hairpin.fa | seqkit range -r 101:150 | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA         50    3,777       63     75.5       96\n\n$ cat tests/hairpin.fa | seqkit range -r -100:-2 | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA         99    8,484       58     85.7      146\n</code></pre> </li> </ol>"},{"location":"usage/#replace","title":"replace","text":"<p>Usage</p> <pre><code>replace name/sequence by regular expression.\n\nNote that the replacement supports capture variables.\ne.g. $1 represents the text of the first submatch.\nATTENTION: use SINGLE quote NOT double quotes in *nix OS.\n\nExamples: Adding space to all bases.\n\n    seqkit replace -p \"(.)\" -r '$1 ' -s\n\nOr use the \\ escape character.\n\n    seqkit replace -p \"(.)\" -r \"\\$1 \" -s\n\nmore on: http://bioinf.shenwei.me/seqkit/usage/#replace\n\nSpecial replacement symbols (only for replacing name not sequence):\n\n    {nr}    Record number, starting from 1\n    {kv}    Corresponding value of the key (captured variable $n) by key-value file,\n            n can be specified by flag -I (--key-capt-idx) (default: 1)\n\nSpecial cases:\n  1. If replacements contain '$', \n    a). If using '{kv}', you need use '$$$$' instead of a single '$':\n            -r '{kv}' -k &lt;(sed 's/\\$/$$$$/' kv.txt)\n    b). If not, use '$$':\n            -r 'xxx$$xx'\n\nFiltering records to edit:\n  You can use flags similar to those in \"seqkit grep\" to choose partly records to edit.\n\nUsage:\n  seqkit replace [flags]\n\nFlags:\n  -s, --by-seq                   replace seq (only FASTA)\n      --f-by-name                [target filter] match by full name instead of just ID\n      --f-by-seq                 [target filter] search subseq on seq, both positive and negative strand\n                                 are searched, and mismatch allowed using flag -m/--max-mismatch\n      --f-ignore-case            [target filter] ignore case\n      --f-invert-match           [target filter] invert the sense of matching, to select non-matching records\n      --f-only-positive-strand   [target filter] only search on positive strand\n      --f-pattern strings        [target filter] search pattern (multiple values supported. Attention:\n                                 use double quotation marks for patterns containing comma, e.g., -p\n                                 '\"A{2,}\"')\n      --f-pattern-file string    [target filter] pattern file (one record per line)\n      --f-use-regexp             [target filter] patterns are regular expression\n  -h, --help                     help for replace\n  -i, --ignore-case              ignore case\n  -K, --keep-key                 keep the key as value when no value found for the key (only for\n                                 sequence name)\n  -U, --keep-untouch             do not change anything when no value found for the key (only for\n                                 sequence name)\n  -I, --key-capt-idx int         capture variable index of key (1-based) (default 1)\n  -m, --key-miss-repl string     replacement for key with no corresponding value\n  -k, --kv-file string           tab-delimited key-value file for replacing key with value when using\n                                 \"{kv}\" in -r (--replacement) (only for sequence name)\n      --nr-width int             minimum width for {nr} in flag -r/--replacement. e.g., formatting \"1\"\n                                 to \"001\" by --nr-width 3 (default 1)\n  -p, --pattern string           search regular expression\n  -r, --replacement string       replacement. supporting capture variables.  e.g. $1 represents the text\n                                 of the first submatch. ATTENTION: for *nix OS, use SINGLE quote NOT\n                                 double quotes or use the \\ escape character. Record number is also\n                                 supported by \"{nr}\".use ${1} instead of $1 when {kv} given!\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Prepend or append to the header</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\"\n&gt;seq1 abc-123\nACGT-ACGT\n\n# prefix\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p ^ -r _prefix_\n&gt;_prefix_seq1 abc-123\nACGT-ACGT\n\n# suffix\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p $ -r _suffix_\n&gt;seq1 abc-123_suffix_\nACGT-ACGT\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit seq -i \\\n    | seqkit replace -p $ -r _suffix_\n&gt;seq1_suffix_\nACGT-ACGT\n</code></pre> </li> <li> <p>Remove descriptions</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\"\n&gt;seq1 abc-123\nACGT-ACGT\n\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"\\s.+\"\n&gt;seq1\nACGT-ACGT\n</code></pre> </li> <li> <p>Replace \"-\" with \"=\"</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"\\-\" -r '='\n&gt;seq1 abc=123\nACGT-ACGT\n</code></pre> </li> <li> <p>Remove gaps in sequences.</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \" |-\" -s\n&gt;seq1 abc-123\nACGTACGT\n</code></pre> </li> <li> <p>Add space to every base. ATTENTION: use SINGLE quote NOT double quotes in *nix OS</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"(.)\" -r '$1 ' -s\n&gt;seq1 abc-123\nA C G T - A C G T\n\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"(.)\" -r \"\\$1 \" -s\n&gt;seq1 abc-123\nA C G T - A C G T\n</code></pre> </li> <li> <p>Transpose sequence with csvtk</p> <pre><code>$ echo -e \"&gt;seq1\\nACTGACGT\\n&gt;seq2\\nactgccgt\" \\\n    | seqkit replace -p \"(.)\" -r     \"\\$1 \" -s \\\n    | seqkit seq -s -u \\\n    | csvtk space2tab \\\n    | csvtk -t transpose\nA       A\nC       C\nT       T\nG       G\nA       C\nC       C\nG       G\nT       T\n</code></pre> </li> <li> <p>Rename with number of record</p> <pre><code>$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    |  seqkit replace -p .+ -r \"seq_{nr}\"\n&gt;seq_1\nACTG\n&gt;seq_2\nATTT\n\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    |  seqkit replace -p .+ -r \"seq_{nr}\" --nr-width 5\n&gt;seq_00001\nACTG\n&gt;seq_00002\nATTT\n</code></pre> </li> <li> <p>Replace key with value by key-value file</p> <pre><code>$ more test.fa\n&gt;seq1 name1\nCCCCAAAACCCCATGATCATGGATC\n&gt;seq2 name2\nCCCCAAAACCCCATGGCATCATTCA\n&gt;seq3 name3\nCCCCAAAACCCCATGTTGCTACTAG\n\n$ more alias.txt\nname0   ABC\nname1   123\nname3   Hello\nname4   World\n\n$ seqkit replace -p ' (.+)$' -r ' {kv}' -k alias.txt test.fa\n[INFO] read key-value file: alias.txt\n[INFO] 4 pairs of key-value loaded\n&gt;seq1 123\nCCCCAAAACCCCATGATCATGGATC\n&gt;seq2\nCCCCAAAACCCCATGGCATCATTCA\n&gt;seq3 Hello\nCCCCAAAACCCCATGTTGCTACTAG\n\n$ seqkit replace -p ' (.+)$' -r ' {kv}' -k alias.txt test.fa --keep-key\n[INFO] read key-value file: alias.txt\n[INFO] 4 pairs of key-value loaded\n&gt;seq1 123\nCCCCAAAACCCCATGATCATGGATC\n&gt;seq2 name2\nCCCCAAAACCCCATGGCATCATTCA\n&gt;seq3 Hello\nCCCCAAAACCCCATGTTGCTACTAG\n</code></pre> <p>If the target to replace is the sequence ID, meanwhile you would like to keep the description.  Please use this:</p> <pre><code># ^(\\S+)   # ID\n# (.+)$    # description\nseqkit replace -p '^(\\S+)(.+?)$' -r '{kv}$2' -k alias.txt test.fa\n</code></pre> </li> <li> <p>convert fasta to genbank style</p> <pre><code>$ cat seq.fa\n&gt;seq1\nTTTAAAGAGACCGGCGATTCTAGTGAAATCGAACGGGCAGGTCAATTTCCAACCAGCGAT\nGACGTAATAGATAGATACAAGGAAGTCATTTTTCTTTTAAAGGATAGAAACGGTTAATGC\nTCTTGGGACGGCGCTTTTCTGTGCATAACT\n&gt;seq2\nAAGGATAGAAACGGTTAATGCTCTTGGGACGGCGCTTTTCTGTGCATAACTCGATGAAGC\nCCAGCAATTGCGTGTTTCTCCGGCAGGCAAAAGGTTGTCGAGAACCGGTGTCGAGGCTGT\nTTCCTTCCTGAGCGAAGCCTGGGGATGAACG\n\n$ cat seq.fa \\\n    | seqkit replace -s -p '(\\w{10})' -r '$1 ' -w 66 \\\n    | perl -ne 'if (/^&gt;/) {print; $n=1} else {s/ \\r?\\n$/\\n/; printf \"%9d %s\", $n, $_; $n+=60;}'\n&gt;seq1\n        1 TTTAAAGAGA CCGGCGATTC TAGTGAAATC GAACGGGCAG GTCAATTTCC AACCAGCGAT\n       61 GACGTAATAG ATAGATACAA GGAAGTCATT TTTCTTTTAA AGGATAGAAA CGGTTAATGC\n      121 TCTTGGGACG GCGCTTTTCT GTGCATAACT\n&gt;seq2\n        1 AAGGATAGAA ACGGTTAATG CTCTTGGGAC GGCGCTTTTC TGTGCATAAC TCGATGAAGC\n       61 CCAGCAATTG CGTGTTTCTC CGGCAGGCAA AAGGTTGTCG AGAACCGGTG TCGAGGCTGT\n      121 TTCCTTCCTG AGCGAAGCCT GGGGATGAAC G\n</code></pre> </li> <li> <p>only edit some of the records via patterns. Eight flags starting with <code>--f-</code> (with the same usage to these in <code>seqkit grep</code>) can be used    to filter records to edit.</p> <pre><code># prepend some text to header\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    | seqkit replace -p ^ -r _\n&gt;_abc\nACTG\n&gt;_123\nATTT\n\n# only edit some records\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    | seqkit replace -p ^ -r _ --f-use-regexp --f-pattern abc\n[INFO] 1 records matched by the filter\n&gt;_abc\nACTG\n&gt;123\nATTT\n\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    | seqkit replace -p ^ -r _ --f-use-regexp --f-invert-match --f-pattern abc \n[INFO] 1 records matched by the filter\n&gt;abc\nACTG\n&gt;_123\nATTT\n</code></pre> </li> </ol>"},{"location":"usage/#rename","title":"rename","text":"<p>Usage</p> <pre><code>rename duplicated IDs\n\nAttention:\n  1. This command only appends \"_N\" to duplicated sequence IDs to make them unique.\n  2. Use \"seqkit replace\" for editing sequence IDs/headers using regular expression.\n\nExample:\n\n    $ seqkit seq seqs.fasta \n    &gt;id comment\n    actg\n    &gt;id description\n    ACTG\n\n    $ seqkit rename seqs.fasta\n    &gt;id comment\n    actg\n    &gt;id_2 description\n    ACTG\n\nUsage:\n  seqkit rename [flags]\n\nFlags:\n  -n, --by-name             check duplication by full name instead of just id\n  -f, --force               overwrite output directory\n  -h, --help                help for rename\n  -m, --multiple-outfiles   write results into separated files for multiple input files\n  -O, --out-dir string      output directory (default \"renamed\")\n  -1, --rename-1st-rec      rename the first record as well\n  -s, --separator string    separator between original ID/name and the counter (default \"_\")\n  -N, --start-num int       starting count number for *duplicated* IDs/names, should be greater than\n                            zero (default 2\n\n</code></pre> <p>Examples</p> <pre><code>$ echo -e \"&gt;a comment\\nacgt\\n&gt;b comment of b\\nACTG\\n&gt;a comment\\naaaa\"\n&gt;a comment\nacgt\n&gt;b comment of b\nACTG\n&gt;a comment\naaaa\n\n$ echo -e \"&gt;a comment\\nacgt\\n&gt;b comment of b\\nACTG\\n&gt;a comment\\naaaa\" \\\n    | seqkit rename\n&gt;a comment\nacgt\n&gt;b comment of b\nACTG\n&gt;a_2 comment\naaaa\n\n$ echo -e \"&gt;a comment\\nacgt\\n&gt;b comment of b\\nACTG\\n&gt;a comment\\naaaa\"  \\\n    | seqkit rename -s '|' -N 2 -1\n&gt;a|1 comment\nacgt\n&gt;b|1 comment of b\nACTG\n&gt;a|2 comment\naaaa\n\n</code></pre>"},{"location":"usage/#restart","title":"restart","text":"<p>Usage</p> <pre><code>reset start position for circular genome\n\nExamples\n\n    $ echo -e \"&gt;seq\\nacgtnACGTN\"\n    &gt;seq\n    acgtnACGTN\n\n    $ echo -e \"&gt;seq\\nacgtnACGTN\" | seqkit restart -i 2\n    &gt;seq\n    cgtnACGTNa\n\n    $ echo -e \"&gt;seq\\nacgtnACGTN\" | seqkit restart -i -2\n    &gt;seq\n    TNacgtnACG\n\nUsage:\n  seqkit restart [flags]\n\nFlags:\n  -i, --new-start int   new start position (1-base, supporting negative value counting from the end) (default 1)\n\n</code></pre>"},{"location":"usage/#concat","title":"concat","text":"<p>Usage</p> <pre><code>concatenate sequences with same ID from multiple files\n\nAttention:\n   1. By default, only sequences with IDs that appear in all files are outputted.\n      use -f/--full to output all sequences.\n   2. If there are more than one sequences of the same ID, we output the Cartesian\n      product of sequences.\n   3. Description are also concatenated with a separator (-s/--separator).\n   4. Order of sequences with different IDs are random.\n\nUsage:\n  seqkit concat [flags]\n\nAliases:\n  concat, concate\n\nFlags:\n  -f, --full               keep all sequences, like full/outer join\n  -h, --help               help for concat\n  -s, --separator string   separator for descriptions of records with the same ID (default \"|\")\n\n</code></pre> <p>Examples</p> <pre><code>$ cat a.fa \n&gt;A 1\na1-\n&gt;A 2\na2-\n&gt;B 1\nb1-\n\n$ cat b.fa \n&gt;A x\nax-\n&gt;C 1\nc1-\n\n$ seqkit concat a.fa b.fa \n&gt;A 1|x\na1-ax-\n&gt;A 2|x\na2-ax-\n\n$ seqkit concat a.fa b.fa --full\n&gt;C 1\nc1-\n&gt;A 1|x\na1-ax-\n&gt;A 2|x\na2-ax-\n&gt;B 1\nb1-\n\n</code></pre>"},{"location":"usage/#mutate","title":"mutate","text":"<p>Usage</p> <pre><code>edit sequence (point mutation, insertion, deletion)\n\nAttention:\n\n  1. Mutiple point mutations (-p/--point) are allowed, but only single \n     insertion (-i/--insertion) OR single deletion (-d/--deletion) is allowed.\n  2. Point mutation takes place before insertion/deletion.\n\nNotes:\n\n  1. You can choose certain sequences to edit using similar flags in\n     'seqkit grep'.\n\nThe definition of position is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit mutate [flags]\n\nFlags:\n  -n, --by-name               [match seqs to mutate] match by full name instead of just id\n  -d, --deletion string       deletion mutation: deleting subsequence in a range. e.g., -d 1:2 for\n                              deleting leading two bases, -d -3:-1 for removing last 3 bases\n  -h, --help                  help for mutate\n  -I, --ignore-case           [match seqs to mutate] ignore case of search pattern\n  -i, --insertion string      insertion mutation: inserting bases behind of given position, e.g., -i\n                              0:ACGT for inserting ACGT at the beginning, -1:* for add * to the end\n  -v, --invert-match          [match seqs to mutate] invert the sense of matching, to select\n                              non-matching records\n  -s, --pattern strings       [match seqs to mutate] search pattern (multiple values supported.\n                              Attention: use double quotation marks for patterns containing comma, e.g.,\n                              -p '\"A{2,}\"'))\n  -f, --pattern-file string   [match seqs to mutate] pattern file (one record per line)\n  -p, --point strings         point mutation: changing base at given position. e.g., -p 2:C for setting\n                              2nd base as C, -p -1:A for change last base as A\n  -r, --use-regexp            [match seqs to mutate] search patterns are regular expression\n\n</code></pre> <p>Examples:</p> <ol> <li> <p>Point mutation:</p> <pre><code>$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\"\n&gt;1\nACTGNactgn\n&gt;2\nactgnACTGN\n\n# first base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p 1:x\n[INFO] edit seq: 1\n[INFO] edit seq: 2\n&gt;1\nxCTGNactgn\n&gt;2\nxctgnACTGN\n\n# 5th base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p 5:x --quiet\n&gt;1\nACTGxactgn\n&gt;2\nactgxACTGN\n\n# last base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p -1:x --quiet\n&gt;1\nACTGNactgx\n&gt;2\nactgnACTGx\n\n# mutiple locations:\n\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p 1:x -p -1:x --quiet\n&gt;1\nxCTGNactgx\n&gt;2\nxctgnACTGx\n</code></pre> </li> <li> <p>Deletion</p> <pre><code># first base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -d 1:1 --quiet\n&gt;1\nCTGNactgn\n&gt;2\nctgnACTGN\n\n# last 3 bases\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -d -3:-1 --quiet\n&gt;1\nACTGNac\n&gt;2\nactgnAC\n</code></pre> </li> <li> <p>Insertion: inserting bases behind of given position</p> <pre><code># at the beginning\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -i 0:xx --quiet\n&gt;1\nxxACTGNactgn\n&gt;2\nxxactgnACTGN\n\n# at the end\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -i -1:xx --quiet\n&gt;1\nACTGNactgnxx\n&gt;2\nactgnACTGNxx\n\n# behind of 5th base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -i 5:x --quiet\n&gt;1\nACTGNxactgn\n&gt;2\nactgnxACTGN\n</code></pre> </li> <li> <p>Choosing which sequences to edit, using similar flags in <code>seqkit grep</code>.</p> <pre><code>$ cat tests/hsa.fa\n&gt;chr1 1th seq\nACTGNactgn\n&gt;chr2 2nd seq\nactgnACTGN\n&gt;chr11 11th seq\nACTGNACTGN\n&gt;MT mitochondrial seq\nactgnactgn\n\n# only edit chr1 and chr2\n# or cat tests/hsa.fa | seqkit mutate -p -1:X -s chr1 -s chr2\n$ cat tests/hsa.fa \\\n    | seqkit mutate -p -1:X -s chr1,chr2\n[INFO] edit seq: chr1 1th seq\n[INFO] edit seq: chr2 2nd seq\n&gt;chr1 1th seq\nACTGNactgX\n&gt;chr2 2nd seq\nactgnACTGX\n&gt;chr11 11th seq\nACTGNACTGN\n&gt;MT mitochondrial seq\nactgnactgn\n\n# using regular expression to match.\n# e,g., editing all chrosomes:\n$ cat tests/hsa.fa \\\n    | seqkit mutate -p -1:X -r -s chr\n[INFO] edit seq: chr1 1th seq\n[INFO] edit seq: chr2 2nd seq\n[INFO] edit seq: chr11 11th seq\n&gt;chr1 1th seq\nACTGNactgX\n&gt;chr2 2nd seq\nactgnACTGX\n&gt;chr11 11th seq\nACTGNACTGX\n&gt;MT mitochondrial seq\nactgnactgn\n\n# excluding seqs\n$ cat tests/hsa.fa \\\n    | seqkit mutate -p -1:X -s chr1 -s chr2 -v \n[INFO] edit seq: chr11 11th seq\n[INFO] edit seq: MT mitochondrial seq\n&gt;chr1 1th seq\nACTGNactgn\n&gt;chr2 2nd seq\nactgnACTGN\n&gt;chr11 11th seq\nACTGNACTGX\n&gt;MT mitochondrial seq\nactgnactgX\n</code></pre> </li> </ol>"},{"location":"usage/#shuffle","title":"shuffle","text":"<p>Usage</p> <pre><code>shuffle sequences.\n\nBy default, all records will be readed into memory.\nFor FASTA format, use flag -2 (--two-pass) to reduce memory usage. FASTQ not\nsupported.\n\nFirstly, seqkit reads the sequence IDs. If the file is not plain FASTA file,\nseqkit will write the sequences to temporary files, and create FASTA index.\n\nSecondly, seqkit shuffles sequence IDs and extract sequences by FASTA index.\n\nAttention:\n  1. For the two-pass mode (-2/--two-pass), The flag -U/--update-faidx is recommended to\n     ensure the .fai file matches the FASTA file.\n\nUsage:\n  seqkit shuffle [flags]\n\nFlags:\n  -h, --help            help for shuffle\n  -k, --keep-temp       keep temporary FASTA and .fai file when using 2-pass mode\n  -s, --rand-seed int   rand seed for shuffle (default 23)\n  -2, --two-pass        two-pass mode read files twice to lower memory usage. (only for FASTA format)\n  -U, --update-faidx    update the fasta index file if it exists. Use this if you are not sure whether\n                        the fasta file changed\n\n</code></pre> <p>Examples</p> <ol> <li> <p>General use.</p> <pre><code>$ seqkit shuffle hairpin.fa.gz &gt; shuffled.fa\n[INFO] read sequences ...\n[INFO] 28645 sequences loaded\n[INFO] shuffle ...\n[INFO] output ...\n</code></pre> </li> <li> <p>For big genome, you'd better use two-pass mode so seqkit could use    FASTA index to reduce memory usage</p> <pre><code>$ time seqkit shuffle -2 hsa.fa &gt; shuffle.fa\n[INFO] create and read FASTA index ...\n[INFO] create FASTA index for hsa.fa\n[INFO] read sequence IDs from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] shuffle ...\n[INFO] output ...\n\nreal    0m35.080s\nuser    0m45.521s\nsys     0m3.411s\n</code></pre> </li> </ol> <p>Note that when sampling on FASTQ files, make sure using same random seed by flag <code>-s</code> (<code>--rand-seed</code>) for read 1 and 2 files.</p>"},{"location":"usage/#sort","title":"sort","text":"<p>Usage</p> <pre><code>sort sequences by id/name/sequence/length.\n\nBy default, all records will be readed into memory.\nFor FASTA format, use flag -2 (--two-pass) to reduce memory usage. FASTQ not\nsupported.\n\nFirstly, seqkit reads the sequence head and length information.\nIf the file is not plain FASTA file,\nseqkit will write the sequences to temporary files, and create FASTA index.\n\nSecondly, seqkit sorts sequence by head and length information\nand extracts sequences by FASTA index.\n\nAttention:\n  1. For the two-pass mode (-2/--two-pass), The flag -U/--update-faidx is recommended to\n     ensure the .fai file matches the FASTA file.\n\nUsage:\n  seqkit sort [flags]\n\nFlags:\n  -b, --by-bases                by non-gap bases\n  -l, --by-length               by sequence length\n  -n, --by-name                 by full name instead of just id\n  -s, --by-seq                  by sequence\n  -G, --gap-letters string      gap letters (default \"- \\t.\")\n  -h, --help                    help for sort\n  -i, --ignore-case             ignore case\n  -k, --keep-temp               keep temporary FASTA and .fai file when using 2-pass mode\n  -N, --natural-order           sort in natural order, when sorting by IDs/full name\n  -r, --reverse                 reverse the result\n  -L, --seq-prefix-length int   length of sequence prefix on which seqkit sorts by sequences (0 for\n                                whole sequence) (default 10000)\n  -2, --two-pass                two-pass mode read files twice to lower memory usage. (only for FASTA format)\n  -U, --update-faidx            update the fasta index file if it exists. Use this if you are not sure\n                                whether the fasta file changed\n\n</code></pre> <p>Examples</p> <p>For FASTA format, use flag -2 (--two-pass) to reduce memory usage</p> <ol> <li> <p>sort by ID</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAA\" \\\n    | seqkit sort --quiet\n&gt;SEQ2\nacgtnAAAA\n&gt;seq1\nACGTNcccc\n</code></pre> </li> <li> <p>sort by ID and in natural order</p> <pre><code>$ echo -e \"&gt;3\\na\\n&gt;1\\na\\n&gt;Y\\na\\n&gt;x\\na\\n&gt;Mt\\na\\n&gt;11\\na\\n&gt;2\\na\\n\" \\\n    | seqkit seq -n -i\n3\n1\nY\nx\nMt\n11\n2\n\n$ echo -e \"&gt;3\\na\\n&gt;1\\na\\n&gt;Y\\na\\n&gt;x\\na\\n&gt;Mt\\na\\n&gt;11\\na\\n&gt;2\\na\\n\" \\\n    | seqkit sort -N -i -2 \\\n    | seqkit seq -n -i\n1\n2\n3\n11\nMt\nx\nY\n</code></pre> </li> <li> <p>sort by ID, ignoring case.</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAA\" \\\n    | seqkit sort --quiet -i\n&gt;seq1\nACGTNcccc\n&gt;SEQ2\nacgtnAAAA\n</code></pre> </li> <li> <p>sort by seq, ignoring case.</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAA\" \\\n    | seqkit sort --quiet -s -i\n&gt;SEQ2\nacgtnAAAA\n&gt;seq1\nACGTNcccc\n</code></pre> </li> <li> <p>sort by sequence length</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAAnnn\\n&gt;seq3\\nacgt\" \\\n    | seqkit sort --quiet -l\n&gt;seq3\nacgt\n&gt;seq1\nACGTNcccc\n&gt;SEQ2\nacgtnAAAAnnn\n</code></pre> </li> </ol>"},{"location":"usage/#bam","title":"bam","text":"<pre><code>monitoring and online histograms of BAM record features\n\nUsage:\n  seqkit bam [flags]\n\nFlags:\n  -B, --bins int             number of histogram bins (default -1)\n  -N, --bundle int           partition BAM file into loci (-1) or bundles with this minimum size\n  -c, --count string         count reads per reference and save to this file\n  -W, --delay int            sleep this many seconds after plotting (default 1)\n  -y, --dump                 print histogram data to stderr instead of plotting\n  -G, --exclude-ids string   exclude records with IDs contained in this file\n  -e, --exec-after string    execute command after reporting\n  -E, --exec-before string   execute command before reporting\n  -f, --field string         target fields\n  -g, --grep-ids string      only keep records with IDs contained in this file\n  -h, --help                 help for bam\n  -C, --idx-count            fast read per reference counting based on the BAM index\n  -i, --idx-stat             fast statistics based on the BAM index\n  -O, --img string           save histogram to this PDF/image file\n  -H, --list-fields          list all available BAM record features\n  -L, --log                  log10(x+1) transform numeric values\n  -q, --map-qual int         minimum mapping quality\n  -x, --pass                 passthrough mode (forward filtered BAM to output)\n  -k, --pretty               pretty print certain TSV outputs\n  -F, --prim-only            filter out non-primary alignment records\n  -p, --print-freq int       print/report after this many records (-1 for print after EOF) (default -1)\n  -Q, --quiet-mode           supress all plotting to stderr\n  -M, --range-max float      discard record with field (-f) value greater than this flag (default NaN)\n  -m, --range-min float      discard record with field (-f) value less than this flag (default NaN)\n  -R, --reset                reset histogram after every report\n  -Z, --silent-mode          supress TSV output to stderr\n  -s, --stat                 print BAM satistics of the input files\n  -T, --tool string          invoke toolbox in YAML format (see documentation)\n  -@, --top-bam string       save the top -? records to this bam file\n  -?, --top-size int         size of the top-mode buffer (default 100)\n</code></pre> <p>Examples</p> <ol> <li> <p>Get detailed statistics from multiple BAM files.</p> <pre><code>seqkit bam -s *.bam\n</code></pre> </li> <li> <p>Get rough statistics from multiple indexed BAM files.</p> <pre><code>seqkit bam -i *.bam\n</code></pre> </li> <li> <p>Count reads mapped to references from a BAM stream.</p> <pre><code>cat sample.bam | seqkit bam -c counts.tsv  -\n</code></pre> </li> <li> <p>Count reads mapped to references using the BAM index.</p> <pre><code>seqkit bam -C sorted_indexed.bam\n</code></pre> </li> <li> <p>Monitor alignment accuracy from a bam stream and report after every 1000 records, use 20 bins.</p> <pre><code>cat sample.bam | seqkit bam -B -f Acc -p 1000 -\n</code></pre> </li> <li> <p>Dump selected fields to TSV.</p> <pre><code>seqkit bam -f Ref,Acc,RefCov,Strand sample.bam\n</code></pre> </li> <li> <p>Save the best 100 records in terms of alignment accuracy to a BAM file.</p> <pre><code>seqkit bam -f Acc -@ top_acc_100.bam -? 100 -Q sample.bam\n</code></pre> </li> <li> <p>Inkvoke the BAM toolbox.</p> </li> </ol> <p>The BAM toolbox is a collection of filters acting on a stream of BAM records, configured via YAML.  The currently available tools can be listed by <code>seqkit bam -T help</code>:</p> <pre><code>Tool        Description\n----        -----------\nAccStats    calculates mean accuracy weighted by aligment lengths\nAlnContext  filter records by the sequence context at start and end\nDump        dump various record properties in TSV format\nhelp        list all tools with description\n</code></pre> <p>Example YAML configs:</p> <p>Invoking the AccStats tool directly from the command line or YAML config:</p> <pre><code>seqkit bam -T '{AccStats: {Tsv: \"-\"}, Sink: True}' input.bam\nseqkit bam -T '{Yaml: \"tests/examples/bam_tool_acc_stats.yml\"}' input.bam\n</code></pre> <p>Where the contents of <code>bam_tool_acc_stats.yml</code> are:</p> <pre><code>AccStats:\n  Tsv: \"-\"\nSink: True\n</code></pre> <p>Invoking the AlnContext tool using YAML:</p> <pre><code>AlnContext:\n  Tsv: \"-\"\n  Ref: \"../SIRV_150601a.fasta\"\n  LeftShift: -10\n  RightShift: 10\n  RegexStart: \"T{4,}\"\n  RegexEnd: \"A{4,}\"\n  Stranded: True\n  Invert: True\nSink: True\n</code></pre> <p>Invoking the Dump tool using YAML:</p> <pre><code>Dump:\n  Tsv: \"-\"\n  Fields: [\"Read\", \"Ref\", \"Pos\", \"EndPos\", \"MapQual\", \"Acc\", \"Match\", \"Mismatch\", \"Ins\", \"Del\", \"AlnLen\", \"  ReadLen\", \"RefLen\", \"RefAln\", \"RefCov\", \"ReadAln\", \"ReadCov\", \"Strand\", \"MeanQual\", \"LeftClip\", \"RightClip\", \"Flags\", \"IsSec\", \"  IsSup\", \"ReadSeq\", \"ReadAlnSeq\", \"LeftSoftClipSeq\", \"RightSoftClip\", \"LeftHardClip\", \"RightHardClip\"]\nSink: True\n</code></pre> <p>The tools can be chained together, for example the YAML using all three tools look like:</p> <pre><code>AlnContext:\n  Tsv: \"context.tsv\"\n  Ref: \"../SIRV_150601a.fasta\"\n  LeftShift: -10\n  RightShift: 10\n  RegexStart: \"T{4,}\"\n  RegexEnd: \"A{4,}\"\n  Stranded: True\n  Invert: True\nDump:\n  Tsv: \"dump.tsv\"\n  Fields: [\"Read\", \"Ref\", \"Pos\", \"EndPos\", \"MapQual\", \"Acc\", \"Match\", \"Mismatch\", \"Ins\", \"Del\", \"AlnLen\", \"  ReadLen\", \"RefLen\", \"RefAln\", \"RefCov\", \"ReadAln\", \"ReadCov\", \"Strand\", \"MeanQual\", \"LeftClip\", \"RightClip\", \"Flags\", \"IsSec\", \"  IsSup\", \"ReadSeq\", \"ReadAlnSeq\", \"LeftSoftClipSeq\", \"RightSoftClip\", \"LeftHardClip\", \"RightHardClip\"]\nAccStats:\n  Tsv: \"-\"\n</code></pre> <p>If the \"Sink\" parameter is not specified in the last pipeline step, the output BAM records are streamed to the standard output and can be piped into standard tools, for example:</p> <pre><code>seqkit bam -T '{Yaml: \"bam_tool_pipeline.yml\"}' ../pcs109_5k_spliced.bam | samtools flagstat -\n</code></pre>"},{"location":"usage/#merge-slides","title":"merge-slides","text":"<p>Usage</p> <pre><code>merge sliding windows generated from seqkit sliding\n\nFor example,\n\n    ref.contig00001_sliding:454531-454680\n    ref.contig00001_sliding:454561-454710\n    ref.contig00001_sliding:454591-454740\n    ref.contig00002_sliding:362281-362430\n    ref.contig00002_sliding:362311-362460\n    ref.contig00002_sliding:362341-362490\n    ref.contig00002_sliding:495991-496140\n    ref.contig00044_sliding:1-150\n    ref.contig00044_sliding:31-180\n    ref.contig00044_sliding:61-210\n    ref.contig00044_sliding:91-240\n\ncould be merged into\n\n    ref.contig00001 454530  454740\n    ref.contig00002 362280  362490\n    ref.contig00002 495990  496140\n    ref.contig00044 0       240\n\nOutput (BED3 format):\n    1. chrom      - chromosome name\n    2. chromStart - starting position (0-based)\n    3. chromEnd   - ending position (0-based)\n\nUsage:\n  seqkit merge-slides [flags]\n\nFlags:\n  -b, --buffer-size string            size of buffer, supported unit: K, M, G. You need increase the\n                                      value when \"bufio.Scanner: token too long\" error reported (default\n                                      \"1G\")\n  -p, --comment-line-prefix strings   comment line prefix (default [#,//])\n  -h, --help                          help for merge-slides\n  -g, --max-gap int                   maximum distance of starting positions of two adjacent regions, 0\n                                      for no limitation, 1 for no merging.\n  -l, --min-overlap int               minimum overlap of two adjacent regions, recommend\n                                      $sliding_step_size - 1. (default 1)\n  -r, --regexp string                 regular expression for extract the reference name and window\n                                      position. (default \"^(.+)_sliding:(\\\\d+)\\\\-(\\\\d+)\")\n\n</code></pre> <p>Example:</p> <pre><code>seqkit merge-slides sliding_windows.tsv -l 50 -o sliding_windows.merged.tsv\n</code></pre>"},{"location":"usage/#genautocomplete","title":"genautocomplete","text":"<p>Usage</p> <pre><code>generate shell autocompletion script\n\nSupported shell: bash|zsh|fish|powershell\n\nBash:\n\n    # generate completion shell\n    seqkit genautocomplete --shell bash\n\n    # configure if never did.\n    # install bash-completion if the \"complete\" command is not found.\n    echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" &gt;&gt; ~/.bash_completion\n    echo \"source ~/.bash_completion\" &gt;&gt; ~/.bashrc\n\nZsh:\n\n    # generate completion shell\n    seqkit genautocomplete --shell zsh --file ~/.zfunc/_seqkit\n\n    # configure if never did\n    echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' &gt;&gt; ~/.zshrc\n    echo \"autoload -U compinit; compinit\" &gt;&gt; ~/.zshrc\n\nfish:\n\n    seqkit genautocomplete --shell fish --file ~/.config/fish/completions/seqkit.fish\n\nUsage:\n  seqkit genautocomplete [flags]\n\nFlags:\n      --file string   autocompletion file (default \"/home/shenwei/.bash_completion.d/seqkit.sh\")\n  -h, --help          help for genautocomplete\n      --type string   autocompletion type (currently only bash supported) (default \"bash\")\n\n</code></pre> Please enable JavaScript to view the comments powered by Disqus."},{"location":"yanlilab/","title":"Yanlilab","text":""}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"SeqKit - a cross-platform and ultrafast toolkit for FASTA/Q file manipulation","text":"<ul> <li>Try SeqKit in your browser (Tutorials and Exercises provided by sandbox.bio)</li> <li>Documents: http://bioinf.shenwei.me/seqkit (Usage, FAQs, Tutorial, and  Benchmark)</li> <li>Source code: https://github.com/shenwei356/seqkit </li> <li>Latest version: </li> <li>Please cite: ,  </li> <li>Others:  </li> </ul>"},{"location":"#features","title":"Features","text":"<ul> <li>Easy to install (download)<ul> <li>Providing statically linked executable binaries for multiple platforms (Linux/Windows/macOS, amd64/arm64) </li> <li>Light weight and out-of-the-box, no dependencies, no compilation, no configuration</li> <li><code>conda install -c bioconda seqkit</code></li> </ul> </li> <li>Easy to use <ul> <li>Ultrafast (see technical-details and benchmark)</li> <li>Seamlessly parsing both FASTA and FASTQ formats</li> <li>Supporting (<code>gzip</code>/<code>xz</code>/<code>zstd</code>/<code>bzip2</code> compressed) STDIN/STDOUT and input/output file, easily integrated in pipe</li> <li>Reproducible results (configurable rand seed in <code>sample</code> and <code>shuffle</code>)</li> <li>Supporting custom sequence ID via regular expression</li> <li>Supporting Bash/Zsh autocompletion</li> </ul> </li> <li>Versatile commands (usages and examples)<ul> <li>Practical functions supported by 38 subcommands</li> </ul> </li> </ul>"},{"location":"#installation","title":"Installation","text":"<p>Go to Download Page for more download options and changelogs, or install via conda:</p> <pre><code>conda install -c bioconda seqkit\n</code></pre>"},{"location":"#subcommands","title":"Subcommands","text":"Category Command Function Input Strand-sensitivity Multi-threads Basic operation seq Transform sequences: extract ID/seq, filter by length/quality, remove gaps\u2026 FASTA/Q stats Simple statistics: #seqs, min/max_len, N50, Q20%, Q30%\u2026 FASTA/Q \u2713 subseq Get subsequences by region/gtf/bed, including flanking sequences FASTA/Q + or/and - sliding Extract subsequences in sliding windows FASTA/Q + only faidx Create the FASTA index file and extract subsequences (with more features than samtools faidx) FASTA + or/and - translate translate DNA/RNA to protein sequence FASTA/Q + or/and - watch  Monitoring and online histograms of sequence features FASTA/Q scat  Real time concatenation and streaming of fastx files FASTA/Q \u2713 Format conversion fq2fa Convert FASTQ to FASTA format FASTQ fx2tab Convert FASTA/Q to tabular format FASTA/Q fa2fq Retrieve corresponding FASTQ records by a FASTA file FASTA/Q + only tab2fx Convert tabular format to FASTA/Q format TSV convert Convert FASTQ quality encoding between Sanger, Solexa and Illumina FASTA/Q Searching grep Search sequences by ID/name/sequence/sequence motifs, mismatch allowed FASTA/Q + and - partly, -m locate Locate subsequences/motifs, mismatch allowed FASTA/Q + and - partly, -m amplicon Extract amplicon (or specific region around it), mismatch allowed FASTA/Q + and - partly, -m fish Look for short sequences in larger sequences FASTA/Q + and - Set operation sample Sample sequences by number or proportion FASTA/Q rmdup Remove duplicated sequences by ID/name/sequence FASTA/Q + and - common Find common sequences of multiple files by id/name/sequence FASTA/Q + and - duplicate Duplicate sequences N times FASTA/Q split Split sequences into files by id/seq region/size/parts (mainly for FASTA) FASTA preffered split2 Split sequences into files by size/parts (FASTA, PE/SE FASTQ) FASTA/Q head Print first N FASTA/Q records FASTA/Q head-genome Print sequences of the first genome with common prefixes in name FASTA/Q range Print FASTA/Q records in a range (start:end) FASTA/Q pair Patch up paired-end reads from two fastq files FASTA/Q Edit replace Replace name/sequence by regular expression FASTA/Q + only rename Rename duplicated IDs FASTA/Q concat Concatenate sequences with same ID from multiple files FASTA/Q + only restart Reset start position for circular genome FASTA/Q + only mutate Edit sequence (point mutation, insertion, deletion) FASTA/Q + only sana Sanitize broken single line FASTQ files FASTQ Ordering sort Sort sequences by id/name/sequence/length FASTA preffered shuffle Shuffle sequences FASTA preffered BAM processing bam Monitoring and online histograms of BAM record features BAM Miscellaneous sum Compute message digest for all sequences in FASTA/Q files FASTA/Q \u2713 merge-slides Merge sliding windows generated from seqkit sliding TSV <p>Notes:</p> <ul> <li>Strand-sensitivity:<ul> <li><code>+ only</code>: only processing on the positive/forward strand.</li> <li><code>+ and -</code>: searching on both strands.</li> <li><code>+ or/and -</code>: depends on users' flags/options/arguments.</li> </ul> </li> <li>Multiple-threads: Using the default 4 threads is fast enough for most commands, some commands can benefit from extra threads.</li> </ul>"},{"location":"#citation","title":"Citation","text":"<ol> <li>Wei Shen*, Botond Sipos, and Liuyang Zhao. 2024. SeqKit2: A Swiss Army Knife for Sequence and Alignment Processing. iMeta e191. doi:10.1002/imt2.191. </li> <li>Wei Shen, Shuai Le, Yan Li*, and Fuquan Hu*. SeqKit: a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PLOS ONE. doi:10.1371/journal.pone.0163962. </li> </ol>"},{"location":"#contributors","title":"Contributors","text":"<ul> <li>Wei Shen</li> <li>Botond Sipos: <code>bam</code>, <code>scat</code>, <code>fish</code>, <code>sana</code>, <code>watch</code>.</li> <li>others</li> </ul>"},{"location":"#acknowledgements","title":"Acknowledgements","text":"<p>We thank all users for their valuable feedback and suggestions. We thank all contributors for improving the code and documentation.</p> <p>We appreciate Klaus Post for his fantastic packages ( compress and pgzip ) which accelerate gzip file reading and writing.</p>"},{"location":"#contact","title":"Contact","text":"<p>Create an issue to report bugs, propose new functions or ask for help.</p>"},{"location":"#license","title":"License","text":"<p>MIT License</p>"},{"location":"#starchart","title":"Starchart","text":""},{"location":"benchmark/","title":"Benchmark","text":"<p>Note: these benchmarks was performed in 2016.</p>"},{"location":"benchmark/#softwares","title":"Softwares","text":"<ol> <li>seqkit. (Go).    Version v0.3.1.1.    Compiled with Go 1.7rc5.</li> <li>fasta_utilities. (Perl).    Version 3dcc0bc.    Lots of dependencies to install.</li> <li>fastx_toolkit. (Perl).    Version 0.0.13.    Can't handle multi-line FASTA files.</li> <li>seqmagick. (Python).    Version 0.6.1</li> <li>seqtk. (C).    Version 1.1-r92-dirty.</li> </ol> <p>Not used:</p> <ol> <li>pyfaidx. (Python).    Version 0.4.7.1. Not used, because it exhausted my memory (10G) when computing reverse-complement on a 5GB fasta file of 250 bp.</li> </ol> <p>A Python script memusg was used to compute running time and peak memory usage of a process.</p>"},{"location":"benchmark/#features","title":"Features","text":"Categories Features seqkit fasta_utilities fastx_toolkit pyfaidx seqmagick seqtk Formats supports Multi-line FASTA Yes Yes -- Yes Yes Yes FASTQ Yes Yes Yes -- Yes Yes Multi-line  FASTQ Yes Yes -- -- Yes Yes Validating sequences Yes -- Yes Yes -- -- Supporting RNA Yes Yes -- -- Yes Yes Functions Searching by motifs Yes Yes -- -- Yes -- Sampling Yes -- -- -- Yes Yes Extracting sub-sequence Yes Yes -- Yes Yes Yes Removing duplicates Yes -- -- -- Partly -- Splitting Yes Yes -- Partly -- -- Splitting by seq Yes -- Yes Yes -- -- Shuffling Yes -- -- -- -- -- Sorting Yes Yes -- -- Yes -- Locating motifs Yes -- -- -- -- -- Common sequences Yes -- -- -- -- -- Cleaning bases Yes Yes Yes Yes -- -- Transcription Yes Yes Yes Yes Yes Yes Translation -- Yes Yes Yes Yes -- Filtering by size Indirect Yes -- Yes Yes -- Renaming header Yes Yes -- -- Yes Yes Other features Cross-platform Yes Partly Partly Yes Yes Yes Reading STDIN Yes Yes Yes -- Yes Yes Reading gzipped file Yes Yes -- -- Yes Yes Writing gzip file Yes -- -- -- Yes -- <p>Note 2: See usage for detailed options of seqkit.</p>"},{"location":"benchmark/#datasets","title":"Datasets","text":"<p>All test data is available here: seqkit-benchmark-data.tar.gz  (2.2G)</p>"},{"location":"benchmark/#dataset_afa-large-number-of-short-sequences","title":"dataset_A.fa - large number of short sequences","text":"<p>Dataset A is reference genomes DNA sequences of gastrointestinal tract from NIH Human Microbiome Project: <code>Gastrointestinal_tract.nuc.fsa</code> (FASTA format, ~2.7G).</p>"},{"location":"benchmark/#dataset_bfa-small-number-of-large-sequences","title":"dataset_B.fa - small number of large sequences","text":"<p>Dataset B is Human genome from ensembl.</p> <ul> <li>Genome DNA:  <code>Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz</code> (Gzipped FASTA file, ~900M) . Decompress it and rename to dataset_B.fa (~2.9G).</li> <li>GTF file:  <code>Homo_sapiens.GRCh38.84.gtf.gz</code> (~44M)</li> <li>BED file: <code>Homo_sapiens.GRCh38.84.bed.gz</code> was converted from <code>Homo_sapiens.GRCh38.84.gtf.gz</code> by  <code>gtf2bed</code>  with command<pre><code>$ zcat Homo_sapiens.GRCh38.84.gtf.gz | gtf2bed --do-not-sort | gzip -c &gt; Homo_sapiens.GRCh38.84.bed.gz\n</code></pre> </li> </ul>"},{"location":"benchmark/#dataset_cfq-illumina-single-end-reads-se100","title":"dataset_C.fq \u2013 Illumina single end reads (SE100)","text":"<p>Dataset C is Illumina single end (SE 100bp) reads file (~2.2G).</p> <p>Summary</p> <pre><code>$ seqkit stat *.fa\nfile          format  type   num_seqs        sum_len  min_len       avg_len      max_len\ndataset_A.fa  FASTA   DNA      67,748  2,807,643,808       56      41,442.5    5,976,145\ndataset_B.fa  FASTA   DNA         194  3,099,750,718      970  15,978,096.5  248,956,422\ndataset_C.fq  FASTQ   DNA   9,186,045    918,604,500      100           100          100\n</code></pre>"},{"location":"benchmark/#sequence-id-list","title":"Sequence ID list","text":"<p>Parts of sequences IDs was sampled and shuffled from original data. They were used in test of extracting sequences by ID list.</p> <p>Commands:</p> <pre><code>$ seqkit sample -p 0.3  dataset_A.fa | seqkit seq --name --only-id | shuf &gt; ids_A.txt\n$ seqkit sample -p 0.3  dataset_B.fa | seqkit seq --name --only-id | shuf &gt; ids_B.txt    \n$ seqkit sample -p 0.03 dataset_C.fq | seqkit seq --name --only-id | shuf &gt; ids_C.txt\n</code></pre> <p>Numbers:</p> <pre><code>$ wc -l ids*.txt\n    20138 ids_A.txt\n    58 ids_B.txt\n2754516 ids_C.txt\n</code></pre>"},{"location":"benchmark/#bed-file","title":"BED file","text":"<p>Only BED data of chromosome 19 was used in test of subsequence with BED file:</p> <pre><code>$ zcat Homo_sapiens.GRCh38.84.bed.gz | grep -E \"^19\" | gzip -c &gt; chr19.bed.gz\n</code></pre>"},{"location":"benchmark/#platform","title":"Platform","text":"<p>PC:</p> <ul> <li>CPU: Intel Core i5-3320M @ 2.60GHz, two cores/4 threads</li> <li>RAM: DDR3 1600MHz, 12GB</li> <li>SSD: SAMSUNG 850 EVO 250G, SATA-3</li> <li>OS: Fedora 24 (Scientific KDE spin),  Kernal: 4.6.4-301.fc24.x86_64</li> </ul> <p>Softwares:</p> <ul> <li>Perl: perl 5, version 22, subversion 2 (v5.22.2) built for x86_64-linux-thread-multi</li> <li>Python: Python 2.7.11 (default, Jul 10 2016, 20:58:20) [GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux2</li> </ul>"},{"location":"benchmark/#tests","title":"Tests","text":"<p>Automatic benchmark and plotting scripts are available at:  https://github.com/shenwei356/seqkit/tree/master/benchmark.</p> <p>All tests were repeated 3 times, and average time and peak memory ware used for plotting.</p> <p>All data were readed once before tests began to minimize the influence of page cache.</p> <p>Output sequences of all softwares were not wrapped to fixed length.</p>"},{"location":"benchmark/#test-1-reverse-complement","title":"Test 1. Reverse Complement","text":"<p><code>revcom_biogo</code> (source,  binary ),  a tool written in Golang (compiled with Go 1.6.3) using biogo  (Version 7ebd71b)  package,  was also used for comparison of FASTA file parsing performance.</p> <p>Note that some softwares (fasta_utilities and biogo) have different converting rules of computing complement sequence on ambiguous bases, there fore the results are different from others.</p> <p>Commands</p>"},{"location":"benchmark/#test-2-extracting-sequences-by-id-list","title":"Test 2. Extracting sequences by ID list","text":"<p>Commands</p>"},{"location":"benchmark/#test-3-sampling-by-number","title":"Test 3. Sampling by number","text":"<p>Note that different softwares have different sampling strategies, the peak memory  depends on size of sampled sequences and the results may not be the same.</p> <p>Commands</p>"},{"location":"benchmark/#test-4-removing-duplicates-by-sequence-content","title":"Test 4. Removing duplicates by sequence content","text":"<p>Commands</p>"},{"location":"benchmark/#test-5-subsequence-with-bed-file","title":"Test 5. Subsequence with BED file","text":"<p>Commands</p>"},{"location":"benchmark/#results","title":"Results","text":"<p>seqkit version: v0.3.1.1</p> <p>FASTA:</p> <p></p> <p>FASTQ:</p> <p></p>"},{"location":"benchmark/#test-of-multiple-threads","title":"Test of multiple threads:","text":"<p>From the results, 2 threads/CPU is enough, so the default threads of seqkit is 2.</p> <p></p>"},{"location":"benchmark/#tests-on-different-file-sizes","title":"Tests on different file sizes","text":"<p>Files are generated by replicating Human genome chr1 for N times.</p> <p></p> Please enable JavaScript to view the comments powered by Disqus."},{"location":"bioinf/","title":"Bioinf","text":""},{"location":"download/","title":"Download","text":"<p>SeqKit is implemented in Go programming language, statically-linked executable binary files are freely available.</p> <p>Please cite:</p> <ol> <li>Wei Shen*, Botond Sipos, and Liuyang Zhao. 2024. SeqKit2: A Swiss Army Knife for Sequence and Alignment Processing. iMeta e191. doi:10.1002/imt2.191. </li> <li>Wei Shen, Shuai Le, Yan Li*, and Fuquan Hu*. SeqKit: a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PLOS ONE. doi:10.1371/journal.pone.0163962. </li> </ol>"},{"location":"download/#current-version","title":"Current Version","text":"<ul> <li>SeqKit v2.9.0 - 2024-11-01 <ul> <li><code>seqkit</code>:<ul> <li>Fix sequence ID parsing with the default regular expression (in this case, we actually use bytes.Index instead) for a rare case: \"xxx\\tyyy zzz\" was wrongly parsed as \"xxx\\tyyy\". #486</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>Fix <code>-G/--non-greedy</code> for tandem repeats, e.g., ATTCGATTCGATTCG (ATTCGx3).</li> </ul> </li> <li><code>seqkit grep/subseq</code>:<ul> <li>Fix negative regions longer than sequence length. #479.</li> </ul> </li> <li><code>seqkit stats</code>:<ul> <li>Add an extra column <code>sum_n</code> to count the number of ambiguous characters. #490</li> </ul> </li> </ul> </li> </ul>"},{"location":"download/#links","title":"Links","text":"OS Arch File, \u4e2d\u56fd\u955c\u50cf Download Count Linux 32-bit seqkit_linux_386.tar.gz,  \u4e2d\u56fd\u955c\u50cf Linux 64-bit seqkit_linux_amd64.tar.gz,  \u4e2d\u56fd\u955c\u50cf Linux arm64 seqkit_linux_arm64.tar.gz,  \u4e2d\u56fd\u955c\u50cf macOS 64-bit seqkit_darwin_amd64.tar.gz,  \u4e2d\u56fd\u955c\u50cf macOS arm64 seqkit_darwin_arm64.tar.gz,  \u4e2d\u56fd\u955c\u50cf Windows 32-bit seqkit_windows_386.exe.tar.gz,  \u4e2d\u56fd\u955c\u50cf Windows 64-bit seqkit_windows_amd64.exe.tar.gz,  \u4e2d\u56fd\u955c\u50cf <p>Notes</p> <ul> <li>please open an issuse to request binaries for other platforms.</li> <li>run <code>seqkit version</code> to check update !!!</li> <li>run <code>seqkit genautocomplete</code> to update shell autocompletion script !!!</li> </ul>"},{"location":"download/#installation","title":"Installation","text":""},{"location":"download/#method-1-download-binaries-latest-stable-version","title":"Method 1: Download binaries (latest stable version)","text":"<p>Just download compressed executable file of your operating system, and decompress it with <code>tar -zxvf *.tar.gz</code> command or other tools. And then:</p> <ol> <li> <p>For Linux-like systems</p> <ol> <li> <p>If you have root privilege simply copy it to <code>/usr/local/bin</code>:</p> <pre><code>sudo cp seqkit /usr/local/bin/\n</code></pre> </li> <li> <p>Or copy to anywhere in the environment variable <code>PATH</code>:</p> <pre><code>mkdir -p $HOME/bin/; cp seqkit $HOME/bin/\n</code></pre> </li> </ol> </li> <li> <p>For windows, just copy <code>seqkit.exe</code> to <code>C:\\WINDOWS\\system32</code>.</p> </li> </ol>"},{"location":"download/#method-2-install-via-conda-latest-stable-version","title":"Method 2: Install via conda (latest stable version)","text":"<pre><code>conda install -c bioconda seqkit\n</code></pre>"},{"location":"download/#method-3-install-via-homebrew-might-not-be-latest-stable-version","title":"Method 3: Install via homebrew (might not be latest stable version)","text":"<pre><code>brew install seqkit\n</code></pre>"},{"location":"download/#method-4-for-go-developer-latest-stabledev-version","title":"Method 4: For Go developer (latest stable/dev version)","text":"<pre><code>go get -u github.com/shenwei356/seqkit/v2/seqkit/\n</code></pre>"},{"location":"download/#method-5-docker-based-installation-might-not-be-latest-stable-versio","title":"Method 5: Docker based installation (might not be latest stable versio)","text":"<p>Install Docker</p> <p>git clone this repo:</p> <pre><code>git clone https://github.com/shenwei356/seqkit\n</code></pre> <p>Run the following commands:</p> <pre><code>cd seqkit\ndocker build -t shenwei356/seqkit .\ndocker run -it shenwei356/seqkit:latest\n</code></pre>"},{"location":"download/#method-6-compiling-from-source-latest-stabledev-version","title":"Method 6: Compiling from source (latest stable/dev version)","text":"<pre><code># ------------------- install golang -----------------\n\n# download Go from https://go.dev/dl\nwget https://go.dev/dl/go1.17.13.linux-amd64.tar.gz\n\ntar -zxf go1.17.13.linux-amd64.tar.gz -C $HOME/\n\n# or \n#   echo \"export PATH=$PATH:$HOME/go/bin\" &gt;&gt; ~/.bashrc\n#   source ~/.bashrc\nexport PATH=$PATH:$HOME/go/bin\n\n\n# ------------- the latest stable version -------------\n\ngo get -v -u github.com/shenwei356/seqkit/seqkit\n\n# The executable binary file is located in:\n#   ~/go/bin/seqkit\n# You can also move it to anywhere in the $PATH\nmkdir -p $HOME/bin\ncp ~/go/bin/seqkit $HOME/bin/\n\n# --------------- the development version --------------\n\ngit clone https://github.com/shenwei356/seqkit\ncd seqkit/seqkit/\ngo build\n\n# The executable binary file is located in:\n#   ./seqkit\n# You can also move it to anywhere in the $PATH\nmkdir -p $HOME/bin\ncp ./seqkit $HOME/bin/\n</code></pre>"},{"location":"download/#shell-completion","title":"Shell-completion","text":"<p>Supported shell: bash|zsh|fish|powershell</p> <p>Bash:</p> <pre><code># generate completion shell\nseqkit genautocomplete --shell bash\n\n# configure if never did.\n# install bash-completion if the \"complete\" command is not found.\necho \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" &gt;&gt; ~/.bash_completion\necho \"source ~/.bash_completion\" &gt;&gt; ~/.bashrc\n</code></pre> <p>Zsh:</p> <pre><code># generate completion shell\nseqkit genautocomplete --shell zsh --file ~/.zfunc/_seqkit\n\n# configure if never did\necho 'fpath=( ~/.zfunc \"${fpath[@]}\" )' &gt;&gt; ~/.zshrc\necho \"autoload -U compinit; compinit\" &gt;&gt; ~/.zshrc\n</code></pre> <p>fish:</p> <pre><code>seqkit genautocomplete --shell fish --file ~/.config/fish/completions/seqkit.fish\n</code></pre>"},{"location":"download/#release-history","title":"Release history","text":"<ul> <li>SeqKit v2.8.2 - 2024-05-17 <ul> <li><code>seqkit amplicon</code>:<ul> <li>Fix a big introduced in v2.7.0. When more than one pairs of primers are given, only the last one is used. #457</li> </ul> </li> <li><code>seqkit translate</code>:<ul> <li>Add option <code>-e/--skip-translate-errors</code> to skip translate error and output empty sequence. #458</li> </ul> </li> <li><code>seqkit split</code>:<ul> <li>Add flag <code>-I/--ignore-case</code> for <code>-i/--by-id</code>. #462</li> </ul> </li> </ul> </li> <li>SeqKit v2.8.1 - 2024-04-07 <ul> <li><code>seqkit sana</code>:<ul> <li>Add support for FASTQ files with IDs in the separator (<code>+</code>, 3rd) lines.. #446, #429, #408</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>Add some docs to show how to keep the original order of sequences when extracting with BED: compress the input FASTA file. #451</li> </ul> </li> </ul> </li> <li>SeqKit v2.8.0 - 2024-03-11 <ul> <li><code>seqkit stats</code>:<ul> <li>Add column <code>N50_num</code>, an alias of L50, #15.</li> </ul> </li> <li><code>seqkit seq/locate/fish/watch</code>:<ul> <li>Removing the flag <code>-V/--validate-seq-length</code>. Now the whole sequence will be checked if <code>-v/--validate-seq</code> is given.</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>Fix the speed problem, introduced in v2.7.0. #439.</li> <li>Slightly faster by reusing objects.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>Change the threshold sequence length for parallelizing complement sequence computation, 1kb-&gt;1Mb.</li> </ul> </li> </ul> </li> <li>SeqKit v2.7.0 - 2024-01-31 <ul> <li><code>seqkit</code>:<ul> <li>Grouping subcommands in help message, which is intuitive for beginners.</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>New flag: <code>-D/--allow-duplicated-patterns</code> for outputting records multiple times when duplicated patterns are given. #427</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>Use the ID regular expression from the option <code>--id-regexp</code> to create FASTA index file. This solves the panic happened for sequences containing tabs in the headers. #432</li> </ul> </li> <li><code>seqkit split/sort/shuffle</code>:<ul> <li>When using the two-pass mode (<code>-2/--two-pass</code>), replace possible tabs in the sequence header.</li> </ul> </li> <li><code>seqkit rmdup</code>:<ul> <li>Write an empty file of duplicate numbers and lists of IDs even if there's no duplicates when using <code>-D/--dup-num-file</code>. #436</li> </ul> </li> <li><code>seqkit stats</code>:<ul> <li>New flag <code>-S/--skip-file-check</code> to skip input file checking when given files or a file list. It's very useful if you run it with millions of files.</li> </ul> </li> </ul> </li> <li>SeqKit v2.6.1 - 2023-11-18 <ul> <li><code>seqkit</code>:<ul> <li>fix panic of nil pointer introduced in v2.6.0, which happens when handling multiple input files and some of them have file sizes of zero.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>fix panic (close of closed channel) when using <code>-v</code> to checking sequences.</li> </ul> </li> </ul> </li> <li>SeqKit v2.6.0 - 2023-11-09 <ul> <li><code>seqkit</code>:<ul> <li>add the shortcut <code>-X</code> for the flag <code>--infile-list</code>.</li> </ul> </li> <li><code>seqkit common</code>:<ul> <li>add a new flag <code>-e/--check-embedded-seqs</code> for detecting embedded sequences.</li> <li>for matching by sequences: reduced the memory occupation and corrected numbers in the log. #416</li> </ul> </li> <li><code>seqkit stat</code>:<ul> <li>add a new column <code>AvgQual</code> for average quality score. #411</li> </ul> </li> <li><code>seqkit split2</code>:<ul> <li>fix the panic for invalid input.</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>add a new flag <code>-R/--region-coord</code> for appending coordinates to sequence ID for <code>-r/--region</code>. #413</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>add a new flag <code>-s/--max-len-to-show</code> to show at most X characters for the search pattern or matched sequences.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>change the nucleotide color theme. #412</li> </ul> </li> </ul> </li> <li>SeqKit v2.5.1 - 2023-08-09 <ul> <li><code>seqkit stats</code>:<ul> <li>fix a concurrency bug (file name error) introduced in v2.5.0. #405</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>sequence/chromosome IDs are case-sensitive now. #400</li> </ul> </li> </ul> </li> <li>SeqKit v2.5.0 - 2023-07-16 <ul> <li>new command <code>seqkit merge-slides</code>: merge sliding windows generated from seqkit sliding. #390</li> <li><code>seqkit stats</code>:<ul> <li>added a new flag <code>-N/--N</code> for appending other N50-like stats as new columns. #393</li> <li>added a progress bar for &gt; 1 input files.</li> <li>write the result of each file immediately (no output buffer) when using <code>-T/--tabular</code>.</li> </ul> </li> <li><code>seqkit translate</code>:<ul> <li>add options <code>-s/--out-subseqs</code> and <code>-m/--min-len</code> to write ORFs longer than <code>x</code> amino acids as individual records. #389</li> </ul> </li> <li><code>seqkit sum</code>:<ul> <li>do not remove possible '*' by default and delete confusing warnings. Thanks to @photocyte. #399</li> <li>added a progress bar for &gt; 1 input files.</li> </ul> </li> <li><code>seqkit pair</code>:<ul> <li>remove the restriction of requiring FASTQ format, i.e., FASTA files are also supported.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>update help messages. #387</li> </ul> </li> <li><code>seqkit fxtab</code>:<ul> <li>faster alphabet computation (<code>-a/--alphabet</code>) with a new data structure. Thanks to @elliotwutingfeng #388</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>accept reverse coordinates in BED/GTF. #392</li> </ul> </li> </ul> </li> <li>SeqKit v2.4.0 - 2023-03-17 <ul> <li><code>seqkit</code>:<ul> <li>support <code>bzip2</code> format. #361</li> <li>support setting compression level for <code>gzip</code>, <code>zstd</code>, and <code>bzip2</code> format via <code>--compress-level</code>. #320</li> <li>the global flag <code>--infile-list</code> accepts stdin (<code>-</code>) now.</li> <li>wrap the help message of flags.</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>do not remove embeded regions when searching with regular expressions. #368</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>fix BED coordinates for amplicons found in the minus strand. #367</li> </ul> </li> <li><code>seqkit split</code>:<ul> <li>fix forgetting to add extension for <code>--two-pass</code>. #332</li> </ul> </li> <li><code>seqkit stats</code>:<ul> <li>fix compute Q1 and Q3 of sequence length for one record. #353</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>fix count number (<code>-C</code>) for matching with mismatch (<code>-m &gt; 0</code>). #370</li> </ul> </li> <li><code>seqkit replace</code>:<ul> <li>add some flags to match partly records to edit; these flags are transplanted from <code>seqkit grep</code>. #348</li> </ul> </li> <li><code>seqkit faidx</code>:<ul> <li>allow empty lines at the end of sequences.</li> </ul> </li> <li><code>seqkit faidx/sort/shuffle/split/subseq</code>:<ul> <li>new flag <code>-U/--update-faidx</code>: update the FASTA index file if it exists, to guarantee the index file matches the FASTA files. #364</li> <li>improve log info and update help message. #365</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>allow filtering sequences of length zero. thanks to @penglbio.</li> </ul> </li> <li><code>seqkit rename</code>:<ul> <li>new flag <code>-s/--separator</code> for setting separator between original ID/name and the counter (default \"_\"). #360</li> <li>new flag <code>-N/--start-num</code> for setting starting count number for duplicated IDs/names (default 2). #360</li> <li>new flag <code>-1/--rename-1st-rec</code> for renaming the first record as well. #360</li> <li>do not append space if there's no description after the sequene ID.</li> </ul> </li> <li><code>seqkit sliding</code>:<ul> <li>new flag <code>-S/--suffix</code> for change the suffix added to the sequence ID (default: \"_sliding\").</li> </ul> </li> </ul> </li> <li>SeqKit v2.3.1 - 2022-09-22 <ul> <li><code>seqkit grep/locate</code>: fix bug of FMIndex building for empty sequences. #321</li> <li><code>seqkit split2</code>: fix bug of splitting two FASTA files. #325</li> <li><code>seqkit faidx</code>: <code>--id-regexp</code> works now.</li> </ul> </li> <li>SeqKit v2.3.0 - 2022-08-12 <ul> <li><code>seqkit grep/rename</code>:<ul> <li>reduce memory comsumption for a lot of searching patterns, and it's faster. #305</li> <li>2X faster <code>-s/--by-seq</code>.</li> </ul> </li> <li><code>seqkit split</code><ul> <li>fix outputting an empty file when the number of sequence equal to the split size. #293</li> <li>add options to set output file prefix and extention. #296</li> </ul> </li> <li><code>seqkit split2</code><ul> <li>reduce memory consumption. #304</li> <li>add options to set output file prefix</li> </ul> </li> <li><code>seqkit stats</code>:<ul> <li>add GC content. #294</li> </ul> </li> </ul> </li> <li> <p>SeqKit v2.2.0 - 2020-03-14 </p> <ul> <li><code>seqkit</code>:<ul> <li>add support of <code>xz</code> and <code>zstd</code> input/output formats. #274</li> <li>fix panic when reading records with header of <code>ID</code> + blanks.</li> </ul> </li> <li>new command <code>seqkit sum</code>: computing message digest for all sequences in FASTA/Q files.   The idea comes from @photocyte and the format borrows from seqhash #262</li> <li>new command <code>seqkit fa2fq</code>: retrieving corresponding FASTQ records by a FASTA file</li> <li><code>seqkit split2</code>:<ul> <li>new flag <code>-e/--extension</code> for forcing compresson or changing compression format. #276</li> <li>support changing output prefix via <code>-o/--out-file</code>. #275</li> </ul> </li> <li><code>seqkit concat</code>: <ul> <li>fix handling of multiple seqs with the same ID in one file. #269</li> <li>performaning out/full join. #270</li> <li>preserve the comments. #271</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>parallelizing <code>-F/--use-fmi</code> and <code>-m</code> for large number of search patterns.</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>new flag <code>-M/--output-mismatches</code> to append the total mismatches and mismatches of 5' end and 3' end. #286</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>detect FASTA/Q symbol <code>@</code> and <code>&gt;</code> in the searching patterns and show warnings.</li> <li>add new flag <code>-C/--count</code>, like <code>grep -c</code> in GNU grep. #267</li> </ul> </li> <li><code>seqkit range</code>:<ul> <li>support removing leading 100 seqs (<code>seqkit range -r 101:-1</code> == <code>tail -n +101</code>). #279</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>report error when no options were given.</li> </ul> </li> <li>update doc:<ul> <li><code>seqkit head</code>: add doc for \"seqkit tail\": <code>seqkit range -N:-1 seqs.fasta</code>. #272</li> <li><code>seqkit rmdup</code>: add the note of only the first record being saved for duplicates. #265</li> </ul> </li> </ul> </li> <li> <p>SeqKit v2.1.0 - 2021-11-15 </p> <ul> <li><code>seqkit seq</code>:<ul> <li>fix filtering by average quality <code>-Q/-R</code>. #257</li> </ul> </li> <li><code>seqkit convert</code>:<ul> <li>fix quality encoding checking, change default value of <code>-N/--thresh-B-in-n-most-common</code> from <code>4</code> to <code>2</code>.  #254 and #239</li> </ul> </li> <li><code>seqkit split</code>:<ul> <li>fix writing an extra empty file when using <code>--two-pass</code>#244</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>fix <code>--bed</code> which fail to recognize strand <code>.</code>.</li> </ul> </li> <li><code>seqkit fq2fa</code>: <ul> <li>faster, and do not wrap sequences.</li> </ul> </li> <li><code>seqkit grep/locate/mutate</code>:<ul> <li>detect unquoted comma and show warning message, e.g., <code>-p 'A{2,}'</code>. #250</li> </ul> </li> </ul> </li> <li>SeqKit v2.0.0 - 2021-08-27 <ul> <li>Performance improvements<ul> <li><code>seqkit</code>:<ul> <li>faster FASTA/Q reading and writing, especially on FASTQ, see the benchmark.<ul> <li>reading (plain text): 4X faster. <code>seqkit stats dataset_C.fq</code></li> <li>reading (gzip files): 45% faster. <code>seqkit stats dataset_C.fq.gz</code></li> <li>reading + writing (plain text): 3.5X faster. <code>seqkit grep -p . -v  dataset_C.fq -o t</code></li> <li>reading + writing (gzip files): 2.2X faster. <code>seqkit grep -p . -v  dataset_C.fq.gz -o t.gz</code></li> </ul> </li> <li>change default value of <code>-j/--threads</code> from 2 to 4, which is faster for writting gzip files.</li> </ul> </li> <li><code>seqkit seq</code>:<ul> <li>fix writing speed, which was slowed down in v0.12.1.</li> </ul> </li> </ul> </li> <li>Breaking changes<ul> <li><code>seqkit grep/rmdup/common</code>: <ul> <li>consider reverse complement sequence by default for comparing by sequence, add flag <code>-P/--only-positive-strand</code>. #215</li> </ul> </li> <li><code>seqkit rename</code>:<ul> <li>rename ID only, do not append original header to new ID. #236</li> </ul> </li> <li><code>seqkit fx2tab</code>:<ul> <li>for <code>-s/--seq-hash</code>: outputing MD5 instead of hash value (integers) of xxhash. #219</li> </ul> </li> </ul> </li> <li>Bugfixes<ul> <li><code>seqkit seq</code>:<ul> <li>fix failing to output gzipped format for file name with extension of <code>.gz</code> since v0.12.1.</li> </ul> </li> <li><code>seqkit tab2fx</code>:<ul> <li>fix bug for very long sequences. #214</li> </ul> </li> <li><code>seqkit fish</code>:<ul> <li>fix range check. #213</li> </ul> </li> <li><code>seqkit grep</code>:<ul> <li>it's not exactly a bug: forgot to use multi-threads for <code>-m</code> &gt; 0.</li> </ul> </li> </ul> </li> <li>New features/enhancements<ul> <li><code>seqkit grep</code>: <ul> <li>allow empty pattern files.</li> </ul> </li> <li><code>seqkit faidx</code>:<ul> <li>support region with <code>begin &gt; end</code>, i.e., returning reverse complement sequence</li> <li>add new flag <code>-l/--region-file</code>:  file containing a list of regions.</li> </ul> </li> <li><code>seqkit fx2tab</code>:<ul> <li>new flag <code>-Q/--no-qual</code> for disabling outputing quality even for FASTQ file. #221</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>new flag <code>-u/--save-unmatched</code> for saving records that do not match any primer.</li> </ul> </li> <li><code>seqkit sort</code>:<ul> <li>new flag <code>-b/--by-bases</code> for sorting by non-gap bases, for multiple sequence alignment files.#216</li> </ul> </li> </ul> </li> </ul> </li> <li>SeqKit v0.16.1 - 2021-05-20 <ul> <li><code>seqkit shuffle --two-pass</code>: fix bug introduced in #173 . #209</li> <li><code>seqkit pair</code>: fix a dangerous bug: when input files are not in current directory, input files were overwritten.</li> </ul> </li> <li>SeqKit v0.16.0 - 2021-04-16 <ul> <li>new command <code>seqkit head-genome</code>:<ul> <li>print sequences of the first genome with common prefixes in name</li> </ul> </li> <li><code>seqkit grep/locate/amplicon -m</code><ul> <li>much faster (300-400x) searching with mismatch allowed by optimizing FM-indexing and parallelization.</li> <li>new flag <code>-I/--immediate-output</code>.</li> </ul> </li> <li><code>seqkit grep/locate</code>:<ul> <li>fix bug of <code>-m</code> when querying contains letters not in alphabet, usually for protein sequences. #178, #179</li> <li>onply search on positive strand when searching unlimited or protein sequences.</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>removing debug info for <code>-r</code> introduced in a0f6b6e. #180</li> </ul> </li> <li><code>seqkit amplicon</code>:<ul> <li>fix bug of <code>-m</code>, when mismatch is allowed.</li> </ul> </li> <li><code>seqkit fx2tab</code>:<ul> <li>new flag <code>-C/--base-count</code> for counting bases. #183</li> </ul> </li> <li><code>seqkit tab2fx</code>:<ul> <li>fix a rare bug. #197</li> </ul> </li> <li><code>seqkit subseq</code>:<ul> <li>fix bug for BED with empty columns. #195</li> </ul> </li> <li><code>seqkit genautocomplete</code>: <ul> <li>support bash|zsh|fish|powershell.</li> </ul> </li> </ul> </li> <li>SeqKit v0.15.0 - 2021-01-12 <ul> <li><code>seqkit grep/locate</code>: update help message.</li> <li><code>seqkit grep</code>: search on both strand when searching by sequence.</li> <li><code>seqkit split2</code>: fix redundant log when using <code>-s</code>.</li> <li><code>seqkit bam</code>: new field <code>RightSoftClipSeq</code>. #172</li> <li><code>seqkit sample -2</code>: remove extra <code>\\n</code>. #173</li> <li><code>seqkit split2 -l</code>: fix bug for splitting by accumulative length, this bug occurs when the first record is longer than <code>-l</code>, no sequences are lost.</li> </ul> </li> <li>SeqKit v0.14.0 - 2020-10-30 <ul> <li>new command <code>seqkit pair</code>: match up paired-end reads from two fastq files, faster than fastq-pair.</li> <li><code>seqkit translate</code>: new flag <code>-F/--append-fram</code> for optional adding frame info to ID. #159</li> <li><code>seqkit stats</code>: reduce memory usage when using <code>-a</code> for calculating N50. #153</li> <li><code>seqkit mutate</code>: fix inserting sequence <code>-i/--insertion</code>,     this bug occurs when <code>insert site</code> is big in some cases, don't worry if no error reported.</li> <li><code>seqkit replace</code>:<ul> <li>new flag <code>-U/--keep-untouched</code>: do not change anything when no value found for the key (only for sequence name).</li> <li>do no support editing FASTQ sequence.</li> </ul> </li> <li><code>seqkit grep/locate</code>: new flag <code>--circular</code> for supporting circular genome. #158</li> </ul> </li> <li>SeqKit v0.13.2 - 2020-07-13 <ul> <li><code>seqkit sana</code>: fix bug causing hanging on empty files. #149</li> </ul> </li> <li>SeqKit v0.13.1 - 2020-07-09 <ul> <li><code>seqkit sana</code>: fix bug causing hanging on empty files. #148</li> </ul> </li> <li>SeqKit v0.13.0 - 2020-07-07 <ul> <li><code>seqkit</code>: fix a rare FASTA/Q parser bug. #127</li> <li><code>seqkit seq</code>: output sequence or quality in single line when <code>-s/--seq</code> or <code>-q/--qual</code> is on. #132</li> <li><code>seqkit translate</code>: delete debug info, #133, and fix typo. #134</li> <li><code>seqkit split2</code>: tiny performance improvement. #137</li> <li><code>seqkit stats</code>: new flag <code>-i/--stdin-label</code> for replacing default \"-\" for stdin. #139</li> <li><code>seqkit fx2tab</code>: new flag <code>-s/--seq-hash</code> for printing hash of sequence (case sensitive). #144</li> <li><code>seqkit amplicon</code>:<ul> <li>fix bug of missing searching reverse strand. #140</li> <li>supporting degenerate bases now. #83</li> <li>new flag <code>-p/--primer-file</code> for reading list of primer pairs. #142</li> <li>new flag <code>--bed</code> for outputing in BED6+1 format. #141</li> </ul> </li> <li>New features and improvements by @bsipos. #130, #147<ul> <li>new command <code>seqkit scat</code>, for real-time robust concatenation of fastx files. </li> <li>Rewrote the parser behind the <code>sana</code> subcommand, now it supports robust parsing of fasta file as well.</li> <li>Added a \"toolbox\" feature to the <code>bam</code> subcommand (<code>-T</code>), which is a collection of filters acting on streams of BAM records configured through a YAML string (see the docs for more).</li> <li>Added the <code>SEQKIT_THREADS</code> environmental variable to override the default number of threads.</li> </ul> </li> </ul> </li> <li>SeqKit v0.12.1 - 2020-04-21 <ul> <li><code>seqkit bam</code>: add colorised and pretty printed output, by @bsipos. #110</li> <li><code>seqkit locate/grep</code>: fix bug of <code>-m</code>, when query contains letters not in subject sequences. #124</li> <li><code>seqkit split2</code>: new flag <code>-l/--by-length</code> for splitting into chunks of N bases.</li> <li><code>seqkit fx2tab</code>:<ul> <li>new flag <code>-I/--case-sensitive</code> for calculating case sensitive base content. #108</li> <li>add missing column name for averge quality for <code>-H -q</code>. #115</li> <li>fix output of <code>-n/--only-name</code>, do not write empty columns of sequence and quality. #104, #115</li> </ul> </li> <li><code>seqkit seq</code>: new flag <code>-k/--color</code>: colorize sequences.</li> </ul> </li> <li>SeqKit v0.12.0 - 2020-02-18 <ul> <li><code>seqkit</code>:<ul> <li>fix checking input file existence.</li> <li>new global flag <code>--infile-list</code> for long list of input files, if given, they are appended to files from cli arguments.</li> </ul> </li> <li><code>seqkit faidx</code>: supporting \"truncated\" (no ending newline charactor) file.</li> <li><code>seqkit seq</code>:<ul> <li>do not force switching on <code>-g</code> when using <code>-m/-M</code>.</li> <li>show recommendation if flag <code>-t/--seq-type</code> is not DNA/RNA when computing complement sequence. #103</li> </ul> </li> <li><code>seqkit translate</code>: supporting multiple frames. #96</li> <li><code>seqkit grep/locate</code>:<ul> <li>add detection and warning for space existing in search pattern/sequence.</li> <li>speed improvement (2X) for <code>-m/--max-mismatch</code>. shenwei356/bwt/issues/3</li> </ul> </li> <li><code>seqkit locate</code>:<ul> <li>new flag <code>-M/--hide-matched</code> for hiding matched sequences. #98</li> <li>new flag <code>-r/--use-regexp</code> for explicitly using regular expression, so improve speed of default <code>index</code> operation. And you have to switch this on if using regexp now. #101</li> <li>new flag <code>-F/--use-fmi</code> for improving search speed for lots of sequence patterns.</li> </ul> </li> <li><code>seqkit rename</code>: making IDs unique across multiple files, and can write into multiple files. #100</li> <li><code>seqkit sample</code>: fix stdin checking for flag <code>-2</code>. #102.</li> <li><code>seqkit rename/split/split2</code>: fix detection of existed outdir.</li> <li><code>split split</code>: fix bug of <code>seqkit split -i -2</code> and parallizing it.</li> <li><code>seqkit version</code>: checking update is optional (<code>-u</code>).</li> </ul> </li> <li>SeqKit v0.11.0 - 2019-09-25 <ul> <li><code>seqkit</code>: fix hanging when reading from truncated gzip file.</li> <li>new commands:<ul> <li><code>seqkit amplicon</code>: retrieve amplicon (or specific region around it) via primer(s).</li> </ul> </li> <li>new commands by @bsipos:<ul> <li><code>seqkit watch</code>: monitoring and online histograms of sequence features.</li> <li><code>seqkit sana</code>: sanitize broken single line fastq files.</li> <li><code>seqkit fish</code>: look for short sequences in larger sequences using local alignment.</li> <li><code>seqkit bam</code>: monitoring and online histograms of BAM record features.</li> </ul> </li> <li><code>seqkit grep/locate</code>: reduce memory occupation when using flag <code>-m/--max-mismatch</code>.</li> <li><code>seqkit seq</code>: fix panic of computing complement sequence for long sequences containing illegal letters without flag <code>-v</code> on. #84</li> </ul> </li> <li>SeqKit v0.10.2 - 2019-07-30 <ul> <li><code>seqkit</code>: fix bug of parsing sequence ID delimited by tab (<code>\\t</code>). #78</li> <li><code>seqkit grep</code>: better logic of <code>--delete-matched</code>.</li> <li><code>seqkit common/rmdup/split</code>: use xxhash to replace MD5 when comparing with sequence, discard flag <code>-m/--md5</code>.</li> <li><code>seqkit stats</code>: new flag <code>-b/--basename</code> for outputting basename instead of full path.</li> </ul> </li> <li>SeqKit v0.10.1 - 2019-02-27 <ul> <li><code>seqkit fx2tab</code>: new option <code>-q/--avg-qual</code> for outputting average read quality. #60</li> <li><code>seqkit grep/locate</code>: fix support of <code>X</code> when using <code>-d/--degenerate</code>. #61</li> <li><code>seqkit translate</code>:<ul> <li>new flag <code>-M/--init-codon-as-M</code> to translate initial codon at beginning to 'M'. #62</li> <li>translates <code>---</code> to <code>-</code> for aligned DNA/RNA, flag <code>-X</code> needed. #63</li> <li>supports codons containing ambiguous bases, e.g., <code>GGN-&gt;G</code>, <code>ATH-&gt;I</code>. #64</li> <li>new flag <code>-l/--list-transl-table</code> to show details of translate table N</li> <li>new flag <code>-l/--list-transl-table-with-amb-codons</code> to show details of translate table N (including ambigugous codons)</li> </ul> </li> <li><code>seqkit split/split2</code>, fix bug of ignoring <code>-O</code> when reading from stdin.</li> </ul> </li> <li>SeqKit v0.10.0 - 2018-12-24 <ul> <li><code>seqkit</code>: report error when input is directory.</li> <li>new command <code>seqkit mutate</code>: edit sequence (point mutation, insertion, deletion).</li> </ul> </li> <li>SeqKit v0.9.3 - 2018-12-02 <ul> <li><code>seqkit stats</code>: fix panic for empty file. #57</li> <li><code>seqkit translate</code>: add flag <code>-x/--allow-unknown-codon</code> to translate unknown codon to <code>X</code>.</li> </ul> </li> <li>SeqKit v0.9.2 - 2018-11-16 <ul> <li><code>seqkit</code>: stricter checking for value of global flag <code>-t/--seq-type</code>.</li> <li><code>seqkit sliding</code>: fix bug for flag <code>-g/--greedy</code>. #54</li> <li><code>seqkit translate</code>: fix bug for frame &lt; 0. #55</li> <li><code>seqkit seq</code>: add TAB to default blank characters (flag <code>-G/--gap-letters</code>), and fix filter result when using flag <code>-g/--remove-gaps</code> along with <code>-m/--min-len</code> or <code>-M/--max-len</code></li> </ul> </li> <li>SeqKit v0.9.1 - 2018-10-12 <ul> <li><code>seqkit faidx</code>: fix bug of retrieving subsequence with multiple regions on same sequence. #48</li> <li><code>seqkit sort/shuffle/split</code>: fix bug when using <code>-2/--two-pass</code> to process <code>.gz</code> files. #52</li> </ul> </li> <li>SeqKit v0.9.0 - 2018-09-26 <ul> <li><code>seqkit</code>: better handle of empty file, no error message shown. #36</li> <li>new subcommand <code>seqkit split2</code>: split sequences into files by size/parts (FASTA, PE/SE FASTQ).  #35</li> <li>new subcommand <code>seqkit translate</code>: translate DNA/RNA to protein sequence. #28</li> <li><code>seqkit sort</code>: fix bug when using <code>-2 -i</code>, and add support for sorting in natural order. #39</li> <li><code>seqkit grep</code> and <code>seqkit locate</code>: add experimental support of mismatch when searching subsequences. #14</li> <li><code>seqkit stats</code>: add stats of Q20 and Q30 for FASTQ. #45</li> </ul> </li> <li>SeqKit v0.8.1 - 2018-06-29 <ul> <li><code>seqkit</code>: do not call <code>pigz</code> or <code>gzip</code> for decompressing gzipped file any more. But you can still utilize <code>pigz</code> or <code>gzip</code> by <code>pigz -d -c seqs.fq.gz | seqkit xxx</code>.</li> <li><code>seqkit subseq</code>: fix bug of missing quality when using <code>--gtf</code> or <code>--bed</code></li> <li><code>seqkit stats</code>: parallelize counting files, it's much faster for lots of small files, especially for files on SSD</li> </ul> </li> <li>SeqKit v0.8.0 - 2018-03-22 </li> <li><code>seqkit</code>, stricter FASTA/Q format requirement, i.e., must starting with <code>&gt;</code> or <code>@</code>.</li> <li><code>seqkit</code>, fix output format for FASTQ files containing zero-length records, yes this happens.</li> <li><code>seqkit</code>, add amino acid code <code>O</code> (pyrrolysine) and <code>U</code> (selenocysteine).</li> <li><code>seqkit replace</code>, add flag <code>--nr-width</code> to fill leading 0s for <code>{nr}</code>, useful for preparing sequence submission (\"&gt;strain_00001 XX\", \"&gt;strain_00002 XX\").</li> <li><code>seqkit subseq</code>, require BED file to be tab-delimited.</li> <li>SeqKit v0.7.2 - 2017-12-03 <ul> <li><code>seqkit tab2fx</code>: fix a concurrency bug that occurs in low proprobability when only 1-column data provided.</li> <li><code>seqkit stats</code>: add quartiles of sequence length</li> <li><code>seqkit faidx</code>: add support for retrieving subsequence using seq ID and region, which is similar with \"samtools faidx\" but has some extra features</li> </ul> </li> <li>SeqKit v0.7.1 - 2017-09-22 <ul> <li><code>seqkit convert</code>: fix bug of read quality containing only 3 or less values.  shenwei356/bio/issues/3</li> <li><code>seqkit stats</code>: add option <code>-T/--tabular</code> to output in machine-friendly tabular format.   #23</li> <li><code>seqkit common</code>: increase speed and decrease memory occupation, and add some notes.</li> <li>fix some typos. #22</li> <li>suggestion: please install pigz to gain better parsing performance for gzipped data.</li> </ul> </li> <li>SeqKit v0.7.0 - 2017-08-12 <ul> <li>add new command <code>convert</code> for converting FASTQ quality encoding between Sanger, Solexa and Illumina. Thanks suggestion from @cviner ( #18). usage &amp; example.</li> <li>add new command <code>range</code> for printing FASTA/Q records in a range (start:end). #19. usage &amp; example.</li> <li>add new command <code>concate</code> for concatenating sequences with same ID from multiple files. usage &amp; example.</li> </ul> </li> <li>SeqKit v0.6.0 - 2017-06-21 <ul> <li>add new command <code>genautocomplete</code> to generate shell autocompletion script! (#17)</li> <li>add new command <code>seqkit dup</code> for duplicating sequences (#16)</li> <li><code>seqkit stats -a</code> does not show L50 which may brings confusion (#15)</li> <li><code>seqkit subseq --bed</code>: more robust for bad BED files</li> </ul> </li> <li>SeqKit v0.5.5 - 2017-05-10 <ul> <li>Increasing speed of reading <code>.gz</code> file by utilizing <code>gzip</code> (1.3X),     it would be much faster if you installed <code>pigz</code> (2X).</li> <li>Fixing colorful output in Windows</li> <li><code>seqkit locate</code>: add flag <code>--gtf</code> and <code>--bed</code> to output GTF/BED6 format,     so the result can be used in <code>seqkit subseq</code>.</li> <li><code>seqkit subseq</code>: fix bug of <code>--bed</code>, add checking coordinate.</li> </ul> </li> <li>SeqKit v0.5.4 - 2017-04-11 <ul> <li><code>seqkit subseq --gtf</code>, add flag <code>--gtf-tag</code> to set tag that's outputted as sequence comment</li> <li>fix <code>seqkit split</code> and <code>seqkit sample</code>: forget not to wrap sequence and quality in output for FASTQ format</li> <li>compile with go1.8.1</li> </ul> </li> <li>SeqKit v0.5.3 - 2017-04-01 <ul> <li><code>seqkit grep</code>: fix bug when using <code>seqkit grep -r -f patternfile</code>:   all records will be retrived due to failing to discarding the blank pattern (<code>\"\"</code>). #11</li> </ul> </li> <li>SeqKit v0.5.2 - 2017-03-24 <ul> <li><code>seqkit stats -a</code> and <code>seqkit seq -g -G</code>: change default gap letters from '- ' to '- .'</li> <li><code>seqkit subseq</code>: fix bug of range overflow when using <code>-d/--down-stream</code>     or <code>-u/--up-stream</code> for retieving subseq using BED (<code>--beb</code>) or GTF (<code>--gtf</code>) file.</li> <li><code>seqkit locate</code>: add flag <code>-G/--non-greedy</code>, non-greedy mode,  faster but may miss motifs overlaping with others.</li> </ul> </li> <li>SeqKit v0.5.1 - 2017-03-12 <ul> <li><code>seqkit restart</code>: fix bug of flag parsing</li> </ul> </li> <li>SeqKit v0.5.0 - 2017-03-11 <ul> <li>new command <code>seqkit restart</code>, for resetting start position for circular genome.</li> <li><code>seqkit sliding</code>: add flag <code>-g/--greedy</code>, exporting last subsequences even shorter than windows size.</li> <li><code>seqkit seq</code>:<ul> <li>add flag <code>-m/--min-len</code> and <code>-M/--max-len</code> to filter sequences by length.</li> <li>rename flag <code>-G/--gap-letter</code> to <code>-G/--gap-letters</code>.</li> </ul> </li> <li><code>seqkit stat</code>:<ul> <li>renamed to <code>seqkit stats</code>, don't worry, old name is still available as an alias.</li> <li>add new flag <code>-a/all</code>, for all statistics, including <code>sum_gap</code>, <code>N50</code>, and <code>L50</code>.</li> </ul> </li> </ul> </li> <li>SeqKit v0.4.5 - 2017-02-26 <ul> <li><code>seqkit seq</code>: fix bug of failing to reverse quality of FASTQ sequence</li> </ul> </li> <li>SeqKit v0.4.4 - 2017-02-17 <ul> <li><code>seqkit locate</code>: fix bug of missing regular-expression motifs containing  non-DNA characters (e.g., <code>ACT.{6,7}CGG</code>) from motif file (<code>-f</code>).</li> <li>compiled with go v1.8.</li> </ul> </li> <li>SeqKit v0.4.3 - 2016-12-22 <ul> <li>fix bug of <code>seqkit stat</code>: <code>min_len</code> always be <code>0</code> in versions: v0.4.0, v0.4.1, v0.4.2</li> </ul> </li> <li>SeqKit v0.4.2 - 2016-12-21 <ul> <li>fix header information of <code>seqkit subseq</code> when restriving up- and down-steam sequences using GTF/BED file.</li> </ul> </li> <li>SeqKit v0.4.1 - 2016-12-16 <ul> <li>enchancement: remove redudant regions for <code>seqkit locate</code>.</li> </ul> </li> <li>SeqKit v0.4.0 - 2016-12-07 <ul> <li>fix bug of <code>seqkit locate</code>, e.g, only find two locations (<code>1-4</code>, <code>7-10</code>, missing <code>4-7</code>) of <code>ACGA</code> in <code>ACGACGACGA</code>.</li> <li>better output of <code>seqkit stat</code> for empty file.</li> </ul> </li> <li>SeqKit v0.3.9 - 2016-12-04 <ul> <li>fix bug of region selection for blank sequences. affected commands include <code>seqkit subseq --region</code>, <code>seqkit grep --region</code>, <code>seqkit split --by-region</code>.</li> <li>compile with go1.8beta1.</li> </ul> </li> <li>SeqKit v0.3.8.1 - 2016-11-25 <ul> <li>enhancement and bugfix of <code>seqkit common</code>: two or more same files allowed, fix log information of number of extracted sequences in the first file.</li> </ul> </li> <li>SeqKit v0.3.8 - 2016-12-24 <ul> <li>enhancement of <code>seqkit common</code>: better handling of files containing replicated sequences</li> </ul> </li> <li>SeqKit v0.3.7 - 2016-12-23 <ul> <li>fix bug in <code>seqkit split --by-id</code> when sequence ID contains invalid characters for system path.</li> <li>add more flags validation for <code>seqkit replace</code>.</li> <li>enhancement: raise error when key pattern matches multiple targes in cases of replacing with key-value files and more controls are added.</li> <li>changes: do not wrap sequence and quality in output for FASTQ  format.</li> </ul> </li> <li>SeqKit v0.3.6 - 2016-11-03 <ul> <li>add new feature for <code>seqkit grep</code>: new flag <code>-R</code> (<code>--region</code>) for specifying sequence region for searching.</li> </ul> </li> <li>SeqKit v0.3.5 - 2016-10-30 <ul> <li>fig bug of <code>seqkit grep</code>: flag <code>-i</code> (<code>--ignore-case</code>) did not work when not using regular expression</li> </ul> </li> <li>SeqKit v0.3.4.1 - 2016-09-21 <ul> <li>improve performance of reading (~10%) and writing (100%) gzip-compressed file by using <code>github.com/klauspost/pgzip</code> package</li> <li>add citation</li> </ul> </li> <li>SeqKit v0.3.4 - 2016-09-17   <ul> <li>bugfix: <code>seq</code> wrongly handles only the first one sequence file when multiple files given</li> <li>new feature: <code>fx2tab</code> can output alphabet letters of a sequence by flag <code>-a</code> (<code>--alphabet</code>)</li> <li>new feature: new flag <code>-K</code> (<code>--keep-key</code>) for <code>replace</code>,  when replacing with key-value file, one can choose keeping the key as value or not.</li> </ul> </li> <li>SeqKit v0.3.3 - 2016-08-18   <ul> <li>fix bug of <code>seqkit replace</code>, wrongly starting from 2 when using <code>{nr}</code>   in <code>-r</code> (<code>--replacement</code>)</li> <li>new feature: <code>seqkit replace</code> supports replacement symbols <code>{nr}</code> (record number)   and <code>{kv}</code> (corresponding value of the key ($1) by key-value file)</li> </ul> </li> <li>SeqKit v0.3.2 - 2016-08-13   <ul> <li>fix bug of <code>seqkit split</code>, error when target file is in a directory.</li> <li>improve performance of <code>seqkit spliding</code> for big sequences, and output   last part even if it's shorter than window sze,   output of FASTQ is also supported.</li> </ul> </li> <li>SeqKit v0.3.1.1 - 2016-08-07   <ul> <li>compile with go1.7rc5, with higher performance and smaller size of binary file</li> </ul> </li> <li>SeqKit v0.3.1 - 2016-08-02   <ul> <li>improve speed of <code>seqkit locate</code></li> </ul> </li> <li>SeqKit v0.3.0 - 2016-07-28   <ul> <li>use fork of github.com/brentp/xopen, using <code>zcat</code> for speedup of .gz file   reading on *nix systems.</li> <li>improve speed of parsing sequence ID when creating FASTA index</li> <li>reduce memory usage of <code>seqkit subseq --gtf</code></li> <li>fix bug of <code>seqkit subseq</code> when using flag <code>--id-ncbi</code></li> <li>fix bug of <code>seqkit split</code>, outdir error</li> <li>fix bug of <code>seqkit seq -p</code>, last base is wrongly failed to convert when   sequence length is odd.</li> <li>add \"sum_len\" result for output of <code>seqkit stat</code></li> </ul> </li> <li>seqkit v0.2.9 - 2016-07-24   <ul> <li>fix minor bug of <code>seqkit split</code> and <code>seqkit shuffle</code>,   header name error due to improper use of pointer</li> <li>add option <code>-O (--out-dir)</code> to <code>seqkit split</code></li> </ul> </li> <li>seqkit v0.2.8 - 2016-07-19   <ul> <li>improve speed of parsing sequence ID, not using regular expression for default <code>--id-regexp</code></li> <li>improve speed of record outputing for small-size sequences</li> <li>fix minor bug: <code>seqkit seq</code> for blank record</li> <li>update benchmark result</li> </ul> </li> <li>seqkit v0.2.7 - 2016-07-18   <ul> <li>reduce memory usage by optimize the outputing of sequences.   detail: using <code>BufferedByteSliceWrapper</code> to resuse bytes.Buffer.</li> <li>reduce memory usage and improve speed by using custom buffered  reading mechanism, instead of using standard library <code>bufio</code>,   which is slow for large genome sequence.</li> <li>discard strategy of \"buffer\" and \"chunk\" of FASTA/Q records,   just parse records one by one.</li> <li>delete global flags <code>-c (--chunk-size)</code> and <code>-b (--buffer-size)</code>.</li> <li>add function testing scripts</li> </ul> </li> <li>seqkit v0.2.6 - 2016-07-01   <ul> <li>fix bug of <code>seqkit subseq</code>: Inplace subseq method leaded to wrong result</li> </ul> </li> <li>seqkit v0.2.5.1 <ul> <li>fix a bug of <code>seqkit subseq</code>: chromesome name was not be converting to lower case when using <code>--gtf</code> or <code>--bed</code></li> </ul> </li> <li>seqkit v0.2.5 - 2016-07-01   <ul> <li>fix a serious bug brought in <code>v0.2.3</code>, using unsafe method to convert <code>string</code> to <code>[]byte</code></li> <li>add awk-like built-in variable of record number (<code>{NR}</code>) for <code>seqkit replace</code></li> </ul> </li> <li>seqkit v0.2.4.1 - 2016-06-12   <ul> <li>fix several bugs from library <code>bio</code>, affected situations:<ul> <li>Locating patterns in sequences by pattern FASTA file: <code>seqkit locate -f</code></li> <li>Reading FASTQ file with record of which the quality starts with <code>+</code></li> </ul> </li> <li>add command <code>version</code></li> </ul> </li> <li>seqkit v0.2.4 - 2016-05-31   <ul> <li>add subcommand <code>head</code></li> </ul> </li> <li>seqkit v0.2.3 - 2016-05-08   <ul> <li>reduce memory occupation by avoid copy data when convert <code>string</code> to <code>[]byte</code></li> <li>speedup reverse-complement by avoid repeatly calling functions</li> </ul> </li> <li>seqkit v0.2.2 - 2016-05-06   <ul> <li>reduce memory occupation of subcommands that use FASTA index</li> </ul> </li> <li>seqkit v0.2.1 - 2016-05-02   <ul> <li>improve performance of outputing.</li> <li>fix bug of <code>seqkit seq -g</code> for FASTA fromat</li> <li>some other minor fix of code and docs</li> <li>update benchmark results</li> </ul> </li> <li>seqkit v0.2.0 - 2016-04-29   <ul> <li>reduce memory usage of writing output</li> <li>fix bug of <code>subseq</code>, <code>shuffle</code>, <code>sort</code> when reading from stdin</li> <li>reduce memory usage of <code>faidx</code></li> <li>make validating sequences an optional option in <code>seq</code> command, it saves some time.</li> </ul> </li> <li>seqkit v0.1.9 - 2016-04-26   <ul> <li>using custom FASTA index file extension: <code>.seqkit.fai</code></li> <li>reducing memory usage of <code>sample --number --two-pass</code></li> <li>change default CPU number to 2 for multi-cpus computer, and 1 for single-CPU computer</li> </ul> </li> <li>seqkit v0.1.8 - 2016-04-24   <ul> <li>add subcommand <code>rename</code> to rename duplicated IDs</li> <li>add subcommand <code>faidx</code> to create FASTA index file</li> <li>utilize faidx to improve performance of <code>subseq</code></li> <li><code>shuffle</code>, <code>sort</code> and split support two-pass mode (by flag <code>-2</code>) with faidx to reduce memory usage.</li> <li>document update</li> </ul> </li> <li>seqkit v0.1.7 - 2016-04-21   <ul> <li>add support for (multi-line) FASTQ format</li> <li>update document, add technical details</li> <li>rename subcommands <code>fa2tab</code> and <code>tab2fa</code> to <code>fx2tab</code> and <code>tab2fx</code></li> <li>add subcommand <code>fq2fa</code></li> <li>add column \"seq_format\" to <code>stat</code></li> <li>add global flag <code>-b</code> (<code>--bufer-size</code>)</li> <li>little change of flag in <code>subseq</code> and some other commands</li> </ul> </li> <li>seqkit v0.1.6 - 2016-04-07   <ul> <li>add subcommand <code>replace</code></li> </ul> </li> <li>seqkit v0.1.5.2 - 2016-04-06   <ul> <li>fix bug of <code>grep</code>, when not using flag <code>-r</code>, flag <code>-i</code> will not take effect.</li> </ul> </li> <li>seqkit v0.1.5.1 <ul> <li>fix result of <code>seqkit sample -n</code></li> <li>fix benchmark script</li> </ul> </li> <li>seqkit v0.1.5 - 2016-03-29   <ul> <li>add global flag <code>--id-ncbi</code></li> <li>add flag <code>-d</code> (<code>--dup-seqs-file</code>) and <code>-D</code> (<code>--dup-num-file</code>) for subcommand <code>rmdup</code></li> <li>make using MD5 as an optional flag <code>-m</code> (<code>--md5</code>) in subcommand <code>rmdup</code> and <code>common</code></li> <li>fix file name suffix of <code>seqkit split</code> result</li> <li>minor modification of <code>sliding</code> output</li> </ul> </li> <li>seqkit v0.1.4.1 - 2016-03-27   <ul> <li>change alignment of <code>stat</code> output</li> <li>preciser CPUs number control</li> </ul> </li> <li>seqkit v0.1.4 - 2016-03-25   <ul> <li>add subcommand <code>sort</code></li> <li>improve subcommand <code>subseq</code>: supporting of getting subsequences by GTF and BED files</li> <li>change name format of <code>sliding</code> result</li> <li>prettier output of <code>stat</code></li> </ul> </li> <li>seqkit v0.1.3.1 - 2016-03-16   <ul> <li>Performance improvement by reducing time of cleaning spaces</li> <li>Document update</li> </ul> </li> <li>seqkit v0.1.3 - 2016-03-15   <ul> <li>Further performance improvement</li> <li>Rename sub command <code>extract</code> to <code>grep</code></li> <li>Change default value of flag <code>--threads</code> back CPU number of current device,   change default value of flag <code>--chunk-size</code> back 10000 sequences.</li> <li>Update benchmark</li> </ul> </li> <li>seqkit v0.1.2 - 2016-03-14   <ul> <li>Add flag <code>--dna2rna</code> and <code>--rna2dna</code> to subcommand <code>seq</code>.</li> </ul> </li> <li>seqkit v0.1.1 - 2016-03-13   <ul> <li>5.5X speedup of FASTA file parsing by avoid using regular expression to remove spaces (detail ) and using slice indexing instead of map to validate letters (detail)</li> <li>Change default value of global flag <code>-- thread</code> to 1. Since most of the subcommands are I/O intensive,  For computation intensive jobs, like extract and locate, you may set a bigger value.</li> <li>Change default value of global flag <code>--chunk-size</code> to 100.</li> <li>Add subcommand <code>stat</code></li> <li>Fix bug of failing to automatically detect alphabet when only one record in file.</li> </ul> </li> <li>seqkit v0.1 - 2016-03-11   <ul> <li>first release of seqkit</li> </ul> </li> </ul> Please enable JavaScript to view the comments powered by Disqus."},{"location":"faq/","title":"FAQs on FASTA/Q manipulations","text":"<p>This page was originally written as the section FASTA/Q manipulations of The Biostar Handbook: A Beginner's Guide to Bioinformatics (discussion on Biostars.org).</p> <p>This page illustrates common FASTA/Q manipulations using SeqKit. Some other utilities, including csvtk (CSV/TSV toolkit) and shell commands were also used.</p> <p>Note: SeqKit seamlessly support FASTA and FASTQ formats both in their original form or in stored in gzipped compressed format. We list FASTA or FASTQ depending on the more common usage but you can always use it on the other type as well.</p> <ul> <li>Example data</li> <li>How to produce an overview of FASTQ files?</li> <li>How to get GC content of every sequence in FASTA/Q file?</li> <li>How to extract sequences subset from FASTA/Q file with name/ID list file?</li> <li>How to find FASTA/Q sequences containing degenerate bases and locate them?</li> <li>How to remove duplicated FASTA/Q records with same sequences?</li> <li>How to locate motif/subsequence/enzyme digest site in FASTA/Q sequence?</li> <li>How to sort huge number of FASTA sequences by length?</li> <li>How to split FASTA sequences according to information in header?</li> <li>How to search and replace FASTA header with known character strings from a text file?</li> <li>How to extract paired reads from two paired-end reads file?</li> <li>How to concatenate two FASTA sequences in to one?</li> </ul>"},{"location":"faq/#example-data","title":"Example data","text":"<p>One FASTQ file (sample reads, 1M) and two FASTA files (Virus DNA and protein sequences from NCBI RefSeq database, 60+40M) are used.</p> <pre><code>wget http://data.biostarhandbook.com/reads/duplicated-reads.fq.gz\nwget ftp://ftp.ncbi.nih.gov/refseq/release/viral/viral.1.1.genomic.fna.gz\nwget ftp://ftp.ncbi.nih.gov/refseq/release/viral/viral.1.protein.faa.gz\n</code></pre>"},{"location":"faq/#how-to-produce-an-overview-of-fastq-files","title":"How to produce an overview of FASTQ files?","text":"<p>Sequence format and type are automatically detected.</p> <pre><code>$ seqkit stat *.gz\nfile                      format  type     num_seqs      sum_len  min_len   avg_len    max_len\nduplicated-reads.fq.gz    FASTQ   DNA        15,000    1,515,000      101       101        101\nviral.1.1.genomic.fna.gz  FASTA   DNA         7,048  203,325,120      200  28,848.6  2,473,870\nviral.1.protein.faa.gz    FASTA   Protein   252,611   62,024,702        5     245.5      8,960\n</code></pre>"},{"location":"faq/#how-to-get-gc-content-of-every-sequence-in-fastaq-file","title":"How to get GC content of every sequence in FASTA/Q file?","text":"<p><code>seqkit fx2tab</code> converts FASTA/Q to 3-column tabular format (1th: name/ID, 2nd: sequence, 3rd: quality), and can also provides various information in new columns, including sequence length, GC content/GC skew, alphabet.</p> <p>GC content:</p> <pre><code>$ seqkit fx2tab --name --only-id --gc viral*.fna.gz\ngi|526245010|ref|NC_021865.1|                   40.94\ngi|460042095|ref|NC_020479.1|                   41.82\ngi|526244636|ref|NC_021857.1|                   49.17\n</code></pre> <p>Custom bases (A, C and A+C):</p> <pre><code>$ seqkit fx2tab -H -n -i -B a -B c -B ac viral.1.1.genomic.fna.gz\n#name   seq     qual    a       c       ac\ngi|526245010|ref|NC_021865.1|                   33.20   18.24   51.44\ngi|460042095|ref|NC_020479.1|                   32.57   19.63   52.20\ngi|526244636|ref|NC_021857.1|                   25.52   23.06   48.59\n</code></pre>"},{"location":"faq/#how-to-extract-sequences-subset-from-fastaq-file-with-nameid-list-file","title":"How to extract sequences subset from FASTA/Q file with name/ID list file?","text":"<p>This is a frequently used manipulation. Let's create a sample ID list file, which may also come from other way like mapping result.</p> <pre><code>$ seqkit sample --proportion 0.001  duplicated-reads.fq.gz \\\n    | seqkit seq --name --only-id &gt; id.txt\n</code></pre> <p>ID list file:</p> <pre><code>$ head id.txt\nSRR1972739.2996\nSRR1972739.3044\nSRR1972739.3562\n</code></pre> <p>Searching by ID list file:</p> <pre><code>$ seqkit grep --pattern-file id.txt duplicated-reads.fq.gz \\\n    &gt; duplicated-reads.subset.fq.gz\n</code></pre>"},{"location":"faq/#how-to-find-fastaq-sequences-containing-degenerate-bases-and-locate-them","title":"How to find FASTA/Q sequences containing degenerate bases and locate them?","text":"<p><code>seqkit fx2tab</code> converts FASTA/Q to tabular format and can output the sequence alphabet in a new column. And then text searching tools can be used to filter the table.</p> <pre><code>$ seqkit fx2tab -n -i -a viral.1.1.genomic.fna.gz \\\n    | csvtk -H -t grep -f 4 -r -i -p \"[^ACGT]\"\ngi|446730228|ref|NC_019782.1|                   ACGNT\ngi|557940284|ref|NC_022800.1|                   ACGKT\ngi|564292828|ref|NC_023009.1|                   ACGNT\n</code></pre> <p>Long-option version of the command:</p> <pre><code>$ seqkit fx2tab --name --only-id --alphabet  viral.1.1.genomic.fna.gz \\\n    | csvtk --no-header-row --tabs grep --fields 4 --use-regexp --ignore-case --pattern \"[^ACGT]\"\n</code></pre> <p>You can then exclude these sequences with <code>seqkit grep</code>:</p> <pre><code># save the sequenece IDs.\n$ seqkit fx2tab -n -i -a viral.1.1.genomic.fna.gz \\\n    | csvtk -H -t grep -f 4 -r -i -p \"[^ACGT]\" | csvtk -H -t cut -f 1 &gt; id2.txt\n\n# search and exclude.\n$ seqkit grep --pattern-file id2.txt --invert-match viral.1.1.genomic.fna.gz &gt; clean.fa\n</code></pre> <p>Or locate the degenerate bases, e.g, <code>N</code> and <code>K</code></p> <pre><code>$ seqkit grep --pattern-file id2.txt viral.1.1.genomic.fna.gz \\\n    | seqkit locate --ignore-case --only-positive-strand --pattern K+ --pattern N+\nseqID   patternName     pattern strand  start   end     matched\ngi|564292828|ref|NC_023009.1|   N+      N+      +       87972   87972   N\ngi|564292828|ref|NC_023009.1|   N+      N+      +       100983  100983  N\ngi|557307918|ref|NC_022755.1|   K+      K+      +       1788    1788    K\ngi|557307918|ref|NC_022755.1|   K+      K+      +       4044    4044    K\ngi|589287065|ref|NC_023585.1|   K+      K+      +       28296   28296   K\ngi|590911929|ref|NC_023639.1|   N+      N+      +       741654  741753  NNNNNNNNNNNNNNNNNNNNNNNNNNN\n</code></pre>"},{"location":"faq/#how-to-remove-duplicated-fastaq-records-with-same-sequences","title":"How to remove duplicated FASTA/Q records with same sequences?","text":"<pre><code>$ seqkit rmdup --by-seq --ignore-case duplicated-reads.fq.gz &gt; duplicated-reads.uniq.fq.gz\n</code></pre> <p>If the FASTA/Q file is very large, please switch on flag <code>-m/--md5</code>, which use MD5 instead of original seqs to reduce memory usage when comparing by sequences.</p> <p>You can also deduplicate according to sequence ID (default) or full name (<code>--by-name</code>).</p>"},{"location":"faq/#how-to-locate-motifsubsequenceenzyme-digest-site-in-fastaq-sequence","title":"How to locate motif/subsequence/enzyme digest site in FASTA/Q sequence?","text":"<p>Related posts: Question: Count and location of strings in fastq file reads, Question: Finding TATAWAA in sequence .</p> <p>Assuming a list of motifs (enzyme digest sites) in FASTA format to be located:</p> <pre><code>$ cat enzymes.fa\n&gt;EcoRI\nGAATTC\n&gt;MmeI\nTCCRAC\n&gt;SacI\nGAGCTC\n&gt;XcmI\nCCANNNNNNNNNTGG\n</code></pre> <p>Flag <code>--degenerate</code> is on because patterns contain degenerate bases. Command:</p> <pre><code>$ seqkit locate --degenerate --ignore-case --pattern-file enzymes.fa viral.1.1.genomic.fna.gz\n</code></pre> <p>Sample output (simplified and reformated by <code>csvtk -t uniq -f 3 | csvtk -t pretty</code>)</p> <pre><code>seqID                           patternName   pattern           strand   start   end     matched\ngi|526245010|ref|NC_021865.1|   MmeI          TCCRAC            +        1816    1821    TCCGAC\ngi|526245010|ref|NC_021865.1|   SacI          GAGCTC            +        19506   19511   GAGCTC\ngi|526245010|ref|NC_021865.1|   XcmI          CCANNNNNNNNNTGG   +        2221    2235    CCATATTTAGTGTGG\n</code></pre>"},{"location":"faq/#how-to-sort-huge-number-of-fasta-sequences-by-length","title":"How to sort huge number of FASTA sequences by length?","text":"<p>Sorting FASTA file in order of sequence size (small to large).</p> <pre><code>$ seqkit sort --by-length viral.1.1.genomic.fna.gz &gt; viral.1.1.genomic.sorted.fa\n</code></pre> <p>If the files are too big, use flag <code>--two-pass</code> which consumes lesser memory.</p> <pre><code>$ seqkit sort --by-length --two-pass viral.1.1.genomic.fna.gz &gt; viral.1.1.genomic.sorted.fa\n</code></pre> <p>You can also sort by sequence ID (default), full header (<code>--by-name</code>) or sequence content (<code>--by-seq</code>).</p>"},{"location":"faq/#how-to-split-fasta-sequences-according-to-information-in-header","title":"How to split FASTA sequences according to information in header?","text":"<p>Related posts: Question: extract same all similar sequences in FASTA based on the header .</p> <p>For example, FASTA header line of <code>viral.1.protein.faa.gz</code> contain species name in square brackets.</p> <p>Overview of FASTA Headers:</p> <pre><code>$ seqkit head -n 3 viral.1.protein.faa.gz | seqkit seq --name\ngi|526245011|ref|YP_008320337.1| terminase small subunit [Paenibacillus phage phiIBB_Pl23]\ngi|526245012|ref|YP_008320338.1| terminase large subunit [Paenibacillus phage phiIBB_Pl23]\ngi|526245013|ref|YP_008320339.1| portal protein [Paenibacillus phage phiIBB_Pl23]\n</code></pre> <p><code>seqkit split</code> can split FASTA/Q files according to ID, number of parts, size of every parts, and sequence region. In this case, we'll split according to sequence ID (species names) which can be specified by flag <code>--id-regexp</code>.</p> <p>Default ID:</p> <pre><code>$ seqkit head -n 3 viral.1.protein.faa.gz \\\n    | seqkit seq --name --only-id\ngi|526245011|ref|YP_008320337.1|\ngi|526245012|ref|YP_008320338.1|\ngi|526245013|ref|YP_008320339.1|\n</code></pre> <p>New ID:</p> <pre><code>$ seqkit head -n 3 viral.1.protein.faa.gz \\\n    | seqkit seq --name --only-id --id-regexp \"\\[(.+)\\]\"\nPaenibacillus phage phiIBB_Pl23\nPaenibacillus phage phiIBB_Pl23\nPaenibacillus phage phiIBB_Pl23\n</code></pre> <p>Split:</p> <pre><code>$ seqkit split --by-id --id-regexp \"\\[(.+)\\]\" viral.1.protein.faa.gz\n</code></pre>"},{"location":"faq/#how-to-search-and-replace-fasta-header-with-known-character-strings-from-a-text-file","title":"How to search and replace FASTA header with known character strings from a text file?","text":"<p>Related posts: Question: Replace names in FASTA file with a known character string from a text file, Question: Fasta header, search and replace...? .</p> <p><code>seqKit replace</code> can find substrings in FASTA/Q header with regular expression and replace them with strings or corresponding values of found substrings provided by the tab-delimited key-value file.</p> <p>For example, to unify names of protein with unknown functions, we want to rename \"hypothetical\" to \"putative\" in lower case. The replacing rules are listed below in tab-delimited file:</p> <pre><code>$ cat changes.tsv\nHypothetical    putative\nhypothetical    putative\nPutative        putative\n</code></pre> <p>Overview the FASTA headers containing \"hypothetical\":</p> <pre><code>$ seqkit grep --by-name --use-regexp --ignore-case --pattern hypothetical viral.1.protein.faa.gz \\\n    | seqkit head -n 3 | seqkit seq --name\ngi|526245016|ref|YP_008320342.1| hypothetical protein IBBPl23_06 [Paenibacillus phage phiIBB_Pl23]\ngi|526245019|ref|YP_008320345.1| hypothetical protein IBBPl23_09 [Paenibacillus phage phiIBB_Pl23]\ngi|526245020|ref|YP_008320346.1| hypothetical protein IBBPl23_10 [Paenibacillus phage phiIBB_Pl23]\n</code></pre> <p>A regular expression, <code>^([^ ]+ )(\\w+)</code>, was used to specify the key to be replaced, which is the first word after sequence ID in this case. Note that we also capture the ID (<code>^([^ ]+ )</code>) so we can restore it in \"replacement\" with capture variable  <code>${1}</code> (robuster than <code>$1</code>). And flag <code>-I/--key-capt-idx</code> (default: 1) is set to 2 because the key <code>(\\w+)</code> is the second captured match. Command:</p> <pre><code>$ seqkit replace --kv-file changes.tsv --pattern \"^([^ ]+ )(\\w+) \" \\\n    --replacement \"\\${1}{kv} \" --key-capt-idx 2 --keep-key viral.1.protein.faa.gz &gt; renamed.fa\n</code></pre>"},{"location":"faq/#how-to-extract-paired-reads-from-two-paired-end-reads-file","title":"How to extract paired reads from two paired-end reads file?","text":"<p>Use seqkit pair.</p>"},{"location":"faq/#how-to-concatenate-two-fasta-sequences-in-to-one","title":"How to concatenate two FASTA sequences in to one?","text":"<p>Related posts: Combining two fasta sequences into one</p> <p>Data (not in same order):</p> <pre><code>$ cat 1.fa\n&gt;seq1\naaaaa\n&gt;seq2\nccccc\n&gt;seq3\nggggg\n\n$ cat 2.fa\n&gt;seq3\nTTTTT\n&gt;seq2\nGGGGG\n&gt;seq1\nCCCCC\n</code></pre> <p>Just one command:</p> <pre><code>$ seqkit concat 1.fa 2.fa\n&gt;seq1\naaaaaCCCCC\n&gt;seq2\ncccccGGGGG\n&gt;seq3\ngggggTTTTT\n</code></pre>"},{"location":"note/","title":"Notes","text":""},{"location":"note/#sequence-parsing-strategies","title":"Sequence Parsing Strategies","text":"<p>In general, most FASTA/Q file processing softwares read one sequence record and process it immediately, and then read the next one (Fig 1A). The early version of SeqKit adopted this strategy too; however, our sequence parsing algorithm was not efficient. Therefore, we tried another strategy by using a separated thread to non-blocking read and parse sequences (Fig 1B): The sequence parsing thread parsed fixed number of sequences and packaged them into Chunks, then one or more chunks were buffered in memory so they could be paralleled processed by one or more processing threads. This strategy saved time by reducing the waiting time between parsing sequences and by paralleled processing of sequences in some cases. However, this strategy consumes more memories especially for long sequences like human genome and it is hard to find an optimum combination of buffer and chunk size for different scales of sequences. </p> <p>During the major revision, we kept on optimizing the sequence parsing algorithm and several optimizations had been made (Seqkit v0.2.7). The most important one is using custom buffered file reading algorithm instead of using standard library \u201cbufio\u201d which is not efficient in memory for large sequences. This increased the speed and lowered memory usage significantly. Note that the \u201cbuffer\u201d here is different from the previously mentioned one in Fig 1B, this \u201cbuffer\u201d evolved in the low level processing of file reading. At last, the FASTA/Q format parsing algorithm was significant improved by factor of about 3\u00d7 in speed and the memory usage was only 1/2 compared to SeqKit version 0.2.4 in the first manuscript (this was not descripted in the manuscript). Therefore, we discarded the strategy of \u201cbuffer and chunk\u201d and adopt the serially reading strategy again (Fig 1C). The only difference between Fig C and Fig A is that the time of parsing one sequence is much shorter.</p> <p></p> <p>Figure 1 Illustration of FASTA/Q file parsing strategies. (A) and (C) Main thread parses one sequence, waits (blocked) it to be processed and then parses next one. (B) Sequence parsing thread continuously (non-blocked) parses sequences and passes them to main thread. The width of rectangles representing sequence parsing and sequence processing is proportional with running time. Sequence parsing speeds in (A) and (B) are the same, which are both much slower than that in (C). The speeds of sequence processing are identical in (A), (B) and (C). In (B), chunks of sequences in buffer can be processed in parallel, but most of the time the main thread needs to serially manipulate the sequences.</p>"},{"location":"note/#effect-of-random-seed-on-results-of-seqkit-sample","title":"Effect of random seed on results of <code>seqkit sample</code>","text":"<p><code>seqkit sample</code> supports FASTA/Q sampling by proportion or amount.</p> <ul> <li>For sampling by proportion (<code>P</code>), SeqKit returns a record if a random number (<code>[0, 1</code>])   is less than <code>P</code>.</li> <li>For sampling by amount (<code>N</code>), Seqkit firstly gets the total amount of records,   and compute the proportion (<code>P</code>), and sampling by proportion (<code>P</code>).   Cause the generated random number is pseudorandom   and affected by the random seed (<code>-s/--rand-seed</code>), the number of sampled records   may not be equal to <code>N</code>.</li> </ul> <p>Here we evaluate the effect of random seed on <code>seqkit sample</code> results. Used softwares</p> <ul> <li>csvtk: https://github.com/shenwei356/csvtk</li> <li>rush: https://github.com/shenwei356/rush</li> </ul>"},{"location":"note/#amount-distribution-of-sampled-records","title":"Amount distribution of sampled records","text":"<p>1000 FASTA records with IDs of <code>1</code>, <code>2</code>, ..., <code>1000</code> were sampled by proportion of <code>0.1</code> with random seeds from <code>1</code> to <code>1000</code>, and the distribution of number of sampled records was plotted in boxplot.</p> <pre><code>seq 1000 \\\n    | rush 'seq 1000 | csvtk -t -H mutate | seqkit tab2fx \\\n        | seqkit sample -p 0.1 -s {} \\\n        | seqkit fx2tab | wc -l' \\\n    &gt; ns.txt\n\ncat ns.txt | csvtk -H -t plot box -f 1 --horiz --height 2 \\\n    --xlab \"# of sampled records\" \\\n    &gt; ns.png\n</code></pre> <p></p>"},{"location":"note/#location-distribution-of-sampled-records","title":"Location distribution of sampled records","text":"<p>1000 FASTA records with IDs of <code>1</code>, <code>2</code>, ..., <code>1000</code> were sampled by proportion of <code>0.1</code> with random seeds from <code>1</code> to <code>10</code>. The record IDs (<code>x axis</code>) was used to plot a scatter plot, which show the location distribution of sampled records. The ideal distribution would produces a straight line in the plot.</p> <pre><code>seq 10 \\\n    | rush 'seq 1000 | csvtk -t -H mutate | seqkit tab2fx \\\n        | seqkit sample -p 0.1 -s {} \\\n        | seqkit fx2tab | csvtk -H -t mutate \\\n        | csvtk -H -t replace -f 2 -p '.+' -r '{nr}' \\\n        | csvtk -H -t replace -f 4 -p '.+' -r {}' \\\n    | csvtk -H -t plot line -x 1 -y 2 -g 4 \\\n        --xlab \"sampled locations\" --ylab \"new ID of sampled records\" \\\n        --title \"location distribution of sampled records\" \\\n        --width 8 --height 6 --point-size 2 \\\n    &gt; pos.png\n</code></pre> <p></p> Please enable JavaScript to view the comments powered by Disqus."},{"location":"tutorial/","title":"Tutorial","text":""},{"location":"tutorial/#real-time-monitoring-mapping-results-with-seqkit-bam-in-time-critical-situations","title":"Real-time monitoring mapping results with seqkit bam in time-critical situations","text":"<p>Some of SeqKit subcommands, including <code>watch</code>, <code>fish</code>, <code>scat</code>, <code>bam</code>, aid the real-time, streaming processing of data in FASTQ/FASTA and BAM formats, enabling the development of analysis pipelines in time-critical situations. To illustrate the possibilities opened up by the these features, we present a small pipeline which maps a dataset of SARS-CoV-2 amplicon reads to the reference sequence using <code>minimap2</code> as they are being downloaded. The stream of SAM alignments is then converted into a stream of BAM records and passed to <code>seqkit bam</code>, which filters for records with aligned reference lengths between <code>310</code> and <code>410</code> and displays the distribution of aligned reference lengths after every <code>5000</code> records also saving it to a PDF file. The next element of the pipeline is another instance of the <code>seqkit bam</code> subcommand, which displays the distribution of alignment accuracies after every 10000 records. The stream of BAM records is then passed on to samtools sort to produce the final sorted BAM file. Finally, <code>seqkit bam</code> is used to display detailed alignment statistics from the final sorted BAM file in a pretty format.</p> <pre><code>#!/bin/bash\nset -o nounset\n\n# Define reference and data URL:\nREF_URL=\"https://www.ncbi.nlm.nih.gov/sviewer/viewer.cgi?tool=portal&amp;save=file&amp;log$=seqview&amp;db=nuccore&amp;report=fasta&amp;id=1798174254&amp;extrafeat=null&amp;conwithfeat=on&amp;hide-cdd=on&amp;ncbi_phid=CE8B108356DDCF110000000005B10489\"\nDATA_URL=\"http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR145/055/SRR14560555/SRR14560555_1.fastq.gz\"\n\n# Map SARS-CoV-2 amplicon reads to reference followed by these steps:\n# - keep only primary reads with mapping quality greater than 1.\n# - Keep only alignments which align to a reference segments with lengths between 310 and 420.\n#   Show an approximate histogram of aligned reference lengths after every 5000 records.\n# - Show an approximate histogram of alignment accuracy after every 10000 records.\n# - Pipe the BAM into samtools for sorting.\nminimap2 -K 10M -t 8 -ax map-ont \\\n        &lt;(wget -q -O - \"$REF_URL\") \\\n        &lt;(wget -q -O - \"$DATA_URL\") 2&gt;/dev/null \\\n    | samtools view -F 2304 -q 1 -b - \\\n    | seqkit bam -O aln_len.pdf -f RefAln -m 310 -M 410 -p 5000 -x - \\\n    | seqkit bam -O aln_acc.pdf -f Acc  -p 10000 -x - \\\n    | samtools sort -o sars_artic_sorted.bam -\n\n# Show statistics from the sorted BAM:\nseqkit bam -s -k sars_artic_sorted.bam\n</code></pre>"},{"location":"tutorial/#removing-duplicated-and-nested-sequences","title":"Removing duplicated and nested sequences","text":"<p>https://twitter.com/bentemperton/status/1673999868933599232</p> <p>sample data</p> <pre><code>$ cat contigs.fa\n&gt;big\nACTGACGATCGATACGCAGCACAGCAG\n&gt;small_in_big_rc\nTGCTGCGTATCG\n&gt;small2\nACTACGACTACGACT\n&gt;small2_alias\nACTACGACTACGACT\n&gt;small2_rc\nAGTCGTAGTCGTAGT\n&gt;another\nACTAACGA\n\n$ seqkit fx2tab -Q contigs.fa  | csvtk pretty -Ht -W 40 --clip\nbig               ACTGACGATCGATACGCAGCACAGCAG\nsmall_in_big_rc   TGCTGCGTATCG\nsmall2            ACTACGACTACGACT\nsmall2_alias      ACTACGACTACGACT\nsmall2_rc         AGTCGTAGTCGTAGT\nanother           ACTAACGA\n</code></pre> <p>Step 1. remove exactly duplicated sequences.</p> <pre><code>$ seqkit rmdup -s -i contigs.fa -o contigs.uniq1.fa\n[INFO] 2 duplicated records removed\n</code></pre> <p>Step 2. remove nested seqs.</p> <pre><code># pair-wise exactly searching\nseqkit locate -M -f contigs.uniq1.fa contigs.uniq1.fa -o match.tsv\n\n$ csvtk pretty  -W 40 --clip -t match.tsv\nseqID             patternName       pattern                       strand   start   end\n---------------   ---------------   ---------------------------   ------   -----   ---\nbig               small_in_big_rc   TGCTGCGTATCG                  -        10      21\nbig               big               ACTGACGATCGATACGCAGCACAGCAG   +        1       27\nsmall_in_big_rc   small_in_big_rc   TGCTGCGTATCG                  +        1       12\nsmall2            small2            ACTACGACTACGACT               +        1       15\nanother           another           ACTAACGA                      +        1       8\n\n# IDs of embeded/nested sequences\n$ sed 1d match.tsv \\\n    | awk '$2 != $1' \\\n    | cut -f 2 \\\n    | tee nested.txt\nsmall_in_big_rc\n\n# remove nested sequences\n$ seqkit grep -v -f nested.txt contigs.uniq1.fa \\\n    -o contigs.uniq2.fa\n</code></pre> <p>Result</p> <pre><code>$ seqkit fx2tab -Q contigs.uniq2.fa | csvtk pretty -Ht -W 40 --clip\nbig       ACTGACGATCGATACGCAGCACAGCAG\nsmall2    ACTACGACTACGACT\nanother   ACTAACGA\n</code></pre>"},{"location":"tutorial/#some-manipulations-on-big-genomes","title":"Some manipulations on big genomes","text":"<p>A script memusg is used to check the peek memory usage of seqkit. Usage: <code>memusg [-t] command</code>.</p> <ol> <li> <p>Human genome</p> <pre><code>$ seqkit stat hsa.fa\nfile    format  type  num_seqs        sum_len  min_len       avg_len      max_len\nhsa.fa  FASTA   DNA        194  3,099,750,718      970  15,978,096.5  248,956,422\n</code></pre> </li> <li> <p>Build FASTA index (optional, when using flag <code>-2</code> (<code>--two-pass</code>),    some commands will automaticlly build it).    For some commands, including <code>subseq</code>, <code>split</code>, <code>sort</code> and <code>shuffle</code>,    when input files are (plain or gzipped) FASTA files or stdin,    FASTA index would be optional used for    rapid acccess of sequences and reducing memory occupation.    ATTENTION: the <code>.seqkit.fai</code> file created by SeqKit is a little different from .fai file    created by samtools. SeqKit uses full sequence head instead of just ID as key.</p> <pre><code>$ memusg -t seqkit faidx --id-regexp \"^(.+)$\"  hsa.fa -o hsa.fa.seqkit.fai\n\nelapsed time: 10.011s\npeak rss: 177.21 MB\n</code></pre> <p>Create common .fai file:</p> <pre><code>$ memusg -t seqkit faidx hsa.fa -o hsa.fa.fai2\n\nelapsed time: 10.454s\npeak rss: 172.82 MB\n</code></pre> <p>Performance of samtools:</p> <pre><code>$ memusg -t samtools faidx hsa.fa\n\nelapsed time: 9.574s\npeak rss: 1.45 MB\n</code></pre> <p>Exactly same content:</p> <pre><code>$ md5sum hsa.fa.fai*\n21e0c25b4d817d1c19ee8107191b9b31  hsa.fa.fai\n21e0c25b4d817d1c19ee8107191b9b31  hsa.fa.fai2\n</code></pre> </li> <li> <p>Sorting by sequence length</p> <pre><code>$ memusg -t seqkit sort --by-length --reverse --two-pass hsa.fa &gt; hsa.sorted.fa\n[INFO] create and read FASTA index ...\n[INFO] read sequence IDs and lengths from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] sorting ...\n[INFO] output ...\n\nelapsed time: 4.892s\npeak rss: 500.15 MB\n</code></pre> <p>Detail:</p> <pre><code>$ seqkit fx2tab --length hsa.sorted.fa --name --only-id | cut -f 1,4 | more\n1       248956422\n2       242193529\n3       198295559\n4       190214555\n5       181538259\n6       170805979\n7       159345973\nX       156040895\n8       145138636\n9       138394717\n11      135086622\n10      133797422\n12      133275309\n13      114364328\n14      107043718\n15      101991189\n16      90338345\n17      83257441\n18      80373285\n20      64444167\n19      58617616\nY       57227415\n22      50818468\n21      46709983\nKI270728.1      1872759\nKI270727.1      448248\n...\n\nreal    0m10.697s\nuser    0m11.153s\nsys     0m0.917s\n</code></pre> </li> <li> <p>Shuffling sequences</p> <pre><code>$ memusg -t seqkit shuffle hsa.fa --two-pass &gt; hsa.shuffled.fa\n[INFO] create and read FASTA index ...\n[INFO] read sequence IDs from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] shuffle ...\n[INFO] output ...\n\nelapsed time: 6.632s\npeak rss: 528.3 MB\n</code></pre> </li> <li> <p>Spliting into files with single sequence</p> <pre><code>$ memusg -t seqkit split --by-id hsa.fa --two-pass\n[INFO] split by ID. idRegexp: ^([^\\s]+)\\s?\n[INFO] create and read FASTA index ...\n[INFO] read sequence IDs from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] write 1 sequences to file: hsa.id_KI270743.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270706.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270717.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270718.1.fa\n[INFO] write 1 sequences to file: hsa.id_KI270468.1.fa\n...\n\nelapsed time: 18.807s\npeak rss: 1.36 GB\n</code></pre> </li> <li> <p>Geting subsequence of some chromesomes</p> <pre><code>$ memusg -t seqkit subseq -r 1:10 --chr X --chr Y  hsa.fa\n&gt;X_1-10 X dna_sm:chromosome chromosome:GRCh38:X:1:156040895:1 REF\nnnnnnnnnnn\n&gt;Y_1-10 Y dna_sm:chromosome chromosome:GRCh38:Y:2781480:56887902:1 REF\nNNNNNNNNNN\n\nelapsed time: 1.276s\npeak rss: 640.92 MB\n</code></pre> </li> <li> <p>Geting CDS sequence of chr 1 by GTF files</p> <pre><code>$ memusg -t seqkit subseq --gtf Homo_sapiens.GRCh38.84.gtf.gz --chr X --feature cds  hsa.fa &gt; chrX.gtf.cds.fa\n[INFO] read GTF file ...\n[INFO] 22420 GTF features loaded\n\nelapsed time: 8.643s\npeak rss: 846.14 MB\n</code></pre> </li> </ol>"},{"location":"tutorial/#remove-contaminated-reads","title":"Remove contaminated reads","text":"<ol> <li> <p>Mapping with reads on some potential contaminate genomes, and get the reads IDs list.</p> <pre><code>$ wc -l contaminate.list\n244 contaminate.list\n\n$ head -n 2 contaminate.list\nHWI-D00523:240:HF3WGBCXX:1:1101:2574:2226\nHWI-D00523:240:HF3WGBCXX:1:1101:12616:2205\n</code></pre> </li> <li> <p>Remove contaminated reads</p> <pre><code>$ seqkit grep -f contaminate.list -v reads_1.fq.gz -o reads_1.clean.fq.gz\n$ seqkit grep -f contaminate.list -v reads_2.fq.gz -o reads_2.clean.fq.gz\n\n$ seqkit stat *.fq.gz\nfile                  seq_format   seq_type   num_seqs   min_len   avg_len   max_len\nreads_1.clean.fq.gz   FASTQ        DNA           2,256       226       227       229\nreads_1.fq.gz         FASTQ        DNA           2,500       226       227       229\nreads_2.clean.fq.gz   FASTQ        DNA           2,256       223       224       225\nreads_2.fq.gz         FASTQ        DNA           2,500       223       224       225\n</code></pre> </li> </ol>"},{"location":"tutorial/#handling-of-aligned-sequences","title":"Handling of aligned sequences","text":"<ol> <li> <p>Some mock sequences (usually they will be much longer)</p> <pre><code>$ cat seqs.fa\n&gt;seq1\nACAACGTCTACTTACGTTGCATCGTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq2\nACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq3\nACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG\n&gt;seq4\nACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> </li> <li> <p>Run multiple sequence alignment (clustalo)</p> <pre><code>clustalo -i seqs.fa -o seqs.msa.fa --force --outfmt fasta --threads=4\n</code></pre> </li> <li> <p>Convert FASTA format to tabular format.</p> <pre><code>$ seqkit fx2tab seqs.msa.fa\nseq1    ACAACGTCTACTTACGTTGCAT----CGTCATGCTGCATTACGTAGTCTGATGATG\nseq2    ---------------ACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\nseq3    ACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG-------------\nseq4    ACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> <p>or</p> <pre><code>$ seqkit fx2tab seqs.msa.fa | cut -f 2\nACAACGTCTACTTACGTTGCAT----CGTCATGCTGCATTACGTAGTCTGATGATG\n---------------ACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\nACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG-------------\nACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> <p>For me, it's useful when 1) manually assembling Sanger sequencing result, 2) designing site specific PCR primers.</p> </li> <li> <p>Remove gaps</p> <pre><code>$ seqkit seq seqs.msa.fa -g\n&gt;seq1\nACAACGTCTACTTACGTTGCATCGTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq2\nACACCGTCTACTTTCATGCTGCATTACGTAGTCTGATGATG\n&gt;seq3\nACAACGTCTACTTACGTTGCATCGTCATGCTGCACTGATGATG\n&gt;seq4\nACAACGTCTACTTACGTTGCATCTTCGGTCATGCTGCATTACGTAGTCTGATGATG\n</code></pre> </li> </ol>"},{"location":"tutorial/#play-with-mirna-hairpins","title":"Play with miRNA hairpins","text":""},{"location":"tutorial/#dataset","title":"Dataset","text":"<p><code>hairpin.fa.gz</code> from The miRBase Sequence Database -- Release 21</p>"},{"location":"tutorial/#quick-glance","title":"Quick glance","text":"<ol> <li> <p>Sequence number</p> <pre><code>$ seqkit stat hairpin.fa.gz\nfile           format  type  num_seqs    sum_len  min_len  avg_len  max_len\nhairpin.fa.gz  FASTA   RNA     28,645  2,949,871       39      103    2,354\n</code></pre> </li> <li> <p>First 10 bases</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit subseq -r 1:10 \\\n    | seqkit sort -s\n    | seqkit seq -s \\\n    | head -n 10\nAAAAAAAAAA\nAAAAAAAAAA\nAAAAAAAAAG\nAAAAAAAAAG\nAAAAAAAAAG\nAAAAAAAAAU\nAAAAAAAAGG\nAAAAAAACAU\nAAAAAAACGA\nAAAAAAAUUA\n</code></pre> <p>hmm, nothing special, non-coding RNA~</p> </li> </ol>"},{"location":"tutorial/#repeated-hairpin-sequences","title":"Repeated hairpin sequences","text":"<p>We may want to check how may identical hairpins among different species there are. <code>seqkit rmdup</code> could remove duplicated sequences by sequence content, and save the replicates to another file (here is <code>duplicated.fa.gz</code>), as well as replicating details (<code>duplicated.detail.txt</code>, 1th column is the repeated number, 2nd column contains sequence IDs seperated by comma).</p> <pre><code>$ seqkit rmdup -s -i hairpin.fa.gz -o clean.fa.gz -d duplicated.fa.gz -D duplicated.detail.txt\n\n$ head -n 5 duplicated.detail.txt\n18      dre-mir-430c-1, dre-mir-430c-2, dre-mir-430c-3, dre-mir-430c-4, dre-mir-430c-5, dre-mir-430c-6, dre-mir-430c-7, dre-mir-430c-8, dre-mir-430c-9, dre-mir-430c-10, dre-mir-430c-11, dre-mir-430c-12, dre-mir-430c-13, dre-mir-430c-14, dre-mir-430c-15, dre-mir-430c-16, dre-mir-430c-17, dre-mir-430c-18\n16      hsa-mir-29b-2, mmu-mir-29b-2, rno-mir-29b-2, ptr-mir-29b-2, ggo-mir-29b-2, ppy-mir-29b-2, sla-mir-29b, mne-mir-29b, ppa-mir-29b-2, bta-mir-29b-2, mml-mir-29b-2, eca-mir-29b-2, aja-mir-29b, oar-mir-29b-1, oar-mir-29b-2, rno-mir-29b-3\n15      dme-mir-125, dps-mir-125, dan-mir-125, der-mir-125, dgr-mir-125-1, dgr-mir-125-2, dmo-mir-125, dpe-mir-125-2, dpe-mir-125-1, dpe-mir-125-3, dse-mir-125, dsi-mir-125, dvi-mir-125, dwi-mir-125, dya-mir-125\n13      hsa-mir-19b-1, ggo-mir-19b-1, age-mir-19b-1, ppa-mir-19b-1, ppy-mir-19b-1, ptr-mir-19b-1, mml-mir-19b-1, sla-mir-19b-1, lla-mir-19b-1, mne-mir-19b-1, bta-mir-19b, oar-mir-19b, chi-mir-19b\n13      hsa-mir-20a, ssc-mir-20a, ggo-mir-20a, age-mir-20, ppa-mir-20, ppy-mir-20a, ptr-mir-20a, mml-mir-20a, sla-mir-20, lla-mir-20, mne-mir-20, bta-mir-20a, eca-mir-20a\n</code></pre> <p>The result shows the most conserved miRNAs among different species, <code>mir-29b</code>, <code>mir-125</code>, <code>mir-19b-1</code> and <code>mir-20a</code>. And the <code>dre-miR-430c</code> has the most multicopies in Danio rerio.</p>"},{"location":"tutorial/#hairpins-in-different-species","title":"Hairpins in different species","text":"<ol> <li> <p>Before spliting by species, let's take a look at the sequence names.</p> <pre><code>$ seqkit seq hairpin.fa.gz -n | head -n 3\ncel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\ncel-lin-4 MI0000002 Caenorhabditis elegans lin-4 stem-loop\ncel-mir-1 MI0000003 Caenorhabditis elegans miR-1 stem-loop\n</code></pre> <p>The first three letters (e.g. <code>cel</code>) are the abbreviation of species names. So we could split hairpins by the first letters by defining custom sequence ID parsing regular expression <code>^([\\w]+)\\-</code>.</p> <p>By default, <code>seqkit</code> takes the first non-space letters as sequence ID. For example,</p> FASTA head ID &gt;123456 gene name 123456 &gt;longname longname &gt;gi|110645304|ref|NC_002516.2| Pseudomona gi|110645304|ref|NC_002516.2| <p>But for some sequences from NCBI, e.g. <code>&gt;gi|110645304|ref|NC_002516.2| Pseudomona</code>, the ID is <code>NC_002516.2</code>. In this case, we could set sequence ID parsing regular expression by flag <code>--id-regexp \"\\|([^\\|]+)\\| \"</code> or just use flag <code>--id-ncbi</code>. If you want the <code>gi</code> number, then use <code>--id-regexp \"^gi\\|([^\\|]+)\\|\"</code>.</p> </li> <li> <p>Split sequences by species. A custom ID parsing regular expression is used, <code>^([\\w]+)\\-</code>.</p> <pre><code>$ seqkit split hairpin.fa.gz -i --id-regexp \"^([\\w]+)\\-\" --two-pass\n</code></pre> <p>To reduce memory usage when splitting big file, we should always use flag <code>--two-pass</code></p> </li> <li> <p>Species with most miRNA hairpins. Third column is the sequences number.</p> <pre><code>$ cd hairpin.fa.gz.split/;\n$ seqkit stat hairpin.id_* \\\n    | csvtk space2tab \\\n    | csvtk -t sort -k num_seqs:nr \\\n    | csvtk -t pretty \\\n    | more\nfile                     format   type   num_seqs   sum_len   min_len   avg_len   max_len\nhairpin.id_hsa.fasta     FASTA    RNA    1,881      154,242   82        82        82\nhairpin.id_mmu.fasta     FASTA    RNA    1,193      107,370   90        90        90\nhairpin.id_bta.fasta     FASTA    RNA    808        61,408    76        76        76\nhairpin.id_gga.fasta     FASTA    RNA    740        42,180    57        57        57\nhairpin.id_eca.fasta     FASTA    RNA    715        89,375    125       125       125\nhairpin.id_mtr.fasta     FASTA    RNA    672        231,840   345       345       345\n</code></pre> <p>Here, a CSV/TSV tool csvtk is used to sort and view the result.</p> </li> </ol> <p>For human miRNA hairpins</p> <ol> <li> <p>Length distribution.  <code>seqkit fx2tab</code> could show extra information like sequence length, GC content.  <code>csvtk</code> is used to plot.</p> <pre><code>$ seqkit grep -r -p '^hsa' hairpin.fa.gz  \\\n    | seqkit fx2tab -l \\\n    | cut -f 4  \\\n    | csvtk -H plot hist --xlab Length --title \"Human pre-miRNA length distribution\"\n</code></pre> <p></p> <pre><code>$ seqkit grep -r -p '^hsa' hairpin.fa.gz \\\n    | seqkit fx2tab -l \\\n    | cut -f 4 \\\n    | csvtk -H plot box --xlab Length --horiz --height 1.5\n</code></pre> <p></p> </li> </ol>"},{"location":"tutorial/#bacteria-genomes","title":"Bacteria genomes","text":""},{"location":"tutorial/#dataset_1","title":"Dataset","text":"<p>Pseudomonas aeruginosa PAO1, files:</p> <ul> <li>Genbank file <code>PAO1.gb</code></li> <li>Genome FASTA file <code>PAO1.fasta</code></li> <li>GTF file <code>PAO1.gtf</code> was created with <code>extract_features_from_genbank_file.py</code>, by<pre><code>extract_features_from_genbank_file.py  PAO1.gb -t . -f gtf &gt; PAO1.gtf\n</code></pre> </li> </ul>"},{"location":"tutorial/#motif-distribution","title":"Motif distribution","text":"<p>Motifs</p> <pre><code>$ cat motifs.fa\n&gt;GTAGCGS\nGTAGCGS\n&gt;GGWGKTCG\nGGWGKTCG\n</code></pre> <ol> <li> <p>Sliding. Remember flag <code>--id-ncbi</code>, do you?   By the way, do not be scared by the long flag <code>--circle-genome</code>, <code>--step</code>   and so on. They have short ones, <code>-c</code>, <code>-s</code></p> <pre><code>$ seqkit sliding --id-ncbi --circular-genome \\\n    --step 20000 --window 200000 PAO1.fasta -o PAO1.fasta.sliding.fa\n\n$ seqkit stat PAO1.fasta.sliding.fa\nfile                   format  type  num_seqs     sum_len  min_len  avg_len  max_len\nPAO1.fasta.sliding.fa  FASTA   DNA        314  62,800,000  200,000  200,000  200,000\n</code></pre> </li> <li> <p>Locating motifs</p> <pre><code>$ seqkit locate --id-ncbi --ignore-case --degenerate \\\n    --pattern-file motifs.fa  PAO1.fasta.sliding.fa -o  PAO1.fasta.sliding.fa.motifs.tsv\n</code></pre> </li> <li> <p>Ploting distribution (plot_motif_distribution.R)</p> <pre><code># preproccess\n$ perl -ne 'if (/_sliding:(\\d+)-(\\d+)\\t(.+)/) {$loc= $1 + 100000; print \"$loc\\t$3\\n\";} else {print}' PAO1.fasta.sliding.fa.motifs.tsv  &gt; PAO1.fasta.sliding.fa.motifs.tsv2\n\n# plot\n$ ./plot_motif_distribution.R\n</code></pre> <p>Result</p> <p></p> </li> </ol>"},{"location":"tutorial/#find-multicopy-genes","title":"Find multicopy genes","text":"<ol> <li> <p>Get all CDS sequences</p> <pre><code>$ seqkit subseq --id-ncbi --gtf PAO1.gtf --feature cds PAO1.fasta -o PAO1.cds.fasta\n\n$ seqkit stat *.fasta\nfile            format  type  num_seqs    sum_len    min_len    avg_len    max_len\nPAO1.cds.fasta  FASTA   DNA      5,572  5,593,306         72    1,003.8     16,884\nPAO1.fasta      FASTA   DNA          1  6,264,404  6,264,404  6,264,404  6,264,404\n</code></pre> </li> <li> <p>Get duplicated sequences</p> <pre><code>$ seqkit rmdup --by-seq --ignore-case PAO1.cds.fasta -o PAO1.cds.uniq.fasta \\\n    --dup-seqs-file PAO1.cds.dup.fasta --dup-num-file PAO1.cds.dup.text\n\n$ cat PAO1.cds.dup.text\n6       NC_002516.2_500104:501120:-, NC_002516.2_2556948:2557964:+, NC_002516.2_3043750:3044766:-, NC_002516.2_3842274:3843290:-, NC_002516.2_4473623:4474639:+, NC_002516.2_5382796:5383812:-\n2       NC_002516.2_2073555:2075438:+, NC_002516.2_4716660:4718543:+\n2       NC_002516.2_2072935:2073558:+, NC_002516.2_4716040:4716663:+\n2       NC_002516.2_2075452:2076288:+, NC_002516.2_4718557:4719393:+\n</code></pre> </li> </ol>"},{"location":"tutorial/#flanking-sequences","title":"Flanking sequences","text":"<ol> <li> <p>Get CDS and 1000 bp upstream sequence</p> <pre><code>$ seqkit subseq --id-ncbi --gtf PAO1.gtf \\\n    --feature cds PAO1.fasta --up-stream 1000\n</code></pre> </li> <li> <p>Get 1000 bp upstream sequence of CDS, NOT including CDS.</p> <pre><code>$ seqkit subseq --id-ncbi --gtf PAO1.gtf \\\n    --feature cds PAO1.fasta --up-stream 1000 --only-flank\n</code></pre> </li> </ol> Please enable JavaScript to view the comments powered by Disqus."},{"location":"usage/","title":"Usage and Examples","text":""},{"location":"usage/#quick-guide","title":"Quick Guide","text":"<ul> <li>Basic: seq, stats, subseq, sliding,   faidx, translate, watch, sana, scat</li> <li>Format conversion: fq2fa, fa2fq, fx2tab, tab2fx,   convert</li> <li>Searching: grep, locate, amplicon, fish</li> <li>Set operation: sample, rmdup, common,   duplicate, split, split2, head,   head-genome, range, pair</li> <li>Edit: concat, replace, restart, mutate,   rename</li> <li>Ordering: sort, shuffle</li> <li>BAM processing: bam</li> <li>Others: sum, merge-slides</li> </ul>"},{"location":"usage/#technical-details-and-guides-for-use","title":"Technical details and guides for use","text":""},{"location":"usage/#fastaq-format-parsing-and-writing","title":"FASTA/Q format parsing and writing","text":"<p>Seqkit also supports reading and writing xz (.xz) and zstd (.zst) formats since v2.2.0. Bzip2 format is supported since v2.4.0.</p> <p>SeqKit uses the author's lightweight and high-performance bioinformatics package bio for FASTA/Q parsing, which has high performance close to the famous C lib klib (kseq.h).</p> <p></p> <p>Notes:</p> <ul> <li><code>seqkit</code> uses 4 threads by default.</li> <li><code>seqkit_t1</code> uses 1 thread.</li> <li><code>seqtk</code> is single-threaded.</li> <li><code>seqtk+gzip</code>: <code>seqtk</code> pipes data to the single-threaded <code>gzip</code>.</li> <li><code>seqtk+pigz</code>: <code>seqtk</code> pipes data to the multithreaded <code>pigz</code> which uses 4 threads here.</li> </ul>"},{"location":"usage/#input-and-output-files","title":"Input and output files","text":"<p>Seqkit accepts input data from standard input (STDIN) and plain or gzip-compressed files. Files can be given via positional arguments or the flag <code>--infile-list</code>. For example:</p> <pre><code>seqkit seq   a.fasta b.fasta\n\nseqkit seq --infile-list file-list.txt\n\nseqkit seq --infile-list &lt;(find -name \"*.fq.gz\" directory)\n</code></pre> <p>Result are printed to standard ouput (STDOUT) by default, you can also specify the output file via the flag <code>-o/--out-file</code>. The file name extension <code>.gz</code> is automatically recognized. For example:</p> <pre><code>seqkit seq a.fasta -o a.fasta.gz\n\nseqkit grep -f IDs.txt read_1.fq.gz -o dir/read_1.fq.gz\n</code></pre> <p>Seqkit utlizies the pgzip package to read and write gzip file, and the outputted gzip file would be slighty larger than files generated by GNU <code>gzip</code>.</p> <p>Seqkit writes gzip files very fast, much faster than the multi-threaded <code>pigz</code>, so there's no need to pipe the result to <code>gzip</code>/<code>pigz</code>.</p>"},{"location":"usage/#sequence-formats-and-types","title":"Sequence formats and types","text":"<p>SeqKit seamlessly support FASTA and FASTQ format. Sequence format is automatically detected. All subcommands except for <code>faidx</code> and <code>bam</code> can handle both formats. And only when some commands (<code>subseq</code>, <code>split</code>, <code>sort</code> and <code>shuffle</code>) which utilise FASTA index to improve perfrmance for large files in two pass mode (by flag <code>--two-pass</code>), only FASTA format is supported.</p> <p>Sequence type (DNA/RNA/Protein) is automatically detected by leading subsequences of the first sequences in file or STDIN. The length of the leading subsequences is configurable by global flag <code>--alphabet-guess-seq-length</code> with default value of 10000. If length of the sequences is less than that, whole sequences will be checked.</p>"},{"location":"usage/#sequence-id","title":"Sequence ID","text":"<p>By default, most softwares, including <code>seqkit</code>, take the leading non-space letters as sequence identifier (ID). For example,</p> FASTA header ID &gt;123456 gene name 123456 &gt;longname longname &gt;gi|110645304|ref|NC_002516.2| Pseudomona gi|110645304|ref|NC_002516.2| <p>But for some sequences from NCBI, e.g. <code>&gt;gi|110645304|ref|NC_002516.2| Pseudomona</code>, the ID is <code>NC_002516.2</code>. In this case, we could set sequence ID parsing regular expression by global flag <code>--id-regexp \"\\|([^\\|]+)\\| \"</code> or just use flag <code>--id-ncbi</code>. If you want the <code>gi</code> number, then use <code>--id-regexp \"^gi\\|([^\\|]+)\\|\"</code>.</p>"},{"location":"usage/#fasta-index","title":"FASTA index","text":"<p>For some commands, including <code>subseq</code>, <code>split</code>, <code>sort</code> and <code>shuffle</code>, when input files are (plain or gzipped) FASTA files, FASTA index would be optional used for rapid access of sequences and reducing memory occupation.</p> <p>ATTENTION: the <code>.seqkit.fai</code> file created by SeqKit is a little different from <code>.fai</code> file created by <code>samtools</code>. SeqKit uses full sequence head instead of just ID as key.</p>"},{"location":"usage/#parallelization-of-cpu-intensive-jobs","title":"Parallelization of CPU intensive jobs","text":"<p>The validation of sequences bases and complement process of sequences are parallelized for large sequences.</p> <p>Parsing of line-based files, including BED/GFF file and ID list file are also parallelized.</p> <p>The <code>pgzip</code> package reads and write gzip files in parallel.</p> <p>The Parallelization is implemented by multiple goroutines in golang  which are similar to but much lighter weight than threads. The concurrency number is configurable with global flag <code>-j</code> or <code>--threads</code> (default value: 4).</p> <p>Using four threads is fast enough for most commands where FASTA/Q reading and writing is the performance bottleneck, and using more threads will not increase the speed. </p> <p>Few commands could benefit from multiple (&gt;4) threads:</p> <ul> <li><code>stats</code></li> <li><code>scat</code></li> <li><code>grep -s -m</code></li> <li><code>locate -s -m</code></li> <li><code>amplicon -s -m</code></li> </ul>"},{"location":"usage/#memory-occupation","title":"Memory occupation","text":"<p>Most of the subcommands do not read whole FASTA/Q records in to memory, including <code>stat</code>, <code>fq2fa</code>, <code>fx2tab</code>, <code>tab2fx</code>, <code>grep</code>, <code>locate</code>, <code>replace</code>,  <code>seq</code>, <code>sliding</code>, <code>subseq</code>.</p> <p>Note that when using <code>subseq --gtf | --bed</code>, if the GTF/BED files are too big, the memory usage will increase. You could use <code>--chr</code> to specify chromesomes and <code>--feature</code> to limit features.</p> <p>Some subcommands could either read all records or read the files twice by flag <code>-2</code> (<code>--two-pass</code>), including <code>sample</code>, <code>split</code>, <code>shuffle</code> and <code>sort</code>. They use FASTA index for rapid acccess of sequences and reducing memory occupation.</p>"},{"location":"usage/#reproducibility","title":"Reproducibility","text":"<p>Subcommands <code>sample</code> and <code>shuffle</code> use random function, random seed could be given by flag <code>-s</code> (<code>--rand-seed</code>). This makes sure that sampling result could be reproduced in different environments with same random seed.</p>"},{"location":"usage/#seqkit","title":"seqkit","text":"<pre><code>SeqKit -- a cross-platform and ultrafast toolkit for FASTA/Q file manipulation\n\nVersion: 2.9.0\n\nAuthor: Wei Shen &lt;shenwei356@gmail.com&gt;\n\nDocuments  : http://bioinf.shenwei.me/seqkit\nSource code: https://github.com/shenwei356/seqkit\nPlease cite: https://doi.org/10.1371/journal.pone.0163962\n\n\nSeqkit utlizies the pgzip (https://github.com/klauspost/pgzip) package to\nread and write gzip file, and the outputted gzip file would be slighty\nlarger than files generated by GNU gzip.\n\nSeqkit writes gzip files very fast, much faster than the multi-threaded pigz,\ntherefore there's no need to pipe the result to gzip/pigz.\n\nSeqkit also supports reading and writing xz (.xz) and zstd (.zst) formats since v2.2.0.\nBzip2 format is supported since v2.4.0.\n\nCompression level:\n  format   range   default  comment\n  gzip     1-9     5        https://github.com/klauspost/pgzip sets 5 as the default value.\n  xz       NA      NA       https://github.com/ulikunitz/xz does not support.\n  zstd     1-4     2        roughly equals to zstd 1, 3, 7, 11, respectively.\n  bzip     1-9     6        https://github.com/dsnet/compress\n\nUsage:\n  seqkit [command]\n\nCommands for Basic Operation:\n  faidx           create the FASTA index file and extract subsequences\n  scat            real time recursive concatenation and streaming of fastx files\n  seq             transform sequences (extract ID, filter by length, remove gaps, reverse complement...)\n  sliding         extract subsequences in sliding windows\n  stats           simple statistics of FASTA/Q files\n  subseq          get subsequences by region/gtf/bed, including flanking sequences\n  translate       translate DNA/RNA to protein sequence (supporting ambiguous bases)\n  watch           monitoring and online histograms of sequence features\n\nCommands for Format Conversion:\n  convert         convert FASTQ quality encoding between Sanger, Solexa and Illumina\n  fa2fq           retrieve corresponding FASTQ records by a FASTA file\n  fq2fa           convert FASTQ to FASTA\n  fx2tab          convert FASTA/Q to tabular format (and length, GC content, average quality...)\n  tab2fx          convert tabular format to FASTA/Q format\n\nCommands for Searching:\n  amplicon        extract amplicon (or specific region around it) via primer(s)\n  fish            look for short sequences in larger sequences using local alignment\n  grep            search sequences by ID/name/sequence/sequence motifs, mismatch allowed\n  locate          locate subsequences/motifs, mismatch allowed\n\nCommands for Set Operation:\n  common          find common/shared sequences of multiple files by id/name/sequence\n  duplicate       duplicate sequences N times\n  head            print first N FASTA/Q records\n  head-genome     print sequences of the first genome with common prefixes in name\n  pair            match up paired-end reads from two fastq files\n  range           print FASTA/Q records in a range (start:end)\n  rmdup           remove duplicated sequences by ID/name/sequence\n  sample          sample sequences by number or proportion\n  split           split sequences into files by id/seq region/size/parts (mainly for FASTA)\n  split2          split sequences into files by size/parts (FASTA, PE/SE FASTQ)\n\nCommands for Edit:\n  concat          concatenate sequences with the same ID from multiple files\n  mutate          edit sequence (point mutation, insertion, deletion)\n  rename          rename duplicated IDs\n  replace         replace name/sequence by regular expression\n  restart         reset start position for circular genome\n  sana            sanitize broken single line FASTQ files\n\nCommands for Ordering:\n  shuffle         shuffle sequences\n  sort            sort sequences by id/name/sequence/length\n\nCommands for BAM Processing:\n  bam             monitoring and online histograms of BAM record features\n\nCommands for Miscellaneous:\n  merge-slides    merge sliding windows generated from seqkit sliding\n  sum             compute message digest for all sequences in FASTA/Q files\n\nAdditional Commands:\n  genautocomplete generate shell autocompletion script (bash|zsh|fish|powershell)\n  version         print version information and check for update\n\nFlags:\n      --alphabet-guess-seq-length int   length of sequence prefix of the first FASTA record based on\n                                        which seqkit guesses the sequence type (0 for whole seq)\n                                        (default 10000)\n      --compress-level int              compression level for gzip, zstd, xz and bzip2. type \"seqkit -h\"\n                                        for the range and default value for each format (default -1)\n  -h, --help                            help for seqkit\n      --id-ncbi                         FASTA head is NCBI-style, e.g. &gt;gi|110645304|ref|NC_002516.2|\n                                        Pseud...\n      --id-regexp string                regular expression for parsing ID (default \"^(\\\\S+)\\\\s?\")\n  -X, --infile-list string              file of input files list (one file per line), if given, they are\n                                        appended to files from cli arguments\n  -w, --line-width int                  line width when outputting FASTA format (0 for no wrap) (default 60)\n  -o, --out-file string                 out file (\"-\" for stdout, suffix .gz for gzipped out) (default \"-\")\n      --quiet                           be quiet and do not show extra information\n  -t, --seq-type string                 sequence type (dna|rna|protein|unlimit|auto) (for auto, it\n                                        automatically detect by the first sequence) (default \"auto\")\n  -j, --threads int                     number of CPUs. can also set with environment variable\n                                        SEQKIT_THREADS) (default 4)\n\nUse \"seqkit [command] --help\" for more information about a command.\n</code></pre>"},{"location":"usage/#datasets","title":"Datasets","text":"<p>Datasets from The miRBase Sequence Database -- Release 21</p> <ul> <li><code>hairpin.fa.gz</code></li> <li><code>mature.fa.gz</code></li> <li><code>miRNA.diff.gz</code></li> </ul> <p>Human genome from ensembl (For <code>seqkit subseq</code>)</p> <ul> <li><code>Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz</code></li> <li><code>Homo_sapiens.GRCh38.84.gtf.gz</code></li> <li><code>Homo_sapiens.GRCh38.84.bed.gz</code> is converted from <code>Homo_sapiens.GRCh38.84.gtf.gz</code> by <code>gtf2bed</code> with the command<pre><code>zcat Homo_sapiens.GRCh38.84.gtf.gz \\\n    | gtf2bed --do-not-sort \\\n    | gzip -c &gt; Homo_sapiens.GRCh38.84.bed.gz\n</code></pre> </li> </ul> <p>Only DNA and gtf/bed data of Chr1 were used:</p> <ul> <li> <p><code>chr1.fa.gz</code></p> <pre><code>seqkit grep -p 1 Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz -o chr1.fa.gz\n</code></pre> </li> <li> <p><code>chr1.gtf.gz</code></p> <pre><code>zcat Homo_sapiens.GRCh38.84.gtf.gz | grep -w '^1' | gzip -c &gt; chr1.gtf.gz\n</code></pre> </li> <li> <p><code>chr1.bed.gz</code></p> <pre><code>zcat Homo_sapiens.GRCh38.84.bed.gz | grep -w '^1' | gzip -c &gt; chr1.bed.gz\n</code></pre> </li> </ul>"},{"location":"usage/#seq","title":"seq","text":"<p>Usage</p> <pre><code>transform sequences (extract ID, filter by length, remove gaps, reverse complement...)\n\nUsage:\n  seqkit seq [flags] \n\nFlags:\n  -k, --color                 colorize sequences - to be piped into \"less -R\"\n  -p, --complement            complement sequence, flag '-v' is recommended to switch on\n      --dna2rna               DNA to RNA\n  -G, --gap-letters string    gap letters to be removed with -g/--remove-gaps (default \"- \\t.\")\n  -h, --help                  help for seq\n  -l, --lower-case            print sequences in lower case\n  -M, --max-len int           only print sequences shorter than or equal to the maximum length (-1 for\n                              no limit) (default -1)\n  -R, --max-qual float        only print sequences with average quality less than this limit (-1 for no\n                              limit) (default -1)\n  -m, --min-len int           only print sequences longer than or equal to the minimum length (-1 for no\n                              limit) (default -1)\n  -Q, --min-qual float        only print sequences with average quality greater or equal than this limit\n                              (-1 for no limit) (default -1)\n  -n, --name                  only print names/sequence headers\n  -i, --only-id               print IDs instead of full headers\n  -q, --qual                  only print qualities\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n  -g, --remove-gaps           remove gaps letters set by -G/--gap-letters, e.g., spaces, tabs, and\n                              dashes (gaps \"-\" in aligned sequences)\n  -r, --reverse               reverse sequence\n      --rna2dna               RNA to DNA\n  -s, --seq                   only print sequences\n  -u, --upper-case            print sequences in upper case\n  -v, --validate-seq          validate bases according to the alphabet\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Read and print</p> <ul> <li> <p>From file:</p> <pre><code>$ seqkit seq hairpin.fa.gz\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n\n$ seqkit seq reads_1.fq.gz\n@HWI-D00523:240:HF3WGBCXX:1:1101:2574:2226 1:N:0:CTGTAG\nTGAGGAATATTGGTCAATGGGCGCGAGCCTGAACCAGCCAAGTAGCGTGAAGGATGACTGCCCTACGGG\n+\nHIHIIIIIHIIHGHHIHHIIIIIIIIIIIIIIIHHIIIIIHHIHIIIIIGIHIIIIHHHHHHGHIHIII\n</code></pre> </li> <li> <p>From stdin:</p> <pre><code>zcat hairpin.fa.gz | seqkit seq\n</code></pre> </li> </ul> </li> <li> <p>Sequence types</p> <ul> <li> <p>By default, <code>seqkit seq</code> automatically detect the sequence type</p> <pre><code>$ echo -e \"&gt;seq\\nacgtryswkmbdhvACGTRYSWKMBDHV\" | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   DNA          1       28       28       28       28\n\n$ echo -e \"&gt;seq\\nACGUN ACGUN\" | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA          1       11       11       11       11\n\n$ echo -e \"&gt;seq\\nabcdefghijklmnpqrstvwyz\" | seqkit stats\nfile  format  type     num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   Protein         1       23       23       23       23\n\n$ echo -e \"@read\\nACTGCN\\n+\\n@IICCG\" | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTQ   DNA          1        6        6        6        6\n</code></pre> </li> <li> <p>You can also set sequence type by flag <code>-t</code> (<code>--seq-type</code>).   But this only take effect on subcommands <code>seq</code> and <code>locate</code>.</p> <pre><code>$ echo -e \"&gt;seq\\nabcdefghijklmnpqrstvwyz\" | seqkit seq -t dna\n[INFO] when flag -t (--seq-type) given, flag -v (--validate-seq) is automatically switched on\n[ERRO] error when parsing seq: seq (invalid DNAredundant letter: e)\n</code></pre> </li> </ul> </li> <li> <p>Only print names</p> <ul> <li> <p>Full name:</p> <pre><code>$ seqkit seq hairpin.fa.gz -n\ncel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\ncel-lin-4 MI0000002 Caenorhabditis elegans lin-4 stem-loop\ncel-mir-1 MI0000003 Caenorhabditis elegans miR-1 stem-loop\n</code></pre> </li> <li> <p>Only ID:</p> <pre><code>$ seqkit seq hairpin.fa.gz -n -i\ncel-let-7\ncel-lin-4\ncel-mir-1\n</code></pre> </li> <li> <p>Custom ID region by regular expression (this could be applied to all subcommands):</p> <pre><code>$ seqkit seq hairpin.fa.gz -n -i --id-regexp \"^[^\\s]+\\s([^\\s]+)\\s\"\nMI0000001\nMI0000002\nMI0000003\n</code></pre> </li> </ul> </li> <li> <p>Only print seq (global flag <code>-w</code> defines the output line width, <code>0</code> for no wrap)</p> <pre><code>$ seqkit seq hairpin.fa.gz -s -w 0\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAACUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\nAUGCUUCCGGCCUGUUCCCUGAGACCUCAAGUGUGAGUGUACUAUUGAUGCUUCACACCUGGGCUCUCCGGGUACCAGGACGGUUUGAGCAGAU\nAAAGUGACCGUACCGAGCUGCAUACUUCCUUACAUGCCCAUACUAUAUCAUAAAUGGAUAUGGAAUGUAAAGAAGUAUGUAGAACGGGGUGGUAGU\n</code></pre> </li> <li> <p>Convert multi-line FASTQ to 4-line FASTQ</p> <pre><code>$ seqkit seq reads_1.fq.gz -w 0\n</code></pre> </li> <li> <p>Reverse comlement sequence</p> <pre><code>$ seqkit seq hairpin.fa.gz -r -p\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUCGAAGAGUUCUGUCUCCGGUAAGGUAGAAAAUUGCAUAGUUCACCGGUGGUAAUAUUCC\nAAACUAUACAACCUACUACCUCACCGGAUCCACAGUGUA\n</code></pre> </li> <li> <p>Remove gaps and to lower/upper case</p> <pre><code>$ echo -e \"&gt;seq\\nACGT-ACTGC-ACC\" | seqkit seq -g -u\n&gt;seq\nACGTACTGCACC\n</code></pre> </li> <li> <p>RNA to DNA</p> <pre><code>$ echo -e \"&gt;seq\\nUCAUAUGCUUGUCUCAAAGAUUA\" | seqkit seq --rna2dna\n&gt;seq\nTCATATGCTTGTCTCAAAGATTA\n</code></pre> </li> <li> <p>Filter by sequence length</p> <pre><code>$ cat hairpin.fa | seqkit seq | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     28,645  2,949,871       39      103    2,354\n\n$ cat hairpin.fa | seqkit seq -m 100 | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     10,975  1,565,486      100    142.6    2,354\n\n$ cat hairpin.fa | seqkit seq -m 100 -M 1000 | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     10,972  1,560,270      100    142.2      938\n</code></pre> </li> </ol>"},{"location":"usage/#subseq","title":"subseq","text":"<p>Usage</p> <pre><code>get subsequences by region/gtf/bed, including flanking sequences.\n\nAttention:\n  1. When extracting with BED/GTF from plain text FASTA files, the order of output sequences\n     are random. To keep the order, just compress the FASTA file (input.fasta) and use the\n     compressed one (input.fasta.gz) as the input.\n  2. Use \"seqkit grep\" for extracting subsets of sequences.\n     \"seqtk subseq seqs.fasta id.txt\" equals to\n     \"seqkit grep -f id.txt seqs.fasta\"\n\nRecommendation:\n  1. Use plain FASTA file, so seqkit could utilize FASTA index.\n  2. The flag -U/--update-faidx is recommended to ensure the .fai file matches the FASTA file.\n\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit subseq [flags] \n\nFlags:\n      --bed string        by tab-delimited BED file\n      --chr strings       select limited sequence with sequence IDs when using --gtf or --bed (multiple\n                          value supported, case ignored)\n  -d, --down-stream int   down stream length\n      --feature strings   select limited feature types (multiple value supported, case ignored, only\n                          works with GTF)\n      --gtf string        by GTF (version 2.2) file\n      --gtf-tag string    output this tag as sequence comment (default \"gene_id\")\n  -h, --help              help for subseq\n  -f, --only-flank        only return up/down stream sequence\n  -r, --region string     by region. e.g 1:12 for first 12 bases, -12:-1 for last 12 bases, 13:-1 for\n                          cutting first 12 bases. type \"seqkit subseq -h\" for more examples\n  -R, --region-coord      append coordinates to sequence ID for -r/--region\n  -u, --up-stream int     up stream length\n  -U, --update-faidx      update the fasta index file if it exists. Use this if you are not sure whether\n                          the fasta file changed\n\n</code></pre> <p>Examples</p> <p>Recommendation: use plain FASTA file, so seqkit could utilize FASTA index.</p> <ol> <li> <p>First 12 bases</p> <pre><code>$ zcat hairpin.fa.gz | seqkit subseq -r 1:12\n</code></pre> </li> <li> <p>Last 12 bases</p> <pre><code>$ zcat hairpin.fa.gz | seqkit subseq -r -12:-1\n</code></pre> </li> <li> <p>Subsequences without first and last 12 bases</p> <pre><code>$ zcat hairpin.fa.gz | seqkit subseq -r 13:-13\n</code></pre> </li> <li> <p>Get subsequence by GTF file</p> <pre><code>$ cat t.fa\n&gt;seq\nactgACTGactgn\n\n$ cat t.gtf\nseq     test    CDS     5       8       .       .       .       gene_id \"A\"; transcript_id \"\";\nseq     test    CDS     5       8       .       -       .       gene_id \"B\"; transcript_id \"\";\n\n$ seqkit subseq --gtf t.gtf t.fa\n&gt;seq_5:8:. A\nACTG\n&gt;seq_5:8:- B\nCAGT\n</code></pre> <p>Human genome example:</p> <p>AVOID loading all data from Homo_sapiens.GRCh38.84.gtf.gz, the uncompressed data are so big and may exhaust your RAM.</p> <p>We could specify chromesomes and features.</p> <pre><code>$ seqkit subseq --gtf Homo_sapiens.GRCh38.84.gtf.gz --chr 1 --feature cds  hsa.fa &gt; chr1.gtf.cds.fa\n\n$ seqkit stats chr1.gtf.cds.fa\nfile             format  type  num_seqs    sum_len  min_len  avg_len  max_len\nchr1.gtf.cds.fa  FASTA   DNA     65,012  9,842,274        1    151.4   12,045\n</code></pre> </li> <li> <p>Get CDS and 3bp up-stream sequences</p> <pre><code>$ seqkit subseq --gtf t.gtf t.fa -u 3\n&gt;seq_5:8:._us:3 A\nctgACTG\n&gt;seq_5:8:-_us:3 B\nagtCAGT\n</code></pre> </li> <li> <p>Get 3bp up-stream sequences of CDS, not including CDS</p> <pre><code>$ seqkit subseq --gtf t.gtf t.fa -u 3 -f\n&gt;seq_5:8:._usf:3 A\nctg\n&gt;seq_5:8:-_usf:3 B\nagt\n</code></pre> </li> <li> <p>Get subsequences by BED file.</p> <p>AVOID loading all data from Homo_sapiens.GRCh38.84.gtf.gz, the uncompressed data are so big and may exhaust your RAM.</p> <pre><code>$ seqkit subseq --bed Homo_sapiens.GRCh38.84.bed.gz --chr 1 hsa.fa \\\n    &gt;  chr1.bed.gz.fa\n</code></pre> <p>We may need to remove duplicated sequences</p> <pre><code>$ seqkit subseq --bed Homo_sapiens.GRCh38.84.bed.gz --chr 1 hsa.fa \\\n    | seqkit rmdup &gt; chr1.bed.rmdup.fa\n[INFO] 141060 duplicated records removed\n</code></pre> <p>Summary:</p> <pre><code>$ seqkit stats chr1.gz.*.gz\nfile               seq_format   seq_type   num_seqs   min_len   avg_len     max_len\nchr1.gz.fa         FASTA        DNA         231,974         1   3,089.5   1,551,957\nchr1.gz.rmdup.fa   FASTA        DNA          90,914         1   6,455.8   1,551,957\n</code></pre> </li> </ol>"},{"location":"usage/#sliding","title":"sliding","text":"<p>Usage</p> <pre><code>extract subsequences in sliding windows\n\nUsage:\n  seqkit sliding [flags]\n\nFlags:\n  -c, --circular          circular genome (same to -C/--circular-genome)\n  -C, --circular-genome   circular genome (same to -c/--circular)\n  -g, --greedy            greedy mode, i.e., exporting last subsequences even shorter than the windows size\n  -h, --help              help for sliding\n  -s, --step int          step size\n  -S, --sufix string      suffix added to the sequence ID\n  -W, --window int        window size\n\n</code></pre> <p>Examples</p> <ol> <li> <p>General use</p> <pre><code>$ echo -e \"&gt;seq\\nACGTacgtNN\" | seqkit sliding -s 3 -W 6\n&gt;seq_sliding:1-6\nACGTac\n&gt;seq_sliding:4-9\nTacgtN\n</code></pre> </li> <li> <p>Greedy mode</p> <pre><code>$ echo -e \"&gt;seq\\nACGTacgtNN\" | seqkit sliding -s 3 -W 6 -g\n&gt;seq_sliding:1-6\nACGTac\n&gt;seq_sliding:4-9\nTacgtN\n&gt;seq_sliding:7-12\ngtNN\n&gt;seq_sliding:10-15\nN\n</code></pre> </li> <li> <p>Circular genome</p> <pre><code>$ echo -e \"&gt;seq\\nACGTacgtNN\" | seqkit sliding -s 3 -W 6 -C\n&gt;seq_sliding:1-6\nACGTac\n&gt;seq_sliding:4-9\nTacgtN\n&gt;seq_sliding:7-2\ngtNNAC\n&gt;seq_sliding:10-5\nNACGTa\n</code></pre> </li> <li> <p>Generate GC content for ploting</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sliding -s 5 -W 30 \\\n    | seqkit fx2tab -n -g\ncel-let-7_sliding:1-30          50.00\ncel-let-7_sliding:6-35          46.67\ncel-let-7_sliding:11-40         43.33\ncel-let-7_sliding:16-45         36.67\ncel-let-7_sliding:21-50         33.33\ncel-let-7_sliding:26-55         40.00\n...\n</code></pre> </li> </ol>"},{"location":"usage/#stats","title":"stats","text":"<p>Usage</p> <pre><code>simple statistics of FASTA/Q files\n\nColumns:\n\n  1.  file      input file, \"-\" for STDIN\n  2.  format    FASTA or FASTQ\n  3.  type      DNA, RNA, Protein or Unlimit\n  4.  num_seqs  number of sequences\n  5.  sum_len   number of bases or residues       , with gaps or spaces counted\n  6.  min_len   minimal sequence length           , with gaps or spaces counted\n  7.  avg_len   average sequence length           , with gaps or spaces counted\n  8.  max_len   miximal sequence length           , with gaps or spaces counted\n  9.  Q1        first quartile of sequence length , with gaps or spaces counted\n  10. Q2        median of sequence length         , with gaps or spaces counted\n  11. Q3        third quartile of sequence length , with gaps or spaces counted\n  12. sum_gap   number of gaps\n  13. N50       N50. https://en.wikipedia.org/wiki/N50,_L50,_and_related_statistics#N50\n  14. N50_num   N50_num or L50. https://en.wikipedia.org/wiki/N50,_L50,_and_related_statistics#L50\n  15. Q20(%)    percentage of bases with the quality score greater than 20\n  16. Q30(%)    percentage of bases with the quality score greater than 30\n  17. AvgQual   average quality\n  18. GC(%)     percentage of GC content\n  19. sum_n     number of ambitious letters (N, n, X, x)\n\nAttention:\n  1. Sequence length metrics (sum_len, min_len, avg_len, max_len, Q1, Q2, Q3)\n     count the number of gaps or spaces. You can remove them with \"seqkit seq -g\":\n         seqkit seq -g input.fasta | seqkit stats\n\nTips:\n  1. For lots of small files (especially on SDD), use big value of '-j' to\n     parallelize counting.\n  2. Extract one metric with csvtk (https://github.com/shenwei356/csvtk):\n         seqkit stats -Ta input.fastq.gz | csvtk cut -t -f \"Q30(%)\" | csvtk del-header\n\nUsage:\n  seqkit stats [flags]\n\nAliases:\n  stats, stat\n\nFlags:\n  -N, --N strings            other N50-like stats. value range [0, 100], multiple values supported,\n                             e.g., -N 50,90 or -N 50 -N 90\n  -a, --all                  all statistics, including quartiles of seq length, sum_gap, N50\n  -b, --basename             only output basename of files\n  -E, --fq-encoding string   fastq quality encoding. available values: 'sanger', 'solexa',\n                             'illumina-1.3+', 'illumina-1.5+', 'illumina-1.8+'. (default \"sanger\")\n  -G, --gap-letters string   gap letters (default \"- .\")\n  -h, --help                 help for stats\n  -e, --skip-err             skip error, only show warning message\n  -S, --skip-file-check      skip input file checking when given files or a file list.\n  -i, --stdin-label string   label for replacing default \"-\" for stdin (default \"-\")\n  -T, --tabular              output in machine-friendly tabular format\n\n</code></pre> <p>Eexamples</p> <ol> <li> <p>General use</p> <pre><code>$ seqkit stats *.f{a,q}.gz\nfile           format  type  num_seqs    sum_len  min_len  avg_len  max_len\nhairpin.fa.gz  FASTA   RNA     28,645  2,949,871       39      103    2,354\nmature.fa.gz   FASTA   RNA     35,828    781,222       15     21.8       34\nreads_1.fq.gz  FASTQ   DNA      2,500    567,516      226      227      229\nreads_2.fq.gz  FASTQ   DNA      2,500    560,002      223      224      225\n</code></pre> </li> <li> <p>Machine-friendly tabular format</p> <pre><code>$ seqkit stats *.f{a,q}.gz -T\nfile    format  type    num_seqs        sum_len min_len avg_len max_len\nhairpin.fa.gz   FASTA   RNA     28645   2949871 39      103.0   2354\nmature.fa.gz    FASTA   RNA     35828   781222  15      21.8    34\nIllimina1.8.fq.gz       FASTQ   DNA     10000   1500000 150     150.0   150\nreads_1.fq.gz   FASTQ   DNA     2500    567516  226     227.0   229\nreads_2.fq.gz   FASTQ   DNA     2500    560002  223     224.0   225\n\n# So you can process the result with tools like csvtk: http://bioinf.shenwei.me/csvtk\n\n$ seqkit stats *.f{a,q}.gz -T | csvtk pretty -t\nfile                format   type   num_seqs   sum_len   min_len   avg_len   max_len\n-----------------   ------   ----   --------   -------   -------   -------   -------\nhairpin.fa.gz       FASTA    RNA    28645      2949871   39        103.0     2354\nmature.fa.gz        FASTA    RNA    35828      781222    15        21.8      34\nIllimina1.8.fq.gz   FASTQ    DNA    10000      1500000   150       150.0     150\nreads_1.fq.gz       FASTQ    DNA    2500       567516    226       227.0     229\nreads_2.fq.gz       FASTQ    DNA    2500       560002    223       224.0     225\n\n# To markdown\n\n$ seqkit stats *.f{a,q}.gz -T | csvtk csv2md -t\n|file             |format|type|num_seqs|sum_len|min_len|avg_len|max_len|\n|:----------------|:-----|:---|:-------|:------|:------|:------|:------|\n|hairpin.fa.gz    |FASTA |RNA |28645   |2949871|39     |103.0  |2354   |\n|mature.fa.gz     |FASTA |RNA |35828   |781222 |15     |21.8   |34     |\n|Illimina1.8.fq.gz|FASTQ |DNA |10000   |1500000|150    |150.0  |150    |\n|reads_1.fq.gz    |FASTQ |DNA |2500    |567516 |226    |227.0  |229    |\n|reads_2.fq.gz    |FASTQ |DNA |2500    |560002 |223    |224.0  |225    |\n</code></pre> file format type num_seqs sum_len min_len avg_len max_len hairpin.fa.gz FASTA RNA 28645 2949871 39 103.0 2354 mature.fa.gz FASTA RNA 35828 781222 15 21.8 34 Illimina1.8.fq.gz FASTQ DNA 10000 1500000 150 150.0 150 reads_1.fq.gz FASTQ DNA 2500 567516 226 227.0 229 reads_2.fq.gz FASTQ DNA 2500 560002 223 224.0 225 </li> <li> <p>Extra information</p> <pre><code>$ seqkit stats *.f{a,q}.gz -a\nfile               format  type  num_seqs    sum_len  min_len  avg_len  max_len   Q1   Q2   Q3  sum_gap  N50  N50_num  Q20(%)  Q30(%)  AvgQual  GC(%)  sum_n\nhairpin.fa.gz      FASTA   RNA     28,645  2,949,871       39      103    2,354   76   91  111        0  101      380       0       0        0  45.77    255\nmature.fa.gz       FASTA   RNA     35,828    781,222       15     21.8       34   21   22   22        0   22       12       0       0        0   47.6      0\nIllimina1.8.fq.gz  FASTQ   DNA     10,000  1,500,000      150      150      150  150  150  150        0  150        1   96.16   89.71    24.82  49.91     38\nnanopore.fq.gz     FASTQ   DNA      4,000  1,798,723      153    449.7    6,006  271  318  391        0  395      585   40.79   12.63     9.48  46.66      0\nreads_1.fq.gz      FASTQ   DNA      2,500    567,516      226      227      229  227  227  227        0  227        3   91.24   86.62    15.45  53.63     44\nreads_2.fq.gz      FASTQ   DNA      2,500    560,002      223      224      225  224  224  224        0  224        2   91.06   87.66    14.62  54.77      2\n</code></pre> </li> <li> <p>Parallelize counting files, it's much faster for lots of small files, especially for files on SSD</p> <pre><code>seqkit stats -j 10 refseq/virual/*.fna.gz\n</code></pre> </li> <li> <p>Skip error</p> <pre><code>$ seqkit stats tests/*\n[ERRO] tests/hairpin.fa.fai: fastx: invalid FASTA/Q format\n\n$ seqkit stats tests/* -e\n[WARN] tests/hairpin.fa.fai: fastx: invalid FASTA/Q format\n[WARN] tests/hairpin.fa.seqkit.fai: fastx: invalid FASTA/Q format\n[WARN] tests/miRNA.diff.gz: fastx: invalid FASTA/Q format\n[WARN] tests/test.sh: fastx: invalid FASTA/Q format\nfile                     format  type  num_seqs    sum_len  min_len  avg_len  max_len\ntests/contigs.fa         FASTA   DNA          9         54        2        6       10\ntests/hairpin.fa         FASTA   RNA     28,645  2,949,871       39      103    2,354\ntests/Illimina1.5.fq     FASTQ   DNA          1        100      100      100      100\ntests/Illimina1.8.fq.gz  FASTQ   DNA     10,000  1,500,000      150      150      150\ntests/hairpin.fa.gz      FASTA   RNA     28,645  2,949,871       39      103    2,354\ntests/reads_1.fq.gz      FASTQ   DNA      2,500    567,516      226      227      229\ntests/mature.fa.gz       FASTA   RNA     35,828    781,222       15     21.8       34\ntests/reads_2.fq.gz      FASTQ   DNA      2,500    560,002      223      224      225\n</code></pre> </li> <li> <p>Output basename instead of full path (<code>-b/--basename</code>)</p> </li> </ol>"},{"location":"usage/#sum","title":"sum","text":"<p>Usage</p> <pre><code>compute message digest for all sequences in FASTA/Q files\n\nAttention:\n  1. Sequence headers and qualities are skipped, only sequences matter.\n  2. The order of sequences records does not matter.\n  3. Circular complete genomes are supported with the flag -c/--circular.\n     - The same double-stranded genomes with different start positions or\n       in reverse complement strand will not affect the result.\n     - For single-stranded genomes like ssRNA genomes, use -s/--single-strand.\n     - The message digest would change with different values of k-mer size.\n  4. Multiple files are processed in parallel (-j/--threads).\n\nMethod:\n  1. Converting the sequences to low cases, optionally removing gaps (-g).\n  2. Computing the hash (xxhash) for all sequences or k-mers of a circular\n     complete genome (-c/--circular).\n  3. Sorting all hash values, for ignoring the order of sequences.\n  4. Computing MD5 digest from the hash values, sequences length, and\n     the number of sequences.\n\nFollowing the seqhash in Poly (https://github.com/TimothyStiles/poly/),\nWe add meta information to the message digest, with the format of:\n\n    seqkit.&lt;version&gt;_&lt;seq type&gt;&lt;seq structure&gt;&lt;strand&gt;_&lt;kmer size&gt;_&lt;seq digest&gt;\n\n    &lt;version&gt;:       digest version\n    &lt;seq type&gt;:      'D' for DNA, 'R' for RNA, 'P' for protein, 'N' for others\n    &lt;seq structure&gt;: 'L' for linear sequence, 'C' for circular genome\n    &lt;strand&gt;:        'D' for double-stranded, 'S' for single-stranded\n    &lt;kmer size&gt;:     0 for linear sequence, other values for circular genome\n\nExamples:\n\n    seqkit.v0.1_DLS_k0_176250c8d1cde6c385397df525aa1a94    DNA.fq.gz\n    seqkit.v0.1_PLS_k0_c244954e4960dd2a1409cd8ee53d92b9    Protein.fasta\n    seqkit.v0.1_RLS_k0_0f1fb263f0c05a259ae179a61a80578d    single-stranded RNA.fasta\n\n    seqkit.v0.1_DCD_k31_e59dad6d561f1f1f28ebf185c6f4c183   double-stranded-circular DNA.fasta\n    seqkit.v0.1_DCS_k31_dd050490cd62ea5f94d73d4d636b7d60   single-stranded-circular DNA.fasta\n\nUsage:\n  seqkit sum [flags]\n\nFlags:\n  -a, --all                  show all information, including the sequences length and the number of sequences\n  -b, --basename             only output basename of files\n  -c, --circular             the file contains a single cicular genome sequence\n  -G, --gap-letters string   gap letters to delete with the flag -g/--remove-gaps (default \"- \\t.*\")\n  -h, --help                 help for sum\n  -k, --kmer-size int        k-mer size for processing circular genomes (default 1000)\n  -g, --remove-gaps          remove gap characters set in the option -G/gap-letters\n      --rna2dna              convert RNA to DNA\n  -s, --single-strand        only consider the positive strand of a circular genome, e.g., ssRNA virus\n                             genomes\n\n</code></pre> <p>Examples:</p> <p>A, B, C, D are the same vircular genomes with different starting positions or strands:</p> <pre><code>$ cat virus-{A,B,C,D}.fasta\n&gt;seq\nTGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAA\n&gt;seq.revcom\nTTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCA\n&gt;seq.new-start\nGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAAT\n&gt;seq.revcom.new-start\nTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCAT\n\n# cat to one file\n$ cat virus-{A,B,C,D}.fasta &gt; virues.fasta\n\n# shuffle and rename\n$ cat virus-{A,B,C,D}.fasta \\\n    | seqkit shuffle \\\n    | seqkit replace -p '.*' -r '{nr}' \\\n    | tee virues.shuffled.fasta\n&gt;1\nTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCAT\n&gt;2\nTGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAA\n&gt;3\nGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAAT\n&gt;4\nTTTTTTTAAAATCTGGCATCATGACACTATACTATACCCATGCTACTCAACTCCCTACCA\n</code></pre> <p>Sum of all files (the sequences order does not matter):</p> <pre><code>$ seqkit sum viru*.fasta\nseqkit.v0.1_DLS_k0_9bbe0abefc26013dffdde952a6725b17    virues.fasta\nseqkit.v0.1_DLS_k0_9bbe0abefc26013dffdde952a6725b17    virues.shuffled.fasta\nseqkit.v0.1_DLS_k0_176250c8d1cde6c385397df525aa1a94    virus-A.fasta\nseqkit.v0.1_DLS_k0_7a813339f9ae686b376b1df55cd596ca    virus-B.fasta\nseqkit.v0.1_DLS_k0_0fd51028bfbfa85ddbdd2b86ef7bd1c1    virus-C.fasta\nseqkit.v0.1_DLS_k0_88b1d20dd0fe0dbf41c00b075fee4e4e    virus-D.fasta\n</code></pre> <p>Circular genomes (the same genomes with different start positions or in reverse complement strand will not affect the result):</p> <pre><code>$ seqkit sum -c -k 21  virus-*.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-A.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-B.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-C.fasta\nseqkit.v0.1_DCD_k21_7efd18ce33380268d3aa335ffd2dd1cc   virus-D.fasta\n\n$ seqkit sum -c -k 51  virus-*.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-A.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-B.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-C.fasta\nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-D.fasta\n\n# collect files with the same genomes\n$ seqkit sum -c -k 51  virus-*.fasta | csvtk fold -Ht -f 1 -v 2 \nseqkit.v0.1_DCD_k51_39e267864fddeafd7a5cacd77e0a6a11   virus-A.fasta; virus-B.fasta; virus-C.fasta; virus-D.fasta\n</code></pre>"},{"location":"usage/#faidx","title":"faidx","text":"<p>Usage</p> <pre><code>create the FASTA index file and extract subsequences\n\nThis command is similar with \"samtools faidx\" but has some extra features:\n\n  1. output full header line with the flag -f\n  2. support regular expression as sequence ID with the flag -r\n  3. if you have large number of IDs, you can use:\n        seqkit faidx seqs.fasta -l IDs.txt\n\nAttention:\n  1. The flag -U/--update-faidx is recommended to ensure the .fai file matches the FASTA file.\n\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit faidx [flags] &lt;fasta-file&gt; [regions...]\n\nFlags:\n  -f, --full-head            print full header line instead of just ID. New fasta index file ending with\n                             .seqkit.fai will be created\n  -h, --help                 help for faidx\n  -i, --ignore-case          ignore case\n  -I, --immediate-output     print output immediately, do not use write buffer\n  -l, --region-file string   file containing a list of regions\n  -U, --update-faidx         update the fasta index file if it exists. Use this if you are not sure\n                             whether the fasta file changed\n  -r, --use-regexp           IDs are regular expression. But subseq region is not supported here.\n\n</code></pre> <p>Example</p> <ol> <li> <p>common usage like <code>samtools faidx</code></p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1 hsa-let-7a-2\n&gt;hsa-let-7a-1\nUGGGAUGAGGUAGUAGGUUGUAUAGUUUUAGGGUCACACCCACCACUGGGAGAUAACUAU\nACAAUCUACUGUCUUUCCUA\n&gt;hsa-let-7a-2\nAGGUUGAGGUAGUAGGUUGUAUAGUUUAGAAUUACAUCAAGGGAGAUAACUGUACAGCCU\nCCUAGCUUUCCU\n</code></pre> </li> <li> <p>output full header, not supported by <code>samtools faidx</code></p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1 hsa-let-7a-2 -f\n&gt;hsa-let-7a-1 MI0000060 Homo sapiens let-7a-1 stem-loop\nUGGGAUGAGGUAGUAGGUUGUAUAGUUUUAGGGUCACACCCACCACUGGGAGAUAACUAU\nACAAUCUACUGUCUUUCCUA\n&gt;hsa-let-7a-2 MI0000061 Homo sapiens let-7a-2 stem-loop\nAGGUUGAGGUAGUAGGUUGUAUAGUUUAGAAUUACAUCAAGGGAGAUAACUGUACAGCCU\nCCUAGCUUUCCU\n</code></pre> </li> <li> <p>extract subsequence of specific region</p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:1-10\n&gt;hsa-let-7a-1:1-10\nUGGGAUGAGG\n\n$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:-10--1\n&gt;hsa-let-7a-1:-10--1\nGUCUUUCCUA\n\n$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:1\n&gt;hsa-let-7a-1:1-1\nU\n</code></pre> </li> <li> <p>supporting <code>begin</code> &gt; <code>start</code>, i.e., returning reverse complement sequence, not supported by <code>samtools faidx</code></p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa-let-7a-1:10-1\n&gt;hsa-let-7a-1:10-1\nCCUCAUCCCA\n</code></pre> </li> <li> <p>use regular expression</p> <pre><code>$ seqkit faidx tests/hairpin.fa hsa -r | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA      1,881  154,002       41     81.9      180\n</code></pre> </li> </ol>"},{"location":"usage/#watch","title":"watch","text":"<p>Usage</p> <pre><code>monitoring and online histograms of sequence features\n\nUsage:\n  seqkit watch [flags]\n\nFlags:\n  -B, --bins int              number of histogram bins (default -1)\n  -W, --delay int             sleep this many seconds after online plotting (default 1)\n  -y, --dump                  print histogram data to stderr instead of plotting\n  -f, --fields string         target fields, available values: ReadLen, MeanQual, GC, GCSkew (default\n                              \"ReadLen\")\n  -h, --help                  help for watch\n  -O, --img string            save histogram to this PDF/image file\n  -H, --list-fields           print out a list of available fields\n  -L, --log                   log10(x+1) transform numeric values\n  -x, --pass                  pass through mode (write input to stdout)\n  -p, --print-freq int        print/report after this many records (-1 for print after EOF) (default -1)\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n  -Q, --quiet-mode            supress all plotting to stderr\n  -R, --reset                 reset histogram after every report\n  -v, --validate-seq          validate bases according to the alphabet\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Histogram of sequence length</p> <pre><code>seqkit watch --fields ReadLen nanopore.fq.gz -O len.png\n</code></pre> <p></p> </li> <li> <p>Dynamic histogram of log sequence length</p> <pre><code>seqkit watch --log --fields ReadLen nanopore.fq.gz\n</code></pre> <p></p> </li> <li> <p>Histogram of mean base qualities every 500 record, also saved as png</p> <pre><code>seqkit watch -p 500 -O qhist.png -f MeanQual nanopore.fq.gz\n</code></pre> </li> </ol>"},{"location":"usage/#sana","title":"sana","text":"<p>Usage</p> <pre><code>sanitize broken single line FASTQ files\n\nSana is a resilient FASTQ/FASTA parser. Unlike many parsers,\nit won't stop at the first error. Instead, it skips malformed records\nand continues processing the file.\n\nSana currently supports this FASTQ dialect:\n\n  - One line for each sequence and quality value\n\nUsage:\n  seqkit sana [flags]\n\nFlags:\n  -A, --allow-gaps            allow gap character (-) in sequences\n  -i, --format string         input and output format: fastq or fasta (default \"fastq\")\n  -h, --help                  help for sana\n  -I, --in-format string      input format: fastq or fasta\n  -O, --out-format string     output format: fastq or fasta\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n</code></pre> <p>Examples</p> <ol> <li> <p>Rescue usable reads from fastq file with malformed records.</p> <pre><code>seqkit sana broken.fq.gz -o rescued.fq.gz\n</code></pre> </li> <li> <p>For paired reads, you can process both files and pair them with <code>pair</code>.</p> <pre><code>seqkit sana broken_1.fq.gz -o rescued_1.fq.gz\nseqkit sana broken_2.fq.gz -o rescued_2.fq.gz\n\nseqkit pair -1 rescued_1.fq.gz -2 rescued_2.fq.gz -O rescued/\n</code></pre> </li> </ol>"},{"location":"usage/#scat","title":"scat","text":"<p>Usage</p> <pre><code>real time recursive concatenation and streaming of fastx files\n\nUsage:\n  seqkit scat [flags]\n\nFlags:\n  -A, --allow-gaps            allow gap character (-) in sequences\n  -d, --delta int             minimum size increase in kilobytes to trigger parsing (default 5)\n  -D, --drop-time string      Notification drop interval (default \"500ms\")\n  -f, --find-only             concatenate exisiting files and quit\n  -i, --format string         input and output format: fastq or fasta (fastq) (default \"fastq\")\n  -g, --gz-only               only look for gzipped files (.gz suffix)\n  -h, --help                  help for scat\n  -I, --in-format string      input format: fastq or fasta (fastq)\n  -O, --out-format string     output format: fastq or fasta\n  -b, --qual-ascii-base int   ASCII BASE, 33 for Phred+33 (default 33)\n  -r, --regexp string         regexp for watched files, by default guessed from the input format\n  -T, --time-limit string     quit after inactive for this time period\n  -p, --wait-pid int          after process with this PID exited (default -1)\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Concatenate all fastq files recursively under a directory</p> <pre><code>seqkit scat -j 4 -f fastq_dir &gt; all_records.fq\n</code></pre> </li> <li> <p>Watch a directory and stream fastq records in real time until interrupt is recieved and plot read lengths using <code>seqkit watch</code>:</p> <pre><code>seqkit scat -j 4 fastq_dir | seqkit watch -f ReadLen -\n</code></pre> </li> <li> <p>Watch a directory and stream fastq records in real time until there is no write activity under the directory for 5 seconds:</p> <pre><code>seqkit scat -j 4 -T \"5s\" fastq_dir &gt; all_records.fq\n</code></pre> </li> <li> <p>Watch a directory and stream fastq records in real time until a process with a specified PID is alive:</p> <pre><code>seqkit scat -j 4 -p $PID fastq_dir &gt; all_records.fq\n</code></pre> </li> </ol> <p>Notes: You might need to increase the <code>ulimit</code> allowance on open files if you intend to stream fastx records from a large number of files.</p>"},{"location":"usage/#fq2fa","title":"fq2fa","text":"<p>Usage</p> <pre><code>convert FASTQ to FASTA\n\nUsage:\n  seqkit fq2fa [flags]\n\n</code></pre> <p>Examples</p> <pre><code>seqkit fq2fa reads_1.fq.gz -o reads_1.fa.gz\n</code></pre>"},{"location":"usage/#fa2fq","title":"fa2fq","text":"<p>Usage</p> <pre><code>retrieve corresponding FASTQ records by a FASTA file\n\nAttention:\n  1. We assume the FASTA file comes from the FASTQ file,\n     so they share sequence IDs, and sequences in FASTA\n     should be subseq of sequences in FASTQ file.\n\nUsage:\n  seqkit fa2fq [flags]\n\nFlags:\n  -f, --fasta-file string      FASTA file)\n  -h, --help                   help for fa2fq\n  -P, --only-positive-strand   only search on positive strand\n</code></pre>"},{"location":"usage/#fx2tab-tab2fx","title":"fx2tab &amp; tab2fx","text":"<p>Usage (fx2tab)</p> <pre><code>convert FASTA/Q to tabular format, and provide various information,\nlike sequence length, GC content/GC skew.\n\nAttention:\n  1. Fixed three columns (ID, sequence, quality) are outputted for either FASTA\n     or FASTQ, except when flag -n/--name is on. This is for format compatibility.\n\nUsage:\n  seqkit fx2tab [flags]\n\nFlags:\n  -a, --alphabet               print alphabet letters\n  -q, --avg-qual               print average quality of a read\n  -B, --base-content strings   print base content. (case ignored, multiple values supported) e.g. -B AT -B N\n  -C, --base-count strings     print base count. (case ignored, multiple values supported) e.g. -C AT -C N\n  -I, --case-sensitive         calculate case sensitive base content/sequence hash\n  -g, --gc                     print GC content\n  -G, --gc-skew                print GC-Skew\n  -H, --header-line            print header line\n  -h, --help                   help for fx2tab\n  -l, --length                 print sequence length\n  -n, --name                   only print names (no sequences and qualities)\n  -Q, --no-qual                only output two column even for FASTQ file\n  -i, --only-id                print ID instead of full head\n  -b, --qual-ascii-base int    ASCII BASE, 33 for Phred+33 (default 33)\n  -s, --seq-hash               print hash (MD5) of sequence\n\n</code></pre> <p>Usage (tab2fx)</p> <pre><code>convert tabular format (first two/three columns) to FASTA/Q format\n\nUsage:\n  seqkit tab2fx [flags] \n\nFlags:\n  -b, --buffer-size string            size of buffer, supported unit: K, M, G. You need increase the\n                                      value when \"bufio.Scanner: token too long\" error reported (default\n                                      \"1G\")\n  -p, --comment-line-prefix strings   comment line prefix (default [#,//])\n  -h, --help                          help for tab2fx\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Default output</p> <pre><code>$ seqkit fx2tab hairpin.fa.gz | head -n 2\ncel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop      UACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAACUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\ncel-lin-4 MI0000002 Caenorhabditis elegans lin-4 stem-loop      AUGCUUCCGGCCUGUUCCCUGAGACCUCAAGUGUGAGUGUACUAUUGAUGCUUCACACCUGGGCUCUCCGGGUACCAGGACGGUUUGAGCAGAU\n</code></pre> </li> <li> <p>Print sequence length, GC content, and only print names (no sequences), we could also print title line by flag <code>-H</code>.</p> <pre><code>$ seqkit fx2tab hairpin.fa.gz -l -g -n -i -H | head -n 4 | csvtk -t -C '&amp;' pretty\n#name       seq   qual   length   GC\ncel-let-7                99       43.43\ncel-lin-4                94       54.26\ncel-mir-1                96       40.62\n</code></pre> </li> <li> <p>Use fx2tab and tab2fx in pipe</p> <pre><code>$ zcat hairpin.fa.gz | seqkit fx2tab | seqkit tab2fx\n\n$ zcat reads_1.fq.gz | seqkit fx2tab | seqkit tab2fx\n</code></pre> </li> <li> <p>Sort sequences by length (use <code>seqkit sort -l</code>)</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit fx2tab -l \\\n    | sort -t\"`echo -e '\\t'`\" -n -k4,4 \\\n    | seqkit tab2fx\n&gt;cin-mir-4129 MI0015684 Ciona intestinalis miR-4129 stem-loop\nUUCGUUAUUGGAAGACCUUAGUCCGUUAAUAAAGGCAUC\n&gt;mmu-mir-7228 MI0023723 Mus musculus miR-7228 stem-loop\nUGGCGACCUGAACAGAUGUCGCAGUGUUCGGUCUCCAGU\n&gt;cin-mir-4103 MI0015657 Ciona intestinalis miR-4103 stem-loop\nACCACGGGUCUGUGACGUAGCAGCGCUGCGGGUCCGCUGU\n\n$ seqkit sort -l hairpin.fa.gz\n</code></pre> <p>Sorting or filtering by GC (or other base by -flag <code>-B</code>) content could also achieved in similar way.</p> </li> <li> <p>Get first 1000 sequences (use <code>seqkit head -n 1000</code>)</p> <pre><code>$ seqkit fx2tab hairpin.fa.gz | head -n 1000 | seqkit tab2fx\n\n$ seqkit fx2tab reads_1.fq.gz | head -n 1000 | seqkit tab2fx\n</code></pre> </li> </ol> <p>Extension</p> <p>After converting FASTA to tabular format with <code>seqkit fx2tab</code>, it could be handled with CSV/TSV tools,  e.g. csvtk, a cross-platform, efficient and practical CSV/TSV toolkit</p> <ul> <li><code>csvtk grep</code> could be used to filter sequences (similar with <code>seqkit grep</code>)</li> <li><code>csvtk inter</code> computates intersection of multiple files. It could achieve similar function as <code>seqkit common -n</code> along with shell.</li> <li><code>csvtk join</code> joins multiple CSV/TSV files by multiple IDs.</li> </ul>"},{"location":"usage/#convert","title":"convert","text":"<p>Usage</p> <pre><code>convert FASTQ quality encoding between Sanger, Solexa and Illumina\n\nUsage:\n  seqkit convert [flags]\n\nFlags:\n  -d, --dry-run                         dry run\n  -f, --force                           for Illumina-1.8+ -&gt; Sanger, truncate scores &gt; 40 to 40\n      --from string                     source quality encoding. if not given, we'll guess it\n  -h, --help                            help for convert\n  -n, --nrecords int                    number of records for guessing quality encoding (default 1000)\n  -N, --thresh-B-in-n-most-common int   threshold of 'B' in top N most common quality for guessing\n                                        Illumina 1.5. (default 2)\n  -F, --thresh-illumina1.5-frac float   threshold of faction of Illumina 1.5 in the leading N records\n                                        (default 0.1)\n      --to string                       target quality encoding (default \"Sanger\")\n</code></pre> <p>Examples:</p> <p>Note that <code>seqkit convert</code> always output sequences.</p> <p>The test dataset contains score 41 (<code>J</code>):</p> <pre><code>$ seqkit head -n 1 tests/Illimina1.8.fq.gz\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n#AAAFAAJFFFJJJ&lt;JJJJJFFFJFJJJJJFJJAJJJFJJFJFJJJJFAFJ&lt;JA&lt;FFJ7FJJFJJAAJJJJ&lt;JJJJJJJFJJJAJJJJJFJJ77&lt;JJJJ-F7A-FJFFJJJJJJ&lt;FFJ-&lt;7FJJJFJJ)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>By default, nothing changes when converting Illumina 1.8 to Sanger. A warning message show that source and target quality encoding match.</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz  | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Sanger\n[WARN] source and target quality encoding match.\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n#AAAFAAJFFFJJJ&lt;JJJJJFFFJFJJJJJFJJAJJJFJJFJFJJJJFAFJ&lt;JA&lt;FFJ7FJJFJJAAJJJJ&lt;JJJJJJJFJJJAJJJJJFJJ77&lt;JJJJ-F7A-FJFFJJJJJJ&lt;FFJ-&lt;7FJJJFJJ)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>When switching flag <code>--force</code> on,  <code>J</code> (41) was converted to <code>I</code> (40).</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz -f | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Sanger\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n#AAAFAAIFFFIII&lt;IIIIIFFFIFIIIIIFIIAIIIFIIFIFIIIIFAFI&lt;IA&lt;FFI7FIIFIIAAIIII&lt;IIIIIIIFIIIAIIIIIFII77&lt;IIII-F7A-FIFFIIIIII&lt;FFI-&lt;7FIIIFII)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>Other cases:</p> <p>To Illumina-1.5.</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz --to Illumina-1.5+ | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Illumina-1.5+\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\nB```e``ieeeiii[iiiiieeeieiiiiieii`iiieiieieiiiie`ei[i`[eeiVeiieii``iiii[iiiiiiieiii`iiiiieiiVV[iiiiLeV`Leieeiiiiii[eeiL[VeiiieiiH`VHV``[VLLH[LVeL`Ve`[\n</code></pre> <p>To Illumina-1.5 and back to Sanger.</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz --to Illumina-1.5+ | seqkit convert | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.8+]\n[INFO] guessed quality encoding: Illumina-1.8+\n[INFO] converting Illumina-1.8+ -&gt; Illumina-1.5+\n[INFO] possible quality encodings: [Illumina-1.5+]\n[INFO] guessed quality encoding: Illumina-1.5+\n[INFO] converting Illumina-1.5+ -&gt; Sanger\n@ST-E00493:56:H33MFALXX:4:1101:23439:1379 1:N:0:NACAACCA\nNCGTGGAAAGACGCTAAGATTGTGATGTGCTTCCCTGACGATTACAACTGGCGTAAGGACGTTTTGCCTACCTATAAGGCTAACCGTAAGGGTTCTCGCAAGCCTGTAGGTTACAAGAGGTTCGTAGCCGAAGTGATGGCTGACTCACGG\n+\n!AAAFAAJFFFJJJ&lt;JJJJJFFFJFJJJJJFJJAJJJFJJFJFJJJJFAFJ&lt;JA&lt;FFJ7FJJFJJAAJJJJ&lt;JJJJJJJFJJJAJJJJJFJJ77&lt;JJJJ-F7A-FJFFJJJJJJ&lt;FFJ-&lt;7FJJJFJJ)A7)7AA&lt;7--)&lt;-7F-A7FA&lt;\n</code></pre> <p>Checking encoding</p> <pre><code>$ seqkit convert tests/Illimina1.8.fq.gz --from Solexa\n[INFO] converting Solexa -&gt; Sanger\n[ERRO] seq: invalid Solexa quality\n</code></pre> <p>Real Illumina 1.5+ data</p> <pre><code>$ seqkit seq tests/Illimina1.5.fq\n@HWI-EAS209_0006_FC706VJ:5:58:5894:21141#ATCACG/1\nTTAATTGGTAAATAAATCTCCTAATAGCTTAGATNTTACCTTNNNNNNNNNNTAGTTTCTTGAGATTTGTTGGGGGAGACATTTTTGTGATTGCCTTGAT\n+\nefcfffffcfeefffcffffffddf`feed]`]_Ba_^__[YBBBBBBBBBBRTT\\]][]dddd`ddd^dddadd^BBBBBBBBBBBBBBBBBBBBBBBB\n\n$ seqkit convert tests/Illimina1.5.fq | seqkit head -n 1\n[INFO] possible quality encodings: [Illumina-1.5+]\n[INFO] guessed quality encoding: Illumina-1.5+\n[INFO] converting Illumina-1.5+ -&gt; Sanger\n@HWI-EAS209_0006_FC706VJ:5:58:5894:21141#ATCACG/1\nTTAATTGGTAAATAAATCTCCTAATAGCTTAGATNTTACCTTNNNNNNNNNNTAGTTTCTTGAGATTTGTTGGGGGAGACATTTTTGTGATTGCCTTGAT\n+\nFGDGGGGGDGFFGGGDGGGGGGEEGAGFFE&gt;A&gt;@!B@?@@&lt;:!!!!!!!!!!355=&gt;&gt;&lt;&gt;EEEEAEEE?EEEBEE?!!!!!!!!!!!!!!!!!!!!!!!!\n</code></pre>"},{"location":"usage/#translate","title":"translate","text":"<p>Usage</p> <pre><code>translate DNA/RNA to protein sequence (supporting ambiguous bases)\n\nNote:\n\n  1. This command supports codons containing any ambiguous base.\n     Please switch on flag -L INT for details. e.g., for standard table:\n\n        ACN -&gt; T\n        CCN -&gt; P\n        CGN -&gt; R\n        CTN -&gt; L\n        GCN -&gt; A\n        GGN -&gt; G\n        GTN -&gt; V\n        TCN -&gt; S\n\n        MGR -&gt; R\n        YTR -&gt; L\n\nTranslate Tables/Genetic Codes:\n\n    # https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html/index.cgi?chapter=tgencodes\n\n     1: The Standard Code\n     2: The Vertebrate Mitochondrial Code\n     3: The Yeast Mitochondrial Code\n     4: The Mold, Protozoan, and Coelenterate Mitochondrial Code and the Mycoplasma/Spiroplasma Code\n     5: The Invertebrate Mitochondrial Code\n     6: The Ciliate, Dasycladacean and Hexamita Nuclear Code\n     9: The Echinoderm and Flatworm Mitochondrial Code\n    10: The Euplotid Nuclear Code\n    11: The Bacterial, Archaeal and Plant Plastid Code\n    12: The Alternative Yeast Nuclear Code\n    13: The Ascidian Mitochondrial Code\n    14: The Alternative Flatworm Mitochondrial Code\n    16: Chlorophycean Mitochondrial Code\n    21: Trematode Mitochondrial Code\n    22: Scenedesmus obliquus Mitochondrial Code\n    23: Thraustochytrium Mitochondrial Code\n    24: Pterobranchia Mitochondrial Code\n    25: Candidate Division SR1 and Gracilibacteria Code\n    26: Pachysolen tannophilus Nuclear Code\n    27: Karyorelict Nuclear\n    28: Condylostoma Nuclear\n    29: Mesodinium Nuclear\n    30: Peritrich Nuclear\n    31: Blastocrithidia Nuclear\n\nUsage:\n  seqkit translate [flags]\n\nFlags:\n  -x, --allow-unknown-codon                     translate unknown code to 'X'. And you may not use flag\n                                                --trim which removes 'X'\n  -F, --append-frame                            append frame information to sequence ID\n      --clean                                   change all STOP codon positions from the '*' character\n                                                to 'X' (an unknown residue)\n  -f, --frame strings                           frame(s) to translate, available value: 1, 2, 3, -1, -2,\n                                                -3, and 6 for all six frames (default [1])\n  -h, --help                                    help for translate\n  -M, --init-codon-as-M                         translate initial codon at beginning to 'M'\n  -l, --list-transl-table int                   show details of translate table N, 0 for all (default -1)\n  -L, --list-transl-table-with-amb-codons int   show details of translate table N (including ambigugous\n                                                codons), 0 for all.  (default -1)\n  -m, --min-len int                             the minimum length of amino acid sequence\n  -s, --out-subseqs                             output individual amino acid subsequences seperated by\n                                                the stop symbol \"*\"\n  -T, --transl-table int                        translate table/genetic code, type 'seqkit translate\n                                                --help' for more details (default 1)\n      --trim                                    remove all 'X' and '*' characters from the right end of\n                                                the translation\n\n</code></pre> <p>Examples</p> <ol> <li> <p>common usage</p> <pre><code>$ seqkit translate tests/mouse-p53-cds.fna\n&gt;lcl|AB021961.1_cds_BAA82344.1_1 [gene=p53] [protein=P53] [protein_id=BAA82344.1] [location=101..1273] [gbkey=CDS]\nMTAMEESQSDISLELPLSQETFSGLWKLLPPEDILPSPHCMDDLLLPQDVEEFFEGPSEA\nLRVSGAPAAQDPVTETPGPVAPAPATPWPLSSFVPSQKTYQGNYGFHLGFLQSGTAKSVM\nCTYSPPLNKLFCQLAKTCPVQLWVSATPPAGSRVRAMAIYKKSQHMTEVVRRCPHHERCS\nDGDGLAPPQHRIRVEGNLYPEYLEDRQTFRHSVVVPYEPPEAGSEYTTIHYKYMCNSSCM\nGGMNRRPILTIITLEDSSGNLLGRDSFEVRVCACPGRDRRTEEENFRKKEVLCPELPPGS\nAKRALPTCTSASPPQKKKPLDGEYFTLKIRGRKRFEMFRELNEALELKDAHATEESGDSR\nAHSSYLKTKKGQSTSRHKKTMVKKVGPDSD*\n</code></pre> </li> <li> <p>trim the <code>*</code></p> <pre><code>$ seqkit translate tests/mouse-p53-cds.fna --trim\n&gt;lcl|AB021961.1_cds_BAA82344.1_1 [gene=p53] [protein=P53] [protein_id=BAA82344.1] [location=101..1273] [gbkey=CDS]\nMTAMEESQSDISLELPLSQETFSGLWKLLPPEDILPSPHCMDDLLLPQDVEEFFEGPSEA\nLRVSGAPAAQDPVTETPGPVAPAPATPWPLSSFVPSQKTYQGNYGFHLGFLQSGTAKSVM\nCTYSPPLNKLFCQLAKTCPVQLWVSATPPAGSRVRAMAIYKKSQHMTEVVRRCPHHERCS\nDGDGLAPPQHRIRVEGNLYPEYLEDRQTFRHSVVVPYEPPEAGSEYTTIHYKYMCNSSCM\nGGMNRRPILTIITLEDSSGNLLGRDSFEVRVCACPGRDRRTEEENFRKKEVLCPELPPGS\nAKRALPTCTSASPPQKKKPLDGEYFTLKIRGRKRFEMFRELNEALELKDAHATEESGDSR\nAHSSYLKTKKGQSTSRHKKTMVKKVGPDSD\n</code></pre> </li> <li> <p>different translate table</p> <pre><code>$ cat tests/Lactococcus-lactis-phage-BK5-T-ORF25.fasta \\\n    | seqkit translate -T 11 --trim\n&gt;CAC80166.1 hypothetical protein [Lactococcus phage BK5-T]\nMEEQAWREVLERLARIETKLDNYETVRDKAERALLIAQSNAKLIEKMEANNKWAWGFMLT\nLAVTVIGYLFTKIRF\n</code></pre> </li> <li> <p>different frame</p> <pre><code>$ cat tests/Lactococcus-lactis-phage-BK5-T-ORF25.fasta \\\n    | seqkit translate -T 11 --frame -1\n&gt;CAC80166.1 hypothetical protein [Lactococcus phage BK5-T]\nSESNFSE*ITNNSYGKSKHKAPSPLIISFHFFYKFRI*LSY*ERSFCFISNCFIVI*LCF\nNSS*TFEDFSPCLFLH\n\n$ cat tests/Lactococcus-lactis-phage-BK5-T-ORF25.fasta \\\n    | seqkit seq -r -p \\\n    | seqkit translate -T 11 --frame -1\n&gt;CAC80166.1 hypothetical protein [Lactococcus phage BK5-T]\nMEEQAWREVLERLARIETKLDNYETVRDKAERALLIAQSNAKLIEKMEANNKWAWGFMLT\nLAVTVIGYLFTKIRF*\n</code></pre> </li> <li> <p>show details of translate table 1</p> <pre><code>$ seqkit translate -l 1\nThe Standard Code (transl_table=1)\nSource: https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html/index.cgi?chapter=tgencodes#SG1\n\nInitiation Codons:\n    ATG, CTG, TTG\n\nStop Codons:\n    TAA, TAG, TGA\n\nStranslate Table:\n    AAA: K, AAC: N, AAG: K, AAT: N\n    ACA: T, ACC: T, ACG: T, ACT: T\n    AGA: R, AGC: S, AGG: R, AGT: S\n    ATA: I, ATC: I, ATG: M, ATT: I\n\n    CAA: Q, CAC: H, CAG: Q, CAT: H\n    CCA: P, CCC: P, CCG: P, CCT: P\n    CGA: R, CGC: R, CGG: R, CGT: R\n    CTA: L, CTC: L, CTG: L, CTT: L\n\n    GAA: E, GAC: D, GAG: E, GAT: D\n    GCA: A, GCC: A, GCG: A, GCT: A\n    GGA: G, GGC: G, GGG: G, GGT: G\n    GTA: V, GTC: V, GTG: V, GTT: V\n\n    TAA: *, TAC: Y, TAG: *, TAT: Y\n    TCA: S, TCC: S, TCG: S, TCT: S\n    TGA: *, TGC: C, TGG: W, TGT: C\n    TTA: L, TTC: F, TTG: L, TTT: F\n</code></pre> </li> <li> <p>show details of translate table 1, including ambigugous codons</p> <pre><code>$ seqkit translate -L 1\nThe Standard Code (transl_table=1)\nSource: https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html/index.cgi?chapter=tgencodes#SG1\n\nInitiation Codons:\n    ATG, CTG, TTG\n\nStop Codons:\n    TAA, TAG, TGA\n\nStranslate Table:\n    AAA: K, AAC: N, AAG: K, AAR: K, AAT: N, AAY: N\n    ACA: T, ACC: T, ACM: T, ACG: T, ACR: T, ACS: T, ACV: T, ACT: T, ACW: T, ACY: T, ACH: T, ACK: T, ACD: T, ACB: T, ACN: T\n    AGA: R, AGC: S, AGG: R, AGR: R, AGT: S, AGY: S\n    ATA: I, ATC: I, ATM: I, ATG: M, ATT: I, ATW: I, ATY: I, ATH: I\n\n    CAA: Q, CAC: H, CAG: Q, CAR: Q, CAT: H, CAY: H\n    CCA: P, CCC: P, CCM: P, CCG: P, CCR: P, CCS: P, CCV: P, CCT: P, CCW: P, CCY: P, CCH: P, CCK: P, CCD: P, CCB: P, CCN: P\n    CGA: R, CGC: R, CGM: R, CGG: R, CGR: R, CGS: R, CGV: R, CGT: R, CGW: R, CGY: R, CGH: R, CGK: R, CGD: R, CGB: R, CGN: R\n    CTA: L, CTC: L, CTM: L, CTG: L, CTR: L, CTS: L, CTV: L, CTT: L, CTW: L, CTY: L, CTH: L, CTK: L, CTD: L, CTB: L, CTN: L\n\n    MGA: R, MGG: R, MGR: R\n\n    GAA: E, GAC: D, GAG: E, GAR: E, GAT: D, GAY: D\n    GCA: A, GCC: A, GCM: A, GCG: A, GCR: A, GCS: A, GCV: A, GCT: A, GCW: A, GCY: A, GCH: A, GCK: A, GCD: A, GCB: A, GCN: A\n    GGA: G, GGC: G, GGM: G, GGG: G, GGR: G, GGS: G, GGV: G, GGT: G, GGW: G, GGY: G, GGH: G, GGK: G, GGD: G, GGB: G, GGN: G\n    GTA: V, GTC: V, GTM: V, GTG: V, GTR: V, GTS: V, GTV: V, GTT: V, GTW: V, GTY: V, GTH: V, GTK: V, GTD: V, GTB: V, GTN: V\n\n    TAA: *, TAC: Y, TAG: *, TAR: *, TAT: Y, TAY: Y\n    TCA: S, TCC: S, TCM: S, TCG: S, TCR: S, TCS: S, TCV: S, TCT: S, TCW: S, TCY: S, TCH: S, TCK: S, TCD: S, TCB: S, TCN: S\n    TGA: *, TGC: C, TGG: W, TGT: C, TGY: C\n    TRA: *\n    TTA: L, TTC: F, TTG: L, TTR: L, TTT: F, TTY: F\n\n    YTA: L, YTG: L, YTR: L\n</code></pre> </li> </ol>"},{"location":"usage/#grep","title":"grep","text":"<p>Usage</p> <pre><code>search sequences by ID/name/sequence/sequence motifs, mismatch allowed\n\nAttention:\n\n  0. By default, we match sequence ID with patterns, use \"-n/--by-name\"\n     for matching full name instead of just ID.\n  1. Unlike POSIX/GNU grep, we compare the pattern to the whole target\n     (ID/full header) by default. Please switch \"-r/--use-regexp\" on\n     for partly matching.\n  2. When searching by sequences, it's partly matching, and both positive\n     and negative strands are searched.\n     Please switch on \"-P/--only-positive-strand\" if you would like to\n     search only on the positive strand.\n     Mismatch is allowed using flag \"-m/--max-mismatch\", you can increase\n     the value of \"-j/--threads\" to accelerate processing.\n  3. Degenerate bases/residues like \"RYMM..\" are also supported by flag -d.\n     But do not use degenerate bases/residues in regular expression, you need\n     convert them to regular expression, e.g., change \"N\" or \"X\"  to \".\".\n  4. When providing search patterns (motifs) via flag '-p',\n     please use double quotation marks for patterns containing comma,\n     e.g., -p '\"A{2,}\"' or -p \"\\\"A{2,}\\\"\". Because the command line argument\n     parser accepts comma-separated-values (CSV) for multiple values (motifs).\n     Patterns in file do not follow this rule.\n  5. The order of sequences in result is consistent with that in original\n     file, not the order of the query patterns.\n     But for FASTA file, you can use:\n        seqkit faidx seqs.fasta --infile-list IDs.txt\n  6. For multiple patterns, you can either set \"-p\" multiple times, i.e.,\n     -p pattern1 -p pattern2, or give a file of patterns via \"-f/--pattern-file\".\n\nYou can specify the sequence region for searching with the flag -R (--region).\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit grep [flags]\n\nFlags:\n  -D, --allow-duplicated-patterns   output records multiple times when duplicated patterns are given\n  -n, --by-name                     match by full name instead of just ID\n  -s, --by-seq                      search subseq on seq. Both positive and negative strand are searched\n                                    by default, you might use -P/--only-positive-strand. Mismatch\n                                    allowed using flag -m/--max-mismatch\n  -c, --circular                    circular genome\n  -C, --count                       just print a count of matching records. with the -v/--invert-match\n                                    flag, count non-matching records\n  -d, --degenerate                  pattern/motif contains degenerate base\n      --delete-matched              delete a pattern right after being matched, this keeps the firstly\n                                    matched data and speedups when using regular expressions\n  -h, --help                        help for grep\n  -i, --ignore-case                 ignore case\n  -I, --immediate-output            print output immediately, do not use write buffer\n  -v, --invert-match                invert the sense of matching, to select non-matching records\n  -m, --max-mismatch int            max mismatch when matching by seq. For large genomes like human\n                                    genome, using mapping/alignment tools would be faster\n  -P, --only-positive-strand        only search on the positive strand\n  -p, --pattern strings             search pattern (multiple values supported. Attention: use double\n                                    quotation marks for patterns containing comma, e.g., -p '\"A{2,}\"')\n  -f, --pattern-file string         pattern file (one record per line)\n  -R, --region string               specify sequence region for searching. e.g 1:12 for first 12 bases,\n                                    -12:-1 for last 12 bases\n  -r, --use-regexp                  patterns are regular expression\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Searching with list of sequence IDs (do not contain whitespace)</p> <pre><code>$ seqkit grep -f id.txt seqs.fq.gz -o result.fq.gz\n\n# ignore case\n$ seqkit grep -i -f id.txt seqs.fq.gz -o result.fq.gz\n</code></pre> </li> <li> <p>Just print the matched number, like GNU grep (<code>grep -c</code>) </p> <pre><code>$ seqkit grep -f id.txt seqs.fq.gz -C\n</code></pre> </li> <li> <p>Serching non-canonical sequence IDs, Using <code>--id-regexp</code> to capture IDs.     Refer to section Sequence ID and seqkit seq for examples.</p> </li> <li> <p>Searching with list of sequence names (they may contain whitespace).</p> <pre><code>$ seqkit grep -n -f name.txt seqs.fa.gz -o result.fa.gz\n</code></pre> </li> <li> <p>Useq <code>-r/--use-regexp</code> for partly matching, but this may produce \"false positive\" matches.    For example, <code>seq_1</code> matches <code>seq_10</code> with <code>-nri</code>.</p> </li> <li> <p>Extract human hairpins (i.e. sequences with name starting with <code>hsa</code>)</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -r -p ^hsa\n&gt;hsa-let-7a-1 MI0000060 Homo sapiens let-7a-1 stem-loop\nUGGGAUGAGGUAGUAGGUUGUAUAGUUUUAGGGUCACACCCACCACUGGGAGAUAACUAU\nACAAUCUACUGUCUUUCCUA\n&gt;hsa-let-7a-2 MI0000061 Homo sapiens let-7a-2 stem-loop\nAGGUUGAGGUAGUAGGUUGUAUAGUUUAGAAUUACAUCAAGGGAGAUAACUGUACAGCCU\nCCUAGCUUUCCU\n</code></pre> </li> <li> <p>Remove human and mice hairpins (invert match with <code>-v</code>)</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -r -p ^hsa -p ^mmu -v\n</code></pre> </li> <li> <p>Extract new entries by information from miRNA.diff.gz</p> <ol> <li> <p>Get IDs of new entries.</p> <pre><code>$ zcat miRNA.diff.gz | grep ^# -v | grep NEW | cut -f 2 &gt; list\n$ more list\ncfa-mir-486\ncfa-mir-339-1\npmi-let-7\n</code></pre> </li> <li> <p>Extract by ID list file</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -f list &gt; new.fa\n</code></pre> </li> </ol> </li> <li> <p>Extract sequences containing AGGCG</p> <pre><code>$ cat hairpin.fa.gz | seqkit grep -s -i -p aggcg\n</code></pre> </li> <li> <p>Circular genome</p> <pre><code>$ echo -e \"&gt;seq\\nACGTTGCA\" \n&gt;seq\nACGTTGCA\n\n$ echo -e \"&gt;seq\\nACGTTGCA\"  | seqkit grep -s -i -p AA\n\n$ echo -e \"&gt;seq\\nACGTTGCA\"  | seqkit grep -s -i -p AA -c\n&gt;seq\nACGTTGCA\n</code></pre> </li> <li> <p>Extract sequences containing AGGCG (allow mismatch)</p> <pre><code>$ time cat hairpin.fa.gz | seqkit grep -s -i -p aggcg | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA      1,181  145,789       49    123.4    2,354\n\nreal    0m0.058s\nuser    0m0.100s\nsys     0m0.017s\n\n$ time zcat hairpin.fa.gz | seqkit grep -s -i -p aggcg -m 1 | seqkit stats\nfile  format  type  num_seqs    sum_len  min_len  avg_len  max_len\n-     FASTA   RNA     22,290  2,375,819       39    106.6    2,354\n\nreal    0m1.081s\nuser    0m1.305s\nsys     0m0.158s\n</code></pre> </li> <li> <p>Extract sequences starting with AGGCG</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -s -r -i -p ^aggcg\n</code></pre> </li> <li> <p>Extract sequences with TTSAA (AgsI digest site) in SEQUENCE. Base S stands for C or G.</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -s -d -i -p TTSAA\n</code></pre> <p>It's equal to but simpler than:</p> <pre><code>$ zcat hairpin.fa.gz | seqkit grep -s -r -i -p TT[CG]AA\n</code></pre> </li> <li> <p>Specify sequence regions for searching. e.g., leading 30 bases.</p> <pre><code>$ seqkit grep -s -R 1:30 -i -r -p GCTGG\n</code></pre> </li> </ol>"},{"location":"usage/#locate","title":"locate","text":"<p>Usage</p> <pre><code>locate subsequences/motifs, mismatch allowed\n\nAttention:\n\n  1. Motifs could be EITHER plain sequence containing \"ACTGN\" OR regular\n     expression like \"A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)\" for ORFs.     \n  2. Degenerate bases/residues like \"RYMM..\" are also supported by flag -d.\n     But do not use degenerate bases/residues in regular expression, you need\n     convert them to regular expression, e.g., change \"N\" or \"X\"  to \".\".\n  3. When providing search patterns (motifs) via flag '-p',\n     please use double quotation marks for patterns containing comma, \n     e.g., -p '\"A{2,}\"' or -p \"\\\"A{2,}\\\"\". Because the command line argument\n     parser accepts comma-separated-values (CSV) for multiple values (motifs).\n     Patterns in file do not follow this rule.     \n  4. Mismatch is allowed using flag \"-m/--max-mismatch\",\n     you can increase the value of \"-j/--threads\" to accelerate processing.\n  5. When using flag --circular, end position of matched subsequence that \n     crossing genome sequence end would be greater than sequence length.\n\nUsage:\n  seqkit locate [flags]\n\nFlags:\n      --bed                    output in BED6 format\n  -c, --circular               circular genome. type \"seqkit locate -h\" for details\n  -d, --degenerate             pattern/motif contains degenerate base\n      --gtf                    output in GTF format\n  -h, --help                   help for locate\n  -M, --hide-matched           do not show matched sequences\n  -i, --ignore-case            ignore case\n  -I, --immediate-output       print output immediately, do not use write buffer\n  -s, --max-len-to-show int    show at most X characters for the search pattern or matched sequences\n  -m, --max-mismatch int       max mismatch when matching by seq. For large genomes like human genome,\n                               using mapping/alignment tools would be faster\n  -G, --non-greedy             non-greedy mode, faster but may miss motifs overlapping with others\n  -P, --only-positive-strand   only search on positive strand\n  -p, --pattern strings        pattern/motif (multiple values supported. Attention: use double quotation\n                               marks for patterns containing comma, e.g., -p '\"A{2,}\"')\n  -f, --pattern-file string    pattern/motif file (FASTA format)\n  -F, --use-fmi                use FM-index for much faster search of lots of sequence patterns\n  -r, --use-regexp             patterns/motifs are regular expression\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Locating subsequences (mismatch allowed)</p> <pre><code>$ cat t.fa\n&gt;seq\nagctggagctacc\n\n$ cat t.fa \\\n  | seqkit locate -p agc \\\n  | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end   matched\nseq     agc           agc       +        1       3     agc\nseq     agc           agc       +        7       9     agc\nseq     agc           agc       -        8       10    agc\nseq     agc           agc       -        2       4     agc\n\n# do not show matched sequences\n$ cat t.fa \\\n  | seqkit locate -p agc -M \\\n  | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end\nseq     agc           agc       +        1       3\nseq     agc           agc       +        7       9\nseq     agc           agc       -        8       10\nseq     agc           agc       -        2       4\n\n# max mismatch: 1\n$ cat t.fa \\\n  | seqkit locate -p agc -m 1 \\\n  | csvtk pretty -t\nseqID   patternName     pattern strand  start   end    matched\nseq     agc           agc       +        1       3     agc\nseq     agc           agc       +        7       9     agc\nseq     agc           agc       +        11      13    acc\nseq     agc           agc       -        8       10    agc\nseq     agc           agc       -        2       4     agc\n\n# max mismatch: 2\n$ cat t.fa \\\n  | seqkit locate -p agc -m 2 \\\n  | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end   matched\nseq     agc           agc       +        1       3     agc\nseq     agc           agc       +        4       6     tgg\nseq     agc           agc       +        5       7     gga\nseq     agc           agc       +        7       9     agc\nseq     agc           agc       +        10      12    tac\nseq     agc           agc       +        11      13    acc\nseq     agc           agc       -        11      13    ggt\nseq     agc           agc       -        8       10    agc\nseq     agc           agc       -        6       8     ctc\nseq     agc           agc       -        5       7     tcc\nseq     agc           agc       -        2       4     agc\n</code></pre> </li> <li> <p>Locate ORFs.</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit locate -i -p \"A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)\" -r \\\n    | head -n 4 \\\n    | csvtk pretty -t\nseqID       patternName                        pattern                            strand   start   end   matched\ncel-lin-4   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   +        1       36    AUGCUUCCGGCCUGUUCCCUGAGACCUCAAGUGUGA\ncel-mir-1   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   +        54      95    AUGGAUAUGGAAUGUAAAGAAGUAUGUAGAACGGGGUGGUAG\ncel-mir-1   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   A[TU]G(?:.{3})+?[TU](?:AG|AA|GA)   -        43      51    AUGAUAUAG\n</code></pre> </li> <li> <p>Locate Motif.</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit locate -i -d -p AUGGACUN \\\n    | head -n 4 \\\n    | csvtk pretty -t \nseqID         patternName   pattern    strand   start   end   matched\ncel-mir-58a   AUGGACUN      AUGGACUN   +        81      88    AUGGACUG\nath-MIR163    AUGGACUN      AUGGACUN   -        122     129   AUGGACUC\ncel-mir-270   AUGGACUN      AUGGACUN   +        84      91    AUGGACUG\n</code></pre> </li> <li> <p>Output in <code>GTF</code> or <code>BED6</code> format, which you can use in <code>seqkit subseq</code></p> <pre><code>$ zcat hairpin.fa.gz | seqkit locate -i -d -p AUGGACUN --bed\ncel-mir-58a     80      88      AUGGACUN        0       +\nath-MIR163      121     129     AUGGACUN        0       -\n\n$ zcat hairpin.fa.gz | seqkit locate -i -d -p AUGGACUN --gtf\ncel-mir-58a     SeqKit  location        81      88      0       +       .       gene_id \"AUGGACUN\";\nath-MIR163      SeqKit  location        122     129     0       -       .       gene_id \"AUGGACUN\";\n</code></pre> </li> <li> <p>Greedy mode (default)</p> <pre><code> $ echo -e '&gt;seq\\nACGACGACGA' | seqkit locate -p ACGA | csvtk -t pretty\n seqID   patternName   pattern   strand   start   end   matched\n seq     ACGA          ACGA      +        1       4     ACGA\n seq     ACGA          ACGA      +        4       7     ACGA\n seq     ACGA          ACGA      +        7       10    ACGA\n</code></pre> </li> <li> <p>Non-greedy mode (<code>-G</code>)</p> <pre><code>$ echo -e '&gt;seq\\nACGACGACGA' | seqkit locate -p ACGA -G | csvtk -t pretty\nseqID   patternName   pattern   strand   start   end   matched\nseq     ACGA          ACGA      +        1       4     ACGA\nseq     ACGA          ACGA      +        7       10    ACGA\n</code></pre> </li> <li> <p>Circular genome. Note that end position of matched subsequence that  crossing genome sequence end would be greater than sequence length.</p> <pre><code>$ echo -e \"&gt;seq\\nACGTTGCA\"\n&gt;seq\nACGTTGCA\n\n$ echo -e \"&gt;seq\\nACGTTGCA\" \\\n    | seqkit locate -i -p aa\nseqID   patternName     pattern strand  start   end     matched\nseq     aa      aa      -       4       5       aa\n\n$ echo -e \"&gt;seq\\nACGTTGCA\" \\\n    | seqkit locate -i -p aa -c \\\n    | csvtk pretty -t\nseqID   patternName   pattern   strand   start   end   matched\nseq     aa            aa        +        8       9     aa\nseq     aa            aa        -        4       5     aa\n</code></pre> </li> </ol>"},{"location":"usage/#fish","title":"fish","text":"<p>Usage</p> <pre><code>look for short sequences in larger sequences using local alignment\n\nAttention:\n  1. output coordinates are BED-like 0-based, left-close and right-open.\n  2. alignment information are printed to STDERR.\n\nUsage:\n  seqkit fish [flags]\n\nFlags:\n  -a, --all                      search all\n  -p, --aln-params string        alignment parameters in format\n                                 \"&lt;match&gt;,&lt;mismatch&gt;,&lt;gap_open&gt;,&lt;gap_extend&gt;\" (default \"4,-4,-2,-1\")\n  -h, --help                     help for fish\n  -i, --invert                   print out references not matching with any query\n  -q, --min-qual float           minimum mapping quality (default 5)\n  -b, --out-bam string           save aligmnets to this BAM file (memory intensive)\n  -x, --pass                     pass through mode (write input to stdout)\n  -g, --print-aln                print sequence alignments\n  -D, --print-desc               print full sequence header\n  -f, --query-fastx string       query fasta\n  -F, --query-sequences string   query sequences\n  -r, --ranges string            target ranges, for example: \":10,30:40,-20:\"\n  -s, --stranded                 search + strand only\n  -v, --validate-seq             validate bases according to the alphabet\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Find best local alignment of a short sequence in reads in a fasta file, print results as tabular</p> <pre><code>$ seqkit fish -q 4.7 -F GGCGGCTGTGACC -g mouse-p53-cds.fna\n</code></pre> </li> <li> <p>Compare to <code>seqkit locate</code>:</p> <pre><code>$ echo -e '&gt;seq\\nACGACGACGA' \\\n    | seqkit locate -p ACGA -G | csvtk -t pretty\nseqID   patternName   pattern   strand   start   end   matched\nseq     ACGA          ACGA      +        1       4     ACGA\nseq     ACGA          ACGA      +        7       10    ACGA\n\n$ echo -e '&gt;seq\\nACGACGACGA' \\\n    | seqkit fish -F ACGA -a 2&gt;&amp;1 | csvtk -t pretty \nRef   RefStart   RefEnd   Query   QueryStart   QueryEnd   Strand   MapQual   RawScore   Acc      ClipAcc   QueryCov\nseq   6          10       q0      0            4          +        60.00     16         100.00   100.00    100.00\nseq   0          4        q0      0            4          +        60.00     16         100.00   100.00    100.00\n</code></pre> </li> <li> <p>Find all local alignment of a short sequences in reads in a fasta file, print results as tabular and save as BAM</p> <pre><code>seqkit fish -a -q 4.67 -f query.fas -b alignments.bam -g mouse-p53-cds.fna\n</code></pre> </li> </ol>"},{"location":"usage/#amplicon","title":"amplicon","text":"<p>Usage</p> <pre><code>extract amplicon (or specific region around it) via primer(s).\n\nAttention:\n  1. Only one (the longest) matching location is returned for every primer pair.\n  2. Mismatch is allowed, but the mismatch location (5' or 3') is not controled. \n     You can increase the value of \"-j/--threads\" to accelerate processing.\n  3. Degenerate bases/residues like \"RYMM..\" are also supported.\n     But do not use degenerate bases/residues in regular expression, you need\n     convert them to regular expression, e.g., change \"N\" or \"X\"  to \".\".\n\nExamples:\n  0. no region given.\n\n                    F\n        -----===============-----\n             F             R\n        -----=====-----=====-----\n\n             ===============         amplicon\n\n  1. inner region (-r x:y).\n\n                    F\n        -----===============-----\n             1 3 5                    x/y\n                      -5-3-1          x/y\n             F             R\n        -----=====-----=====-----     x:y\n\n             ===============          1:-1\n             =======                  1:7\n               =====                  3:7\n                  =====               6:10\n                  =====             -10:-6\n                     =====           -7:-3\n                                     -x:y (invalid)\n\n  2. flanking region (-r x:y -f)\n\n                    F\n        -----===============-----\n         -3-1                        x/y\n                            1 3 5    x/y\n             F             R\n        -----=====-----=====-----\n\n        =====                        -5:-1\n        ===                          -5:-3\n                            =====     1:5\n                              ===     3:5\n            =================        -1:1\n        =========================    -5:5\n                                      x:-y (invalid)\n\nUsage:\n  seqkit amplicon [flags]\n\nFlags:\n      --bed                    output in BED6+1 format with amplicon as the 7th column\n  -f, --flanking-region        region is flanking region\n  -F, --forward string         forward primer (5'-primer-3'), degenerate bases allowed\n  -h, --help                   help for amplicon\n  -I, --immediate-output       print output immediately, do not use write buffer\n  -m, --max-mismatch int       max mismatch when matching primers, no degenerate bases allowed\n  -P, --only-positive-strand   only search on positive strand\n  -M, --output-mismatches      append the total mismatches and mismatches of 5' end and 3' end\n  -p, --primer-file string     3- or 2-column tabular primer file, with first column as primer name\n  -r, --region string          specify region to return. type \"seqkit amplicon -h\" for detail\n  -R, --reverse string         reverse primer (5'-primer-3'), degenerate bases allowed\n  -u, --save-unmatched         also save records that do not match any primer\n  -s, --strict-mode            strict mode, i.e., discarding seqs not fully matching (shorter) given\n                               region range\n\n</code></pre> <p>Examples</p> <ol> <li> <p>No region given.</p> <pre><code>$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \n&gt;seq\nacgcccactgaaatga\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt\n&gt;seq\ncccactgaaa\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt\n\n# BED6+1\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt --bed\nseq     3       13      .       0       +       cccactgaaa\n\n# supporting degenerate bases.\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccR -R ttt --bed\nseq     4       13      .       0       +       ccactgaaa\n</code></pre> </li> <li> <p>Output mismatches:</p> <pre><code>$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -M\n&gt;seq mismatches=0(0+0)\ncccactgaaa\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -m 1 -M\n&gt;seq mismatches=2(1+1)\ncgcccactgaaat\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -m 1 -M --bed\nseq     1       14      .       0       +       cgcccactgaaat   2       1       1\n</code></pre> </li> <li> <p>Load primers from 3- or 2-column tabular primer file, with first column as primer name.</p> <pre><code>$ cat seqs4amplicon.fa \n&gt;seq1\nACGCCCACTGAAATGA\n&gt;seq2\nACGTACGGTCAGATCCA\n\n$ cat primers.tsv \np1      ccc     ttt\np2      ttt     ccc\np3      ttt\n\np4      CG      TG\nP5      CG      GA\n\n# containing degenerate bases\np6      TRC     WGG\n\n$ cat seqs4amplicon.fa | seqkit amplicon -p primers.tsv --bed\nseq1    3       13      p1      0       +       CCCACTGAAA\nseq1    1       7       p4      0       +       CGCCCA\nseq1    3       13      p2      0       -       TTTCAGTGGG\nseq1    10      13      p3      0       -       TTT\nseq2    1       17      p4      0       +       CGTACGGTCAGATCCA\nseq2    1       15      P5      0       +       CGTACGGTCAGATC\nseq2    3       17      p6      0       +       TACGGTCAGATCCA\n</code></pre> </li> <li> <p>Inner region</p> <pre><code># region right behind forward primer\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -r 4:7\n&gt;seq\nactg\n\n# BED\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -r 4:7 --bed\nseq     6       10      .       0       +       actg\n\n# more common case is triming primers\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -R ttt -r 4:-4\n&gt;seq\nactg\n</code></pre> </li> <li> <p>flanking region</p> <pre><code># in one of my sequencing data, I only care about \n# region downstream of forward primer\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -f -r 3:6\n&gt;seq\ntgaa\n\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F ccc -f -r 3:6 --bed\nseq     8       12      .       0       +       tgaa\n\n# if given region if out scope of sequence. e.g,\n# 2-5bp downstream of aaa, we can get part of region (2-4) by default\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F aaa -f -r 2:5\n&gt;seq\nga\n\n# you can also use strict mode to discard those cases\n$ echo -ne \"&gt;seq\\nacgcccactgaaatga\\n\" \\\n    | seqkit amplicon -F aaa -f -r 2:5 -s\n</code></pre> </li> </ol>"},{"location":"usage/#duplicate","title":"duplicate","text":"<p>Usage</p> <pre><code>duplicate sequences N times\n\nYou may need \"seqkit rename\" to make the the sequence IDs unique.\n\nUsage:\n  seqkit duplicate [flags]\n\nAliases:\n  duplicate, dup\n\nFlags:\n  -h, --help        help for duplicate\n  -n, --times int   duplication number (default 1)\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Data</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 1\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> <li> <p>Duplicate 2 times</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 1 \\\n    | seqkit duplicate -n 2\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> <li> <p>use <code>seqkit rename</code> to make the the sequence IDs unique.</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 1 \\\n    | seqkit duplicate -n 2 | seqkit rename\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n&gt;cel-let-7_2 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> </ol>"},{"location":"usage/#rmdup","title":"rmdup","text":"<p>Usage</p> <pre><code>remove duplicated sequences by ID/name/sequence\n\nAttention:\n  1. When comparing by sequences, both positive and negative strands are\n     compared. Switch on -P/--only-positive-strand for considering the\n     positive strand only.\n  2. Only the first record is saved for duplicates.\n\nUsage:\n  seqkit rmdup [flags]\n\nFlags:\n  -n, --by-name                by full name instead of just id\n  -s, --by-seq                 by seq\n  -D, --dup-num-file string    file to save numbers and ID lists of duplicated seqs\n  -d, --dup-seqs-file string   file to save duplicated seqs\n  -h, --help                   help for rmdup\n  -i, --ignore-case            ignore case\n  -P, --only-positive-strand   only considering positive strand when comparing by sequence\n\n</code></pre> <p>Examples</p> <p>Similar to <code>common</code>.</p> <ol> <li> <p>General use</p> <pre><code>$ zcat hairpin.fa.gz | seqkit rmdup -s -o clean.fa.gz\n[INFO] 2226 duplicated records removed\n\n$ zcat reads_1.fq.gz | seqkit rmdup -s -o clean.fa.gz\n[INFO] 1086 duplicated records removed\n</code></pre> </li> <li> <p>Save duplicated sequences to file</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit rmdup -s -i -o clean.fa.gz -d duplicated.fa.gz -D duplicated.detail.txt\n\n$ cat duplicated.detail.txt   # here is not the entire list\n3   hsa-mir-424, mml-mir-424, ppy-mir-424\n3   hsa-mir-342, mml-mir-342, ppy-mir-342\n2   ngi-mir-932, nlo-mir-932\n2   ssc-mir-9784-1, ssc-mir-9784-2\n</code></pre> </li> </ol>"},{"location":"usage/#common","title":"common","text":"<p>Usage</p> <pre><code>find common sequences of multiple files by id/name/sequence\n\nNote:\n  1. 'seqkit common' is designed to support 2 and MORE files.\n  2. When comparing by sequences,\n     a) Both positive and negative strands are compared. You can switch on\n        -P/--only-positive-strand for considering the positive strand only.\n     b) You can switch on -e/--check-embedded-seqs to check embedded sequences.\n          e.g, for file A and B, the reverse complement sequence of CCCC from file B\n          is a part of TTGGGGTT from file A, we will extract and output GGGG from file A.\n          If sequences CCC exist in other files except file A, we will skip it,\n          as it is an embedded subsequence of GGGG.\n        It is recommended to put the smallest file as the first file, for saving\n        memory usage.\n  3. For 2 files, 'seqkit grep' is much faster and consumes lesser memory:\n       seqkit grep -f &lt;(seqkit seq -n -i small.fq.gz) big.fq.gz # by seq ID\n     But note that searching by sequence would be much slower, as it's partly\n     string matching.\n       seqkit grep -s -f &lt;(seqkit seq -s small.fq.gz) big.fq.gz # much slower!!!!\n  4. Some records in one file may have same sequences/IDs. They will ALL be\n     retrieved if the sequence/ID was shared in multiple files.\n     So the records number may be larger than that of the smallest file.\n\nUsage:\n  seqkit common [flags]\n\nFlags:\n  -n, --by-name                match by full name instead of just id\n  -s, --by-seq                 match by sequence\n  -e, --check-embedded-seqs    check embedded sequences, e.g., if a sequence is part of another one,\n                               we'll keep the shorter one\n  -h, --help                   help for common\n  -i, --ignore-case            ignore case\n  -P, --only-positive-strand   only considering the positive strand when comparing by sequence\n\n</code></pre> <p>Examples</p> <ol> <li> <p>By ID (default)</p> <pre><code>seqkit common file*.fa -o common.fasta\n</code></pre> </li> <li> <p>By full name</p> <pre><code>seqkit common file*.fa -n -o common.fasta\n</code></pre> </li> <li> <p>By sequence</p> <pre><code>seqkit common file*.fa -s -i -o common.fasta\n\n# seqkit common file*.fa -s -i -o common.fasta\nseqkit common file*.fa -s -i -o common.fasta -P\n</code></pre> </li> <li> <p>Since v2.6.0, we can use the new flag <code>-e/--check-embedded-seqs</code> to detect embedded sequences.</p> <pre><code>$ cat common_a.fasta\n&gt;A_a\nACTTTGA\n&gt;A_b\nACC\n&gt;A_c\nGGG\n&gt;A_d\nCCCCCCCCCCCCC\n&gt;A_c1\nAGGGCCC\n&gt;A_x\ngggg\n&gt;A_y\nACGTAC\n\n$ cat common_b.fasta\n&gt;B_a\nAAA\n&gt;B_b\nGGT\n&gt;B_c\nAGGGCC\n&gt;B_e\nACCCCCGGGAACC\n&gt;B_z\nTTT\n\n$ cat common_c.fasta\n&gt;C_a\nACTTTGA\n&gt;C_b\nACC\n&gt;C_c\nGGG\n&gt;C_d\nCCCCCCCCCCCCC\n&gt;C_c1\nAGGGCCC\n&gt;C_x\ngggg\n&gt;C_y\nACGTAC\n\n# excactly same sequences\n\n$ seqkit common -s -i common_a.fasta common_b.fasta common_c.fasta\n[INFO] read file 1/3: common_a.fasta\n[INFO] read file 2/3: common_b.fasta\n[INFO] read file 3/3: common_c.fasta\n[INFO] find common seqs ...\n[INFO] 1 unique sequences found in 3 files, which belong to 1 records in the first file: common_a.fasta\n[INFO] retrieve 1 seqs from the first file: common_a.fasta\n&gt;A_b\nACC\n\n# with -e\n\n$ seqkit common -s -i common_a.fasta common_b.fasta common_c.fasta -e\n[INFO] read file 1/3: common_a.fasta\n[INFO]   7 seqs loaded\n[INFO] read file 2/3: common_b.fasta\n[INFO]   5 seqs left\n[INFO] read file 3/3: common_c.fasta\n[INFO]   5 seqs left\n[INFO] 5 unique sequences found in 3 files, which belong to 5 records in the first file: common_a.fasta\n[INFO] 5 common/shared sequences saved to: -\n&gt;A_a:3-5\nTTT\n&gt;A_b\nACC\n&gt;A_c\nGGG\n&gt;A_c1:1-6\nAGGGCC\n&gt;A_x\ngggg\n\n# change the order of file, the sequences are the same\n\n$ seqkit common -s -i common_c.fasta common_b.fasta common_a.fasta -e\n[INFO] read file 1/3: common_c.fasta\n[INFO]   7 seqs loaded\n[INFO] read file 2/3: common_b.fasta\n[INFO]   5 seqs left\n[INFO] read file 3/3: common_a.fasta\n[INFO]   5 seqs left\n[INFO] 5 unique sequences found in 3 files, which belong to 5 records in the first file: common_c.fasta\n[INFO] 5 common/shared sequences saved to: -\n&gt;C_a:3-5\nTTT\n&gt;C_b\nACC\n&gt;C_c\nGGG\n&gt;C_c1:1-6\nAGGGCC\n&gt;C_x\ngggg\n</code></pre> </li> </ol>"},{"location":"usage/#split","title":"split","text":"<p>Usage</p> <pre><code>split sequences into files by name ID, subsequence of given region,\npart size or number of parts.\n\nIf you just want to split by parts or sizes, please use \"seqkit split2\",\nwhich can apply to paired- and single-end FASTQ.\n\nIf you want to cut a sequence into multiple segments.\n  1. For cutting into even chunks, please use 'kmcp utils split-genomes'\n     (https://bioinf.shenwei.me/kmcp/usage/#split-genomes).\n     E.g., cutting into 4 segments of equal size, with no overlap between adjacent segments:\n        kmcp utils split-genomes -m 1 -k 1 --split-number 4 --split-overlap 0 input.fasta -O out\n  2. For cutting into multiple chunks of fixed size, please using 'seqkit sliding'.\n     E.g., cutting into segments of 40 bp and keeping the last segment which can be shorter than 40 bp.\n        seqkit sliding -g -s 40 -W 40 input.fasta -o out.fasta\n\nAttention:\n  1. For the two-pass mode (-2/--two-pass), The flag -U/--update-faidx is recommended to\n     ensure the .fai file matches the FASTA file.\n  2. For splitting by sequence IDs in Windows/MacOS, where the file systems might be case-insensitive,\n     output files might be overwritten if they are only different in cases, like Abc and ABC.\n     To avoid this, please switch one -I/--ignore-case.\n\nThe definition of region is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit split [flags]\n\nFlags:\n  -i, --by-id                     split squences according to sequence ID\n      --by-id-prefix string       file prefix for --by-id\n  -p, --by-part int               split sequences into N parts\n      --by-part-prefix string     file prefix for --by-part\n  -r, --by-region string          split squences according to subsequence of given region. e.g 1:12 for\n                                  first 12 bases, -12:-1 for last 12 bases. type \"seqkit split -h\" for\n                                  more examples\n      --by-region-prefix string   file prefix for --by-region\n  -s, --by-size int               split sequences into multi parts with N sequences\n      --by-size-prefix string     file prefix for --by-size\n  -d, --dry-run                   dry run, just print message and no files will be created.\n  -e, --extension string          set output file extension, e.g., \".gz\", \".xz\", or \".zst\"\n  -f, --force                     overwrite output directory\n  -h, --help                      help for split\n      --ignore-case               ignore case when using -i/--by-id\n  -k, --keep-temp                 keep temporary FASTA and .fai file when using 2-pass mode\n  -O, --out-dir string            output directory (default value is $infile.split)\n  -2, --two-pass                  two-pass mode read files twice to lower memory usage. (only for FASTA\n                                  format)\n  -U, --update-faidx              update the fasta index file if it exists. Use this if you are not sure\n                                  whether the fasta file changed\n</code></pre> <p>Examples</p> <ol> <li> <p>Split sequences into parts with at most 10000 sequences</p> <pre><code>$ seqkit split hairpin.fa.gz -s 10000\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: hairpin.fa.part_001.gz\n[INFO] write 10000 sequences to file: hairpin.fa.part_002.gz\n[INFO] write 8645 sequences to file: hairpin.fa.part_003.gz\n</code></pre> </li> <li> <p>Split sequences into 4 parts</p> <pre><code>$ seqkit split hairpin.fa.gz -p 4\n[INFO] split into 4 parts\n[INFO] read sequences ...\n[INFO] read 28645 sequences\n[INFO] write 7162 sequences to file: hairpin.fa.part_001.gz\n[INFO] write 7162 sequences to file: hairpin.fa.part_002.gz\n[INFO] write 7162 sequences to file: hairpin.fa.part_003.gz\n[INFO] write 7159 sequences to file: hairpin.fa.part_004.gz\n</code></pre> <p>To reduce memory usage when spliting big file, we should alwasy use flag <code>--two-pass</code></p> <pre><code>$ seqkit split hairpin.fa.gz -p 4 -2\n[INFO] split into 4 parts\n[INFO] read and write sequences to temporary file: hairpin.fa.gz.fastx ...\n[INFO] 28645 sequences saved\n[INFO] create or read FASTA index ...\n[INFO] create FASTA index for hairpin.fa.gz.fastx\n[INFO]   28645 records loaded from hairpin.fa.gz.fastx.seqkit.fai\n[INFO] write 7162 sequences to file: hairpin.part_001.fa.gz\n[INFO] write 7162 sequences to file: hairpin.part_002.fa.gz\n[INFO] write 7162 sequences to file: hairpin.part_003.fa.gz\n[INFO] write 7159 sequences to file: hairpin.part_004.fa.gz\n</code></pre> </li> <li> <p>Split sequences by species. i.e. by custom IDs (first three letters)</p> <pre><code>$ seqkit split hairpin.fa.gz -i --id-regexp \"^([\\w]+)\\-\" -2\n[INFO] split by ID. idRegexp: ^([\\w]+)\\-\n[INFO] read and write sequences to temporary file: hairpin.fa.gz.fastx ...\n[INFO] 28645 sequences saved\n[INFO] create or read FASTA index ...\n[INFO] create FASTA index for hairpin.fa.gz.fastx\n[INFO]   28645 records loaded from hairpin.fa.gz.fastx.seqkit.fai\n[INFO] write 48 sequences to file: hairpin.id_cca.fa.gz\n[INFO] write 3 sequences to file: hairpin.id_hci.fa.gz\n[INFO] write 106 sequences to file: hairpin.id_str.fa.gz\n[INFO] write 1 sequences to file: hairpin.id_bkv.fa.gz\n...\n</code></pre> </li> <li> <p>Split sequences by sequence region (for example, sequence barcode)</p> <pre><code>$ seqkit split hairpin.fa.gz -r 1:3 -2\n[INFO] split by region: 1:3\n[INFO] read and write sequences to temporary file: hairpin.fa.gz.fastx ...\n[INFO] 28645 sequences saved\n[INFO] read sequence IDs and sequence region from FASTA file ...\n[INFO] create or read FASTA index ...\n[INFO] create FASTA index for hairpin.fa.gz.fastx\n[INFO]   28645 records loaded from hairpin.fa.gz.fastx.seqkit.fai\n[INFO] write 463 sequences to file: hairpin.region_1:3_AUG.fa.gz\n[INFO] write 349 sequences to file: hairpin.region_1:3_ACU.fa.gz\n[INFO] write 311 sequences to file: hairpin.region_1:3_CGG.fa.gz\n</code></pre> <p>Sequence suffix could be defined as <code>-r -12:-1</code></p> </li> </ol>"},{"location":"usage/#split2","title":"split2","text":"<p>Usage</p> <pre><code>split sequences into files by part size or number of parts\n\nThis command supports FASTA and paired- or single-end FASTQ with low memory\noccupation and fast speed.\n\nThe prefix of output files:\n  1. For stdin: stdin\n  2. Others: same to the input file\n  3. Set via the options: --by-length-prefix, --by-part-prefix, or --by-size-prefix\n\nThe extension of output files:\n  1. For stdin: .fast[aq]\n  2. Others: same to the input file\n  3. Additional extension via the option -e/--extension, e.g.\uff0c outputting\n     gzipped files for plain text input:\n         seqkit split2 -p 2 -O test tests/hairpin.fa -e .gz\n\n\nIf you want to cut a sequence into multiple segments.\n  1. For cutting into even chunks, please use 'kmcp utils split-genomes'\n     (https://bioinf.shenwei.me/kmcp/usage/#split-genomes).\n     E.g., cutting into 4 segments of equal size, with no overlap between adjacent segments:\n        kmcp utils split-genomes -m 1 -k 1 --split-number 4 --split-overlap 0 input.fasta -O out\n  2. For cutting into multiple chunks of fixed size, please using 'seqkit sliding'.\n     E.g., cutting into segments of 40 bp and keeping the last segment which can be shorter than 40 bp.\n        seqkit sliding -g -s 40 -W 40 input.fasta -o out.fasta\n\nUsage:\n  seqkit split2 [flags]\n\nFlags:\n  -l, --by-length string          split sequences into chunks of &gt;=N bases, supports K/M/G suffix\n      --by-length-prefix string   file prefix for --by-length\n  -p, --by-part int               split sequences into N parts\n      --by-part-prefix string     file prefix for --by-part\n  -s, --by-size int               split sequences into multi parts with N sequences\n      --by-size-prefix string     file prefix for --by-size\n  -e, --extension string          set output file extension, e.g., \".gz\", \".xz\", or \".zst\"\n  -f, --force                     overwrite output directory\n  -h, --help                      help for split2\n  -O, --out-dir string            output directory (default value is $infile.split)\n  -1, --read1 string              (gzipped) read1 file\n  -2, --read2 string              (gzipped) read2 file\n</code></pre> <p>Examples</p> <ol> <li> <p>Split sequences into parts with at most 10000 sequences:</p> <pre><code>$ seqkit split2 hairpin.fa -s 10000\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: hairpin.fa.split/hairpin.part_001.fa\n[INFO] write 10000 sequences to file: hairpin.fa.split/hairpin.part_002.fa\n[INFO] write 8645 sequences to file: hairpin.fa.split/hairpin.part_003.fa\n</code></pre> </li> <li> <p>Force compression for plain text input by adding an extra extension:</p> <pre><code># gzip\n$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .gz\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/hairpin.part_001.fa.gz\n[INFO] write 10000 sequences to file: test/hairpin.part_002.fa.gz\n[INFO] write 8645 sequences to file: test/hairpin.part_003.fa.gz\n\n# xz\n$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .xz\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/hairpin.part_001.fa.xz\n[INFO] write 10000 sequences to file: test/hairpin.part_002.fa.xz\n[INFO] write 8645 sequences to file: test/hairpin.part_003.fa.xz\n\n# zstd\n$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .zst\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/hairpin.part_001.fa.zst\n[INFO] write 10000 sequences to file: test/hairpin.part_002.fa.zst\n[INFO] write 8645 sequences to file: test/hairpin.part_003.fa.zst\n</code></pre> </li> <li> <p>Change the prefix of output files:</p> <pre><code>$ seqkit split2 hairpin.fa -O test -f -s 10000 -e .gz -o xxx\n[INFO] split seqs from hairpin.fa\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/xxx.part_001.fa.gz\n[INFO] write 10000 sequences to file: test/xxx.part_002.fa.gz\n[INFO] write 8645 sequences to file: test/xxx.part_003.fa.gz\n\n# here, we also change the compression format from xz to zstd\n$ cat hairpin.fa.xz | seqkit split2 -O test -f -s 10000 -e .zst\n[INFO] split seqs from stdin\n[INFO] split into 10000 seqs per file\n[INFO] write 10000 sequences to file: test/stdin.part_001.fasta.zst\n[INFO] write 10000 sequences to file: test/stdin.part_002.fasta.zst\n[INFO] write 8645 sequences to file: test/stdin.part_003.fasta.zst\n</code></pre> </li> <li> <p>Split sequences into 4 parts</p> <pre><code>$ seqkit split hairpin.fa.gz -p 4 -f\n[INFO] split into 4 parts\n[INFO] read sequences ...\n[INFO] read 28645 sequences\n[INFO] write 7162 sequences to file: hairpin.fa.gz.split/hairpin.part_001.fa.gz\n[INFO] write 7162 sequences to file: hairpin.fa.gz.split/hairpin.part_002.fa.gz\n[INFO] write 7162 sequences to file: hairpin.fa.gz.split/hairpin.part_003.fa.gz\n[INFO] write 7159 sequences to file: hairpin.fa.gz.split/hairpin.part_004.fa.gz\n</code></pre> </li> <li> <p>For FASTQ files (paired-end)</p> <pre><code>$ seqkit split2 -1 reads_1.fq.gz -2 reads_2.fq.gz -p 2 -O out -f\n[INFO] split seqs from reads_1.fq.gz and reads_2.fq.gz\n[INFO] split into 2 parts\n[INFO] write 1250 sequences to file: out/reads_2.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_2.part_002.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_002.fq.gz\n</code></pre> </li> <li> <p>For FASTA files (single-end)</p> <pre><code>$ seqkit split2 -1 reads_1.fq.gz reads_2.fq.gz -p 2 -O out -f\n[INFO] flag -1/--read1 given, ignore: reads_2.fq.gz\n[INFO] split seqs from reads_1.fq.gz\n[INFO] split into 2 parts\n[INFO] write 1250 sequences to file: out/reads_1.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_002.fq.gz\n\n$ seqkit split2 reads_1.fq.gz -p 2 -O out -f\n[INFO] split seqs from reads_1.fq.gz\n[INFO] split into 2 parts\n[INFO] write 1250 sequences to file: out/reads_1.part_001.fq.gz\n[INFO] write 1250 sequences to file: out/reads_1.part_002.fq.gz\n</code></pre> </li> </ol>"},{"location":"usage/#pair","title":"pair","text":"<p>Usage</p> <pre><code>match up paired-end reads from two fastq files\n\nAttention:\n1. Orders of headers in the two files should be the same (not shuffled),\n   otherwise, it consumes a huge number of memory for buffering reads in memory.\n2. Unpaired reads are optional outputted with the flag -u/--save-unpaired.\n3. If the flag -O/--out-dir is not given, the output will be saved in the same directory\n   of input, with the suffix \"paired\", e.g., read_1.paired.fq.gz.\n   Otherwise, names are kept untouched in the given output directory.\n4. Paired gzipped files may be slightly larger than original files, because\n   of using a different gzip package/library, don't worry.\n\nTips:\n1. Support for '/1 'and '/2' tags for paired read files generated by platforms like MGI.\n   You can simply specify the regular expression for extracting sequence IDs:\n     --id-regexp '^(\\S+)\\/[12]'\n\nUsage:\n  seqkit pair [flags]\n\nFlags:\n  -f, --force            overwrite output directory\n  -h, --help             help for pair\n  -O, --out-dir string   output directory\n  -1, --read1 string     (gzipped) read1 file\n  -2, --read2 string     (gzipped) read2 file\n  -u, --save-unpaired    save unpaired reads if there are\n</code></pre> <p>Examples</p> <ol> <li> <p>Simple one</p> <pre><code>$ seqkit pair -1 reads_1.fq.gz -2 reads_2.fq.gz\n\n# output\nreads_1.paired.fq.gz\nreads_2.paired.fq.gz\n</code></pre> </li> <li> <p>Set output directory, file names are kept untouched.</p> <pre><code>$ seqkit pair -1 reads_1.fq.gz -2 reads_2.fq.gz -O result\n\n$ tree result        \nresult/\n\u251c\u2500\u2500 reads_1.fq.gz\n\u2514\u2500\u2500 reads_2.fq.gz\n</code></pre> </li> <li> <p>Save unpaired reads if there are.</p> <pre><code>$ seqkit pair -1 reads_1.fq.gz -2 reads_2.fq.gz -O result -u\n\n$ tree result\nresult\n\u251c\u2500\u2500 reads_1.fq.gz\n\u251c\u2500\u2500 reads_1.unpaired.fq.gz\n\u251c\u2500\u2500 reads_2.fq.gz\n\u2514\u2500\u2500 reads_2.unpaired.fq.gz\n</code></pre> </li> <li> <p>Support for '/1 'and '/2' tags for paired read files generated by platforms like MGI.    You can simply specify the regular expression for extracting sequence IDs:</p> <pre><code>seqkit pair --id-regexp '^(\\S+)\\/[12]' -1 reads_1.fq.gz -2 reads_2.fq.gz\n</code></pre> </li> </ol>"},{"location":"usage/#sample","title":"sample","text":"<p>Usage</p> <pre><code>sample sequences by number or proportion.\n\nAttention:\n1. Do not use '-n' on large FASTQ files, it loads all seqs into memory!\n   use 'seqkit sample -p 0.1 seqs.fq.gz | seqkit head -n N' instead!\n\nUsage:\n  seqkit sample [flags]\n\nFlags:\n  -h, --help               help for sample\n  -n, --number int         sample by number (result may not exactly match), DO NOT use on large FASTQ files.\n  -p, --proportion float   sample by proportion\n  -s, --rand-seed int      random seed. For paired-end data, use the same seed across fastq files to\n                           sample the same read pairs (default 11)\n  -2, --two-pass           2-pass mode read files twice to lower memory usage. Not allowed when reading\n                           from stdin\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Sample by proportion</p> <pre><code>$ zcat hairpin.fa.gz | seqkit sample -p 0.1 -o sample.fa.gz\n[INFO] sample by proportion\n[INFO] 2814 sequences outputed\n</code></pre> </li> <li> <p>Sample by number</p> <pre><code>$ zcat hairpin.fa.gz | seqkit sample -n 1000 -o sample.fa.gz\n[INFO] sample by number\n[INFO] 949 sequences outputed\n</code></pre> <p>949 != 1000 ??? see Effect of random seed on results of <code>seqkit sample</code></p> <p>To reduce memory usage when spliting big file, we could use flag <code>--two-pass</code></p> <p>We can also use <code>seqkit sample -p</code> followed with <code>seqkit head -n</code>:</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sample -p 0.1 \\\n    | seqkit head -n 1000 -o sample.fa.gz\n</code></pre> </li> <li> <p>Set rand seed to reproduce the result</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sample -p 0.1 -s 11\n</code></pre> </li> <li> <p>Most of the time, we could shuffle after sampling</p> <pre><code>$ zcat hairpin.fa.gz \\\n    | seqkit sample -p 0.1 \\\n    | seqkit shuffle -o sample.fa.gz\n</code></pre> </li> </ol> <p>Note that when sampling on FASTQ files, make sure using same random seed by flag <code>-s</code> (<code>--rand-seed</code>)</p>"},{"location":"usage/#head","title":"head","text":"<p>Usage</p> <pre><code>print first N FASTA/Q records\n\nFor returning the last N records, use:\n    seqkit range -r -N:-1 seqs.fasta\n\nUsage:\n  seqkit head [flags]\n\nFlags:\n  -n, --number int   print first N FASTA/Q records (default 10)\n\n</code></pre> <p>Examples</p> <ol> <li> <p>FASTA</p> <pre><code>$ seqkit head -n 1 hairpin.fa.gz\n&gt;cel-let-7 MI0000001 Caenorhabditis elegans let-7 stem-loop\nUACACUGUGGAUCCGGUGAGGUAGUAGGUUGUAUAGUUUGGAAUAUUACCACCGGUGAAC\nUAUGCAAUUUUCUACCUUACCGGAGACAGAACUCUUCGA\n</code></pre> </li> <li> <p>FASTQ</p> <pre><code>$ seqkit head -n 1 reads_1.fq.gz\n@HWI-D00523:240:HF3WGBCXX:1:1101:2574:2226 1:N:0:CTGTAG\nTGAGGAATATTGGTCAATGGGCGCGAGCCTGAACCAGCCAAGTAGCGTGAAGGATGACTGCCCTACGGGTTGTAA\n+\nHIHIIIIIHIIHGHHIHHIIIIIIIIIIIIIIIHHIIIIIHHIHIIIIIGIHIIIIHHHHHHGHIHIIIIIIIII\n</code></pre> </li> </ol>"},{"location":"usage/#head-genome","title":"head-genome","text":"<p>Usage</p> <pre><code>\nprint sequences of the first genome with common prefixes in name\n\nFor a FASTA file containing multiple contigs of strains (see example below),\nthese's no list of IDs available for retrieving sequences of a certain strain,\nwhile descriptions of each strain share the same prefix.\n\nThis command is used to restrieve sequences of the first strain,\ni.e., \"Vibrio cholerae strain M29\".\n\n&gt;NZ_JFGR01000001.1 Vibrio cholerae strain M29 Contig_1, whole genome shotgun sequence\n&gt;NZ_JFGR01000002.1 Vibrio cholerae strain M29 Contig_2, whole genome shotgun sequence\n&gt;NZ_JFGR01000003.1 Vibrio cholerae strain M29 Contig_3, whole genome shotgun sequence\n&gt;NZ_JSTP01000001.1 Vibrio cholerae strain 2012HC-12 NODE_79, whole genome shotgun sequence\n&gt;NZ_JSTP01000002.1 Vibrio cholerae strain 2012HC-12 NODE_78, whole genome shotgun sequence\n\nAttention:\n\n  1. Sequences in file should be well organized.\n\nUsage:\n  seqkit head-genome [flags]\n\nFlags:\n  -h, --help                    help for head-genome\n  -m, --mini-common-words int   minimal shared prefix words (default 4)\n\n</code></pre>"},{"location":"usage/#range","title":"range","text":"<p>Usage</p> <pre><code>print FASTA/Q records in a range (start:end)\n\nExamples:\n  1. leading 100 records (head -n 100)\n      seqkit range -r 1:100\n  2. last 100 records (tail -n 100)\n      seqkit range -r -100:-1\n  3. remove leading 100 records (tail -n +101)\n      seqkit range -r 101:-1\n  4. other ranges:\n      seqkit range -r 10:100\n      seqkit range -r -100:-10\n\nUsage:\n  seqkit range [flags]\n\nFlags:\n  -h, --help           help for range\n  -r, --range string   range. e.g., 1:12 for first 12 records (head -n 12), -12:-1 for last 12 records (tail -n 12)\n</code></pre> <p>Examples</p> <ol> <li> <p>leading N records (head)</p> <pre><code>$ cat tests/hairpin.fa | seqkit head -n 100 | md5sum\nf65116af7d9298d93ba4b3d19077bbf1  -\n$ cat tests/hairpin.fa | seqkit range -r 1:100 | md5sum\nf65116af7d9298d93ba4b3d19077bbf1  -\n</code></pre> </li> <li> <p>last N records (tail)</p> <pre><code>$ cat tests/hairpin.fa | seqkit range -r -100:-1 | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA        100    8,656       58     86.6      172\n</code></pre> </li> <li> <p>remove leading 100 records (tail -n +101)</p> <pre><code>$ seqkit range -r 101:-1 tests/hairpin.fa | seqkit sum -a\nseqkit.v0.1_RLS_k0_e1feced9bb0be653afa8205dec4987db     -       28545   2940929\n\n$ seqkit fx2tab tests/hairpin.fa | tail -n +101 | seqkit tab2fx | seqkit sum -a\nseqkit.v0.1_RLS_k0_e1feced9bb0be653afa8205dec4987db     -       28545   2940929\n</code></pre> </li> <li> <p>Other ranges</p> <pre><code>$ cat tests/hairpin.fa | seqkit range -r 101:150 | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA         50    3,777       63     75.5       96\n\n$ cat tests/hairpin.fa | seqkit range -r -100:-2 | seqkit stats\nfile  format  type  num_seqs  sum_len  min_len  avg_len  max_len\n-     FASTA   RNA         99    8,484       58     85.7      146\n</code></pre> </li> </ol>"},{"location":"usage/#replace","title":"replace","text":"<p>Usage</p> <pre><code>replace name/sequence by regular expression.\n\nNote that the replacement supports capture variables.\ne.g. $1 represents the text of the first submatch.\nATTENTION: use SINGLE quote NOT double quotes in *nix OS.\n\nExamples: Adding space to all bases.\n\n    seqkit replace -p \"(.)\" -r '$1 ' -s\n\nOr use the \\ escape character.\n\n    seqkit replace -p \"(.)\" -r \"\\$1 \" -s\n\nmore on: http://bioinf.shenwei.me/seqkit/usage/#replace\n\nSpecial replacement symbols (only for replacing name not sequence):\n\n    {fn}    File name\n    {fbn}   File base name\n    {fbne}  File base name without any extension\n    {kv}    Corresponding value of the key (captured variable $n) by key-value file,\n            n can be specified by flag -I (--key-capt-idx) (default: 1)\n\nSpecial cases:\n  1. If replacements contain '$', \n    a). If using '{kv}', you need use '$$$$' instead of a single '$':\n            -r '{kv}' -k &lt;(sed 's/\\$/$$$$/' kv.txt)\n    b). If not, use '$$':\n            -r 'xxx$$xx'\n\nFiltering records to edit:\n  You can use flags similar to those in \"seqkit grep\" to choose partly records to edit.\n\nUsage:\n  seqkit replace [flags]\n\nFlags:\n  -s, --by-seq                   replace seq (only FASTA)\n      --f-by-name                [target filter] match by full name instead of just ID\n      --f-by-seq                 [target filter] search subseq on seq, both positive and negative strand\n                                 are searched, and mismatch allowed using flag -m/--max-mismatch\n      --f-ignore-case            [target filter] ignore case\n      --f-invert-match           [target filter] invert the sense of matching, to select non-matching records\n      --f-only-positive-strand   [target filter] only search on positive strand\n      --f-pattern strings        [target filter] search pattern (multiple values supported. Attention:\n                                 use double quotation marks for patterns containing comma, e.g., -p\n                                 '\"A{2,}\"')\n      --f-pattern-file string    [target filter] pattern file (one record per line)\n      --f-use-regexp             [target filter] patterns are regular expression\n  -h, --help                     help for replace\n  -i, --ignore-case              ignore case\n  -K, --keep-key                 keep the key as value when no value found for the key (only for\n                                 sequence name)\n  -U, --keep-untouch             do not change anything when no value found for the key (only for\n                                 sequence name)\n  -I, --key-capt-idx int         capture variable index of key (1-based) (default 1)\n  -m, --key-miss-repl string     replacement for key with no corresponding value\n  -k, --kv-file string           tab-delimited key-value file for replacing key with value when using\n                                 \"{kv}\" in -r (--replacement) (only for sequence name)\n      --nr-width int             minimum width for {nr} in flag -r/--replacement. e.g., formatting \"1\"\n                                 to \"001\" by --nr-width 3 (default 1)\n  -p, --pattern string           search regular expression\n  -r, --replacement string       replacement. supporting capture variables.  e.g. $1 represents the text\n                                 of the first submatch. ATTENTION: for *nix OS, use SINGLE quote NOT\n                                 double quotes or use the \\ escape character. Record number is also\n                                 supported by \"{nr}\".use ${1} instead of $1 when {kv} given!\n\n</code></pre> <p>Examples</p> <ol> <li> <p>Prepend or append to the header</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\"\n&gt;seq1 abc-123\nACGT-ACGT\n\n# prefix\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p ^ -r _prefix_\n&gt;_prefix_seq1 abc-123\nACGT-ACGT\n\n# suffix\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p $ -r _suffix_\n&gt;seq1 abc-123_suffix_\nACGT-ACGT\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit seq -i \\\n    | seqkit replace -p $ -r _suffix_\n&gt;seq1_suffix_\nACGT-ACGT\n</code></pre> </li> <li> <p>Remove descriptions</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\"\n&gt;seq1 abc-123\nACGT-ACGT\n\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"\\s.+\"\n&gt;seq1\nACGT-ACGT\n</code></pre> </li> <li> <p>Replace \"-\" with \"=\"</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"\\-\" -r '='\n&gt;seq1 abc=123\nACGT-ACGT\n</code></pre> </li> <li> <p>Remove gaps in sequences.</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \" |-\" -s\n&gt;seq1 abc-123\nACGTACGT\n</code></pre> </li> <li> <p>Add space to every base. ATTENTION: use SINGLE quote NOT double quotes in *nix OS</p> <pre><code>$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"(.)\" -r '$1 ' -s\n&gt;seq1 abc-123\nA C G T - A C G T\n\n$ echo -e \"&gt;seq1 abc-123\\nACGT-ACGT\" \\\n    | seqkit replace -p \"(.)\" -r \"\\$1 \" -s\n&gt;seq1 abc-123\nA C G T - A C G T\n</code></pre> </li> <li> <p>Transpose sequence with csvtk</p> <pre><code>$ echo -e \"&gt;seq1\\nACTGACGT\\n&gt;seq2\\nactgccgt\" \\\n    | seqkit replace -p \"(.)\" -r     \"\\$1 \" -s \\\n    | seqkit seq -s -u \\\n    | csvtk space2tab \\\n    | csvtk -t transpose\nA       A\nC       C\nT       T\nG       G\nA       C\nC       C\nG       G\nT       T\n</code></pre> </li> <li> <p>Rename with number of record</p> <pre><code>$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    |  seqkit replace -p .+ -r \"seq_{nr}\"\n&gt;seq_1\nACTG\n&gt;seq_2\nATTT\n\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    |  seqkit replace -p .+ -r \"seq_{nr}\" --nr-width 5\n&gt;seq_00001\nACTG\n&gt;seq_00002\nATTT\n</code></pre> </li> <li> <p>Add file names.</p> <pre><code>$ seqkit replace ../tests/hairpin.fa -p '.+' -r '{fn}__{fbn}__{fbne}__{nr}' | seqkit seq -n | head -n 3\n../tests/hairpin.fa__hairpin.fa__hairpin__1\n../tests/hairpin.fa__hairpin.fa__hairpin__2\n../tests/hairpin.fa__hairpin.fa__hairpin__3\n</code></pre> </li> <li> <p>Replace key with value by key-value file</p> <pre><code>$ more test.fa\n&gt;seq1 name1\nCCCCAAAACCCCATGATCATGGATC\n&gt;seq2 name2\nCCCCAAAACCCCATGGCATCATTCA\n&gt;seq3 name3\nCCCCAAAACCCCATGTTGCTACTAG\n\n$ more alias.txt\nname0   ABC\nname1   123\nname3   Hello\nname4   World\n\n$ seqkit replace -p ' (.+)$' -r ' {kv}' -k alias.txt test.fa\n[INFO] read key-value file: alias.txt\n[INFO] 4 pairs of key-value loaded\n&gt;seq1 123\nCCCCAAAACCCCATGATCATGGATC\n&gt;seq2\nCCCCAAAACCCCATGGCATCATTCA\n&gt;seq3 Hello\nCCCCAAAACCCCATGTTGCTACTAG\n\n$ seqkit replace -p ' (.+)$' -r ' {kv}' -k alias.txt test.fa --keep-key\n[INFO] read key-value file: alias.txt\n[INFO] 4 pairs of key-value loaded\n&gt;seq1 123\nCCCCAAAACCCCATGATCATGGATC\n&gt;seq2 name2\nCCCCAAAACCCCATGGCATCATTCA\n&gt;seq3 Hello\nCCCCAAAACCCCATGTTGCTACTAG\n</code></pre> <p>If the target to replace is the sequence ID, meanwhile you would like to keep the description.  Please use this:</p> <pre><code># ^(\\S+)   # ID\n# (.+)$    # description\nseqkit replace -p '^(\\S+)(.+?)$' -r '{kv}$2' -k alias.txt test.fa\n</code></pre> </li> <li> <p>convert fasta to genbank style</p> <pre><code>$ cat seq.fa\n&gt;seq1\nTTTAAAGAGACCGGCGATTCTAGTGAAATCGAACGGGCAGGTCAATTTCCAACCAGCGAT\nGACGTAATAGATAGATACAAGGAAGTCATTTTTCTTTTAAAGGATAGAAACGGTTAATGC\nTCTTGGGACGGCGCTTTTCTGTGCATAACT\n&gt;seq2\nAAGGATAGAAACGGTTAATGCTCTTGGGACGGCGCTTTTCTGTGCATAACTCGATGAAGC\nCCAGCAATTGCGTGTTTCTCCGGCAGGCAAAAGGTTGTCGAGAACCGGTGTCGAGGCTGT\nTTCCTTCCTGAGCGAAGCCTGGGGATGAACG\n\n$ cat seq.fa \\\n    | seqkit replace -s -p '(\\w{10})' -r '$1 ' -w 66 \\\n    | perl -ne 'if (/^&gt;/) {print; $n=1} else {s/ \\r?\\n$/\\n/; printf \"%9d %s\", $n, $_; $n+=60;}'\n&gt;seq1\n        1 TTTAAAGAGA CCGGCGATTC TAGTGAAATC GAACGGGCAG GTCAATTTCC AACCAGCGAT\n       61 GACGTAATAG ATAGATACAA GGAAGTCATT TTTCTTTTAA AGGATAGAAA CGGTTAATGC\n      121 TCTTGGGACG GCGCTTTTCT GTGCATAACT\n&gt;seq2\n        1 AAGGATAGAA ACGGTTAATG CTCTTGGGAC GGCGCTTTTC TGTGCATAAC TCGATGAAGC\n       61 CCAGCAATTG CGTGTTTCTC CGGCAGGCAA AAGGTTGTCG AGAACCGGTG TCGAGGCTGT\n      121 TTCCTTCCTG AGCGAAGCCT GGGGATGAAC G\n</code></pre> </li> <li> <p>only edit some of the records via patterns. Eight flags starting with <code>--f-</code> (with the same usage to these in <code>seqkit grep</code>) can be used    to filter records to edit.</p> <pre><code># prepend some text to header\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    | seqkit replace -p ^ -r _\n&gt;_abc\nACTG\n&gt;_123\nATTT\n\n# only edit some records\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    | seqkit replace -p ^ -r _ --f-use-regexp --f-pattern abc\n[INFO] 1 records matched by the filter\n&gt;_abc\nACTG\n&gt;123\nATTT\n\n$ echo -e \"&gt;abc\\nACTG\\n&gt;123\\nATTT\" \\\n    | seqkit replace -p ^ -r _ --f-use-regexp --f-invert-match --f-pattern abc \n[INFO] 1 records matched by the filter\n&gt;abc\nACTG\n&gt;_123\nATTT\n</code></pre> </li> </ol>"},{"location":"usage/#rename","title":"rename","text":"<p>Usage</p> <pre><code>rename duplicated IDs\n\nAttention:\n  1. This command only appends \"_N\" to duplicated sequence IDs to make them unique.\n  2. Use \"seqkit replace\" for editing sequence IDs/headers using regular expression.\n\nExample:\n\n    $ seqkit seq seqs.fasta \n    &gt;id comment\n    actg\n    &gt;id description\n    ACTG\n\n    $ seqkit rename seqs.fasta\n    &gt;id comment\n    actg\n    &gt;id_2 description\n    ACTG\n\nUsage:\n  seqkit rename [flags]\n\nFlags:\n  -n, --by-name             check duplication by full name instead of just id\n  -f, --force               overwrite output directory\n  -h, --help                help for rename\n  -m, --multiple-outfiles   write results into separated files for multiple input files\n  -O, --out-dir string      output directory (default \"renamed\")\n  -1, --rename-1st-rec      rename the first record as well\n  -s, --separator string    separator between original ID/name and the counter (default \"_\")\n  -N, --start-num int       starting count number for *duplicated* IDs/names, should be greater than\n                            zero (default 2\n\n</code></pre> <p>Examples</p> <pre><code>$ echo -e \"&gt;a comment\\nacgt\\n&gt;b comment of b\\nACTG\\n&gt;a comment\\naaaa\"\n&gt;a comment\nacgt\n&gt;b comment of b\nACTG\n&gt;a comment\naaaa\n\n$ echo -e \"&gt;a comment\\nacgt\\n&gt;b comment of b\\nACTG\\n&gt;a comment\\naaaa\" \\\n    | seqkit rename\n&gt;a comment\nacgt\n&gt;b comment of b\nACTG\n&gt;a_2 comment\naaaa\n\n$ echo -e \"&gt;a comment\\nacgt\\n&gt;b comment of b\\nACTG\\n&gt;a comment\\naaaa\"  \\\n    | seqkit rename -s '|' -N 2 -1\n&gt;a|1 comment\nacgt\n&gt;b|1 comment of b\nACTG\n&gt;a|2 comment\naaaa\n\n</code></pre>"},{"location":"usage/#restart","title":"restart","text":"<p>Usage</p> <pre><code>reset start position for circular genome\n\nExamples\n\n    $ echo -e \"&gt;seq\\nacgtnACGTN\"\n    &gt;seq\n    acgtnACGTN\n\n    $ echo -e \"&gt;seq\\nacgtnACGTN\" | seqkit restart -i 2\n    &gt;seq\n    cgtnACGTNa\n\n    $ echo -e \"&gt;seq\\nacgtnACGTN\" | seqkit restart -i -2\n    &gt;seq\n    TNacgtnACG\n\nUsage:\n  seqkit restart [flags]\n\nFlags:\n  -i, --new-start int   new start position (1-base, supporting negative value counting from the end) (default 1)\n\n</code></pre>"},{"location":"usage/#concat","title":"concat","text":"<p>Usage</p> <pre><code>concatenate sequences with same ID from multiple files\n\nAttention:\n   1. By default, only sequences with IDs that appear in all files are outputted.\n      use -f/--full to output all sequences.\n   2. If there are more than one sequences of the same ID, we output the Cartesian\n      product of sequences.\n   3. Description are also concatenated with a separator (-s/--separator).\n   4. Order of sequences with different IDs are random.\n\nUsage:\n  seqkit concat [flags]\n\nAliases:\n  concat, concate\n\nFlags:\n  -f, --full               keep all sequences, like full/outer join\n  -h, --help               help for concat\n  -s, --separator string   separator for descriptions of records with the same ID (default \"|\")\n\n</code></pre> <p>Examples</p> <pre><code>$ cat a.fa \n&gt;A 1\na1-\n&gt;A 2\na2-\n&gt;B 1\nb1-\n\n$ cat b.fa \n&gt;A x\nax-\n&gt;C 1\nc1-\n\n$ seqkit concat a.fa b.fa \n&gt;A 1|x\na1-ax-\n&gt;A 2|x\na2-ax-\n\n$ seqkit concat a.fa b.fa --full\n&gt;C 1\nc1-\n&gt;A 1|x\na1-ax-\n&gt;A 2|x\na2-ax-\n&gt;B 1\nb1-\n\n</code></pre>"},{"location":"usage/#mutate","title":"mutate","text":"<p>Usage</p> <pre><code>edit sequence (point mutation, insertion, deletion)\n\nAttention:\n\n  1. Mutiple point mutations (-p/--point) are allowed, but only single \n     insertion (-i/--insertion) OR single deletion (-d/--deletion) is allowed.\n  2. Point mutation takes place before insertion/deletion.\n\nNotes:\n\n  1. You can choose certain sequences to edit using similar flags in\n     'seqkit grep'.\n\nThe definition of position is 1-based and with some custom design.\n\nExamples:\n\n 1-based index    1 2 3 4 5 6 7 8 9 10\nnegative index    0-9-8-7-6-5-4-3-2-1\n           seq    A C G T N a c g t n\n           1:1    A\n           2:4      C G T\n         -4:-2                c g t\n         -4:-1                c g t n\n         -1:-1                      n\n          2:-2      C G T N a c g t\n          1:-1    A C G T N a c g t n\n          1:12    A C G T N a c g t n\n        -12:-1    A C G T N a c g t n\n\nUsage:\n  seqkit mutate [flags]\n\nFlags:\n  -n, --by-name               [match seqs to mutate] match by full name instead of just id\n  -d, --deletion string       deletion mutation: deleting subsequence in a range. e.g., -d 1:2 for\n                              deleting leading two bases, -d -3:-1 for removing last 3 bases\n  -h, --help                  help for mutate\n  -I, --ignore-case           [match seqs to mutate] ignore case of search pattern\n  -i, --insertion string      insertion mutation: inserting bases behind of given position, e.g., -i\n                              0:ACGT for inserting ACGT at the beginning, -1:* for add * to the end\n  -v, --invert-match          [match seqs to mutate] invert the sense of matching, to select\n                              non-matching records\n  -s, --pattern strings       [match seqs to mutate] search pattern (multiple values supported.\n                              Attention: use double quotation marks for patterns containing comma, e.g.,\n                              -p '\"A{2,}\"'))\n  -f, --pattern-file string   [match seqs to mutate] pattern file (one record per line)\n  -p, --point strings         point mutation: changing base at given position. e.g., -p 2:C for setting\n                              2nd base as C, -p -1:A for change last base as A\n  -r, --use-regexp            [match seqs to mutate] search patterns are regular expression\n\n</code></pre> <p>Examples:</p> <ol> <li> <p>Point mutation:</p> <pre><code>$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\"\n&gt;1\nACTGNactgn\n&gt;2\nactgnACTGN\n\n# first base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p 1:x\n[INFO] edit seq: 1\n[INFO] edit seq: 2\n&gt;1\nxCTGNactgn\n&gt;2\nxctgnACTGN\n\n# 5th base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p 5:x --quiet\n&gt;1\nACTGxactgn\n&gt;2\nactgxACTGN\n\n# last base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p -1:x --quiet\n&gt;1\nACTGNactgx\n&gt;2\nactgnACTGx\n\n# mutiple locations:\n\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -p 1:x -p -1:x --quiet\n&gt;1\nxCTGNactgx\n&gt;2\nxctgnACTGx\n</code></pre> </li> <li> <p>Deletion</p> <pre><code># first base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -d 1:1 --quiet\n&gt;1\nCTGNactgn\n&gt;2\nctgnACTGN\n\n# last 3 bases\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -d -3:-1 --quiet\n&gt;1\nACTGNac\n&gt;2\nactgnAC\n</code></pre> </li> <li> <p>Insertion: inserting bases behind of given position</p> <pre><code># at the beginning\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -i 0:xx --quiet\n&gt;1\nxxACTGNactgn\n&gt;2\nxxactgnACTGN\n\n# at the end\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -i -1:xx --quiet\n&gt;1\nACTGNactgnxx\n&gt;2\nactgnACTGNxx\n\n# behind of 5th base\n$ echo -ne \"&gt;1\\nACTGNactgn\\n&gt;2\\nactgnACTGN\\n\" \\\n    | seqkit mutate -i 5:x --quiet\n&gt;1\nACTGNxactgn\n&gt;2\nactgnxACTGN\n</code></pre> </li> <li> <p>Choosing which sequences to edit, using similar flags in <code>seqkit grep</code>.</p> <pre><code>$ cat tests/hsa.fa\n&gt;chr1 1th seq\nACTGNactgn\n&gt;chr2 2nd seq\nactgnACTGN\n&gt;chr11 11th seq\nACTGNACTGN\n&gt;MT mitochondrial seq\nactgnactgn\n\n# only edit chr1 and chr2\n# or cat tests/hsa.fa | seqkit mutate -p -1:X -s chr1 -s chr2\n$ cat tests/hsa.fa \\\n    | seqkit mutate -p -1:X -s chr1,chr2\n[INFO] edit seq: chr1 1th seq\n[INFO] edit seq: chr2 2nd seq\n&gt;chr1 1th seq\nACTGNactgX\n&gt;chr2 2nd seq\nactgnACTGX\n&gt;chr11 11th seq\nACTGNACTGN\n&gt;MT mitochondrial seq\nactgnactgn\n\n# using regular expression to match.\n# e,g., editing all chrosomes:\n$ cat tests/hsa.fa \\\n    | seqkit mutate -p -1:X -r -s chr\n[INFO] edit seq: chr1 1th seq\n[INFO] edit seq: chr2 2nd seq\n[INFO] edit seq: chr11 11th seq\n&gt;chr1 1th seq\nACTGNactgX\n&gt;chr2 2nd seq\nactgnACTGX\n&gt;chr11 11th seq\nACTGNACTGX\n&gt;MT mitochondrial seq\nactgnactgn\n\n# excluding seqs\n$ cat tests/hsa.fa \\\n    | seqkit mutate -p -1:X -s chr1 -s chr2 -v \n[INFO] edit seq: chr11 11th seq\n[INFO] edit seq: MT mitochondrial seq\n&gt;chr1 1th seq\nACTGNactgn\n&gt;chr2 2nd seq\nactgnACTGN\n&gt;chr11 11th seq\nACTGNACTGX\n&gt;MT mitochondrial seq\nactgnactgX\n</code></pre> </li> </ol>"},{"location":"usage/#shuffle","title":"shuffle","text":"<p>Usage</p> <pre><code>shuffle sequences.\n\nBy default, all records will be readed into memory.\nFor FASTA format, use flag -2 (--two-pass) to reduce memory usage. FASTQ not\nsupported.\n\nFirstly, seqkit reads the sequence IDs. If the file is not plain FASTA file,\nseqkit will write the sequences to temporary files, and create FASTA index.\n\nSecondly, seqkit shuffles sequence IDs and extract sequences by FASTA index.\n\nAttention:\n  1. For the two-pass mode (-2/--two-pass), The flag -U/--update-faidx is recommended to\n     ensure the .fai file matches the FASTA file.\n\nUsage:\n  seqkit shuffle [flags]\n\nFlags:\n  -h, --help            help for shuffle\n  -k, --keep-temp       keep temporary FASTA and .fai file when using 2-pass mode\n  -s, --rand-seed int   rand seed for shuffle (default 23)\n  -2, --two-pass        two-pass mode read files twice to lower memory usage. (only for FASTA format)\n  -U, --update-faidx    update the fasta index file if it exists. Use this if you are not sure whether\n                        the fasta file changed\n\n</code></pre> <p>Examples</p> <ol> <li> <p>General use.</p> <pre><code>$ seqkit shuffle hairpin.fa.gz &gt; shuffled.fa\n[INFO] read sequences ...\n[INFO] 28645 sequences loaded\n[INFO] shuffle ...\n[INFO] output ...\n</code></pre> </li> <li> <p>For big genome, you'd better use two-pass mode so seqkit could use    FASTA index to reduce memory usage</p> <pre><code>$ time seqkit shuffle -2 hsa.fa &gt; shuffle.fa\n[INFO] create and read FASTA index ...\n[INFO] create FASTA index for hsa.fa\n[INFO] read sequence IDs from FASTA index ...\n[INFO] 194 sequences loaded\n[INFO] shuffle ...\n[INFO] output ...\n\nreal    0m35.080s\nuser    0m45.521s\nsys     0m3.411s\n</code></pre> </li> </ol> <p>Note that when sampling on FASTQ files, make sure using same random seed by flag <code>-s</code> (<code>--rand-seed</code>) for read 1 and 2 files.</p>"},{"location":"usage/#sort","title":"sort","text":"<p>Usage</p> <pre><code>sort sequences by id/name/sequence/length.\n\nBy default, all records will be readed into memory.\nFor FASTA format, use flag -2 (--two-pass) to reduce memory usage. FASTQ not\nsupported.\n\nFirstly, seqkit reads the sequence head and length information.\nIf the file is not plain FASTA file,\nseqkit will write the sequences to temporary files, and create FASTA index.\n\nSecondly, seqkit sorts sequence by head and length information\nand extracts sequences by FASTA index.\n\nAttention:\n  1. For the two-pass mode (-2/--two-pass), The flag -U/--update-faidx is recommended to\n     ensure the .fai file matches the FASTA file.\n\nUsage:\n  seqkit sort [flags]\n\nFlags:\n  -b, --by-bases                by non-gap bases\n  -l, --by-length               by sequence length\n  -n, --by-name                 by full name instead of just id\n  -s, --by-seq                  by sequence\n  -G, --gap-letters string      gap letters (default \"- \\t.\")\n  -h, --help                    help for sort\n  -i, --ignore-case             ignore case\n  -k, --keep-temp               keep temporary FASTA and .fai file when using 2-pass mode\n  -N, --natural-order           sort in natural order, when sorting by IDs/full name\n  -r, --reverse                 reverse the result\n  -L, --seq-prefix-length int   length of sequence prefix on which seqkit sorts by sequences (0 for\n                                whole sequence) (default 10000)\n  -2, --two-pass                two-pass mode read files twice to lower memory usage. (only for FASTA format)\n  -U, --update-faidx            update the fasta index file if it exists. Use this if you are not sure\n                                whether the fasta file changed\n\n</code></pre> <p>Examples</p> <p>For FASTA format, use flag -2 (--two-pass) to reduce memory usage</p> <ol> <li> <p>sort by ID</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAA\" \\\n    | seqkit sort --quiet\n&gt;SEQ2\nacgtnAAAA\n&gt;seq1\nACGTNcccc\n</code></pre> </li> <li> <p>sort by ID and in natural order</p> <pre><code>$ echo -e \"&gt;3\\na\\n&gt;1\\na\\n&gt;Y\\na\\n&gt;x\\na\\n&gt;Mt\\na\\n&gt;11\\na\\n&gt;2\\na\\n\" \\\n    | seqkit seq -n -i\n3\n1\nY\nx\nMt\n11\n2\n\n$ echo -e \"&gt;3\\na\\n&gt;1\\na\\n&gt;Y\\na\\n&gt;x\\na\\n&gt;Mt\\na\\n&gt;11\\na\\n&gt;2\\na\\n\" \\\n    | seqkit sort -N -i -2 \\\n    | seqkit seq -n -i\n1\n2\n3\n11\nMt\nx\nY\n</code></pre> </li> <li> <p>sort by ID, ignoring case.</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAA\" \\\n    | seqkit sort --quiet -i\n&gt;seq1\nACGTNcccc\n&gt;SEQ2\nacgtnAAAA\n</code></pre> </li> <li> <p>sort by seq, ignoring case.</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAA\" \\\n    | seqkit sort --quiet -s -i\n&gt;SEQ2\nacgtnAAAA\n&gt;seq1\nACGTNcccc\n</code></pre> </li> <li> <p>sort by sequence length</p> <pre><code>$ echo -e \"&gt;seq1\\nACGTNcccc\\n&gt;SEQ2\\nacgtnAAAAnnn\\n&gt;seq3\\nacgt\" \\\n    | seqkit sort --quiet -l\n&gt;seq3\nacgt\n&gt;seq1\nACGTNcccc\n&gt;SEQ2\nacgtnAAAAnnn\n</code></pre> </li> </ol>"},{"location":"usage/#bam","title":"bam","text":"<pre><code>monitoring and online histograms of BAM record features\n\nUsage:\n  seqkit bam [flags]\n\nFlags:\n  -B, --bins int             number of histogram bins (default -1)\n  -N, --bundle int           partition BAM file into loci (-1) or bundles with this minimum size\n  -c, --count string         count reads per reference and save to this file\n  -W, --delay int            sleep this many seconds after plotting (default 1)\n  -y, --dump                 print histogram data to stderr instead of plotting\n  -G, --exclude-ids string   exclude records with IDs contained in this file\n  -e, --exec-after string    execute command after reporting\n  -E, --exec-before string   execute command before reporting\n  -f, --field string         target fields\n  -g, --grep-ids string      only keep records with IDs contained in this file\n  -h, --help                 help for bam\n  -C, --idx-count            fast read per reference counting based on the BAM index\n  -i, --idx-stat             fast statistics based on the BAM index\n  -O, --img string           save histogram to this PDF/image file\n  -H, --list-fields          list all available BAM record features\n  -L, --log                  log10(x+1) transform numeric values\n  -q, --map-qual int         minimum mapping quality\n  -x, --pass                 passthrough mode (forward filtered BAM to output)\n  -k, --pretty               pretty print certain TSV outputs\n  -F, --prim-only            filter out non-primary alignment records\n  -p, --print-freq int       print/report after this many records (-1 for print after EOF) (default -1)\n  -Q, --quiet-mode           supress all plotting to stderr\n  -M, --range-max float      discard record with field (-f) value greater than this flag (default NaN)\n  -m, --range-min float      discard record with field (-f) value less than this flag (default NaN)\n  -R, --reset                reset histogram after every report\n  -Z, --silent-mode          supress TSV output to stderr\n  -s, --stat                 print BAM satistics of the input files\n  -T, --tool string          invoke toolbox in YAML format (see documentation)\n  -@, --top-bam string       save the top -? records to this bam file\n  -?, --top-size int         size of the top-mode buffer (default 100)\n</code></pre> <p>Examples</p> <ol> <li> <p>Get detailed statistics from multiple BAM files.</p> <pre><code>seqkit bam -s *.bam\n</code></pre> </li> <li> <p>Get rough statistics from multiple indexed BAM files.</p> <pre><code>seqkit bam -i *.bam\n</code></pre> </li> <li> <p>Count reads mapped to references from a BAM stream.</p> <pre><code>cat sample.bam | seqkit bam -c counts.tsv  -\n</code></pre> </li> <li> <p>Count reads mapped to references using the BAM index.</p> <pre><code>seqkit bam -C sorted_indexed.bam\n</code></pre> </li> <li> <p>Monitor alignment accuracy from a bam stream and report after every 1000 records, use 20 bins.</p> <pre><code>cat sample.bam | seqkit bam -B -f Acc -p 1000 -\n</code></pre> </li> <li> <p>Dump selected fields to TSV.</p> <pre><code>seqkit bam -f Ref,Acc,RefCov,Strand sample.bam\n</code></pre> </li> <li> <p>Save the best 100 records in terms of alignment accuracy to a BAM file.</p> <pre><code>seqkit bam -f Acc -@ top_acc_100.bam -? 100 -Q sample.bam\n</code></pre> </li> <li> <p>Inkvoke the BAM toolbox.</p> </li> </ol> <p>The BAM toolbox is a collection of filters acting on a stream of BAM records, configured via YAML.  The currently available tools can be listed by <code>seqkit bam -T help</code>:</p> <pre><code>Tool        Description\n----        -----------\nAccStats    calculates mean accuracy weighted by aligment lengths\nAlnContext  filter records by the sequence context at start and end\nDump        dump various record properties in TSV format\nhelp        list all tools with description\n</code></pre> <p>Example YAML configs:</p> <p>Invoking the AccStats tool directly from the command line or YAML config:</p> <pre><code>seqkit bam -T '{AccStats: {Tsv: \"-\"}, Sink: True}' input.bam\nseqkit bam -T '{Yaml: \"tests/examples/bam_tool_acc_stats.yml\"}' input.bam\n</code></pre> <p>Where the contents of <code>bam_tool_acc_stats.yml</code> are:</p> <pre><code>AccStats:\n  Tsv: \"-\"\nSink: True\n</code></pre> <p>Invoking the AlnContext tool using YAML:</p> <pre><code>AlnContext:\n  Tsv: \"-\"\n  Ref: \"../SIRV_150601a.fasta\"\n  LeftShift: -10\n  RightShift: 10\n  RegexStart: \"T{4,}\"\n  RegexEnd: \"A{4,}\"\n  Stranded: True\n  Invert: True\nSink: True\n</code></pre> <p>Invoking the Dump tool using YAML:</p> <pre><code>Dump:\n  Tsv: \"-\"\n  Fields: [\"Read\", \"Ref\", \"Pos\", \"EndPos\", \"MapQual\", \"Acc\", \"Match\", \"Mismatch\", \"Ins\", \"Del\", \"AlnLen\", \"  ReadLen\", \"RefLen\", \"RefAln\", \"RefCov\", \"ReadAln\", \"ReadCov\", \"Strand\", \"MeanQual\", \"LeftClip\", \"RightClip\", \"Flags\", \"IsSec\", \"  IsSup\", \"ReadSeq\", \"ReadAlnSeq\", \"LeftSoftClipSeq\", \"RightSoftClip\", \"LeftHardClip\", \"RightHardClip\"]\nSink: True\n</code></pre> <p>The tools can be chained together, for example the YAML using all three tools look like:</p> <pre><code>AlnContext:\n  Tsv: \"context.tsv\"\n  Ref: \"../SIRV_150601a.fasta\"\n  LeftShift: -10\n  RightShift: 10\n  RegexStart: \"T{4,}\"\n  RegexEnd: \"A{4,}\"\n  Stranded: True\n  Invert: True\nDump:\n  Tsv: \"dump.tsv\"\n  Fields: [\"Read\", \"Ref\", \"Pos\", \"EndPos\", \"MapQual\", \"Acc\", \"Match\", \"Mismatch\", \"Ins\", \"Del\", \"AlnLen\", \"  ReadLen\", \"RefLen\", \"RefAln\", \"RefCov\", \"ReadAln\", \"ReadCov\", \"Strand\", \"MeanQual\", \"LeftClip\", \"RightClip\", \"Flags\", \"IsSec\", \"  IsSup\", \"ReadSeq\", \"ReadAlnSeq\", \"LeftSoftClipSeq\", \"RightSoftClip\", \"LeftHardClip\", \"RightHardClip\"]\nAccStats:\n  Tsv: \"-\"\n</code></pre> <p>If the \"Sink\" parameter is not specified in the last pipeline step, the output BAM records are streamed to the standard output and can be piped into standard tools, for example:</p> <pre><code>seqkit bam -T '{Yaml: \"bam_tool_pipeline.yml\"}' ../pcs109_5k_spliced.bam | samtools flagstat -\n</code></pre>"},{"location":"usage/#merge-slides","title":"merge-slides","text":"<p>Usage</p> <pre><code>merge sliding windows generated from seqkit sliding\n\nFor example,\n\n    ref.contig00001_sliding:454531-454680\n    ref.contig00001_sliding:454561-454710\n    ref.contig00001_sliding:454591-454740\n    ref.contig00002_sliding:362281-362430\n    ref.contig00002_sliding:362311-362460\n    ref.contig00002_sliding:362341-362490\n    ref.contig00002_sliding:495991-496140\n    ref.contig00044_sliding:1-150\n    ref.contig00044_sliding:31-180\n    ref.contig00044_sliding:61-210\n    ref.contig00044_sliding:91-240\n\ncould be merged into\n\n    ref.contig00001 454530  454740\n    ref.contig00002 362280  362490\n    ref.contig00002 495990  496140\n    ref.contig00044 0       240\n\nOutput (BED3 format):\n    1. chrom      - chromosome name\n    2. chromStart - starting position (0-based)\n    3. chromEnd   - ending position (0-based)\n\nUsage:\n  seqkit merge-slides [flags]\n\nFlags:\n  -b, --buffer-size string            size of buffer, supported unit: K, M, G. You need increase the\n                                      value when \"bufio.Scanner: token too long\" error reported (default\n                                      \"1G\")\n  -p, --comment-line-prefix strings   comment line prefix (default [#,//])\n  -h, --help                          help for merge-slides\n  -g, --max-gap int                   maximum distance of starting positions of two adjacent regions, 0\n                                      for no limitation, 1 for no merging.\n  -l, --min-overlap int               minimum overlap of two adjacent regions, recommend\n                                      $sliding_step_size - 1. (default 1)\n  -r, --regexp string                 regular expression for extract the reference name and window\n                                      position. (default \"^(.+)_sliding:(\\\\d+)\\\\-(\\\\d+)\")\n\n</code></pre> <p>Example:</p> <pre><code>seqkit merge-slides sliding_windows.tsv -l 50 -o sliding_windows.merged.tsv\n</code></pre>"},{"location":"usage/#genautocomplete","title":"genautocomplete","text":"<p>Usage</p> <pre><code>generate shell autocompletion script\n\nSupported shell: bash|zsh|fish|powershell\n\nBash:\n\n    # generate completion shell\n    seqkit genautocomplete --shell bash\n\n    # configure if never did.\n    # install bash-completion if the \"complete\" command is not found.\n    echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" &gt;&gt; ~/.bash_completion\n    echo \"source ~/.bash_completion\" &gt;&gt; ~/.bashrc\n\nZsh:\n\n    # generate completion shell\n    seqkit genautocomplete --shell zsh --file ~/.zfunc/_seqkit\n\n    # configure if never did\n    echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' &gt;&gt; ~/.zshrc\n    echo \"autoload -U compinit; compinit\" &gt;&gt; ~/.zshrc\n\nfish:\n\n    seqkit genautocomplete --shell fish --file ~/.config/fish/completions/seqkit.fish\n\nUsage:\n  seqkit genautocomplete [flags]\n\nFlags:\n      --file string   autocompletion file (default \"/home/shenwei/.bash_completion.d/seqkit.sh\")\n  -h, --help          help for genautocomplete\n      --type string   autocompletion type (currently only bash supported) (default \"bash\")\n\n</code></pre> Please enable JavaScript to view the comments powered by Disqus."},{"location":"yanlilab/","title":"Yanlilab","text":""}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index d66724d28a1914f69acb8376a476bbe13013524c..ed981feec36fb18f1805e8e77fdf6931e08a1856 100644
GIT binary patch
delta 14
Vcmb=g=aBE_;PBg_l0K268~`911Y!UH

delta 14
Vcmb=g=aBE_;LzFmGHoJ9IRGOE1qA>A

diff --git a/usage/index.html b/usage/index.html
index 5170129..4e0e7f4 100644
--- a/usage/index.html
+++ b/usage/index.html
@@ -1322,7 +1322,7 @@ <h3 id="reproducibility">Reproducibility</h3>
 <h2 id="seqkit">seqkit</h2>
 <pre><code class="language-text">SeqKit -- a cross-platform and ultrafast toolkit for FASTA/Q file manipulation
 
-Version: 2.8.0
+Version: 2.9.0
 
 Author: Wei Shen &lt;shenwei356@gmail.com&gt;
 
@@ -1873,6 +1873,7 @@ <h2 id="stats">stats</h2>
   16. Q30(%)    percentage of bases with the quality score greater than 30
   17. AvgQual   average quality
   18. GC(%)     percentage of GC content
+  19. sum_n     number of ambitious letters (N, n, X, x)
 
 Attention:
   1. Sequence length metrics (sum_len, min_len, avg_len, max_len, Q1, Q2, Q3)
@@ -2020,13 +2021,13 @@ <h2 id="stats">stats</h2>
 <li>
 <p>Extra information</p>
 <pre><code>$ seqkit stats *.f{a,q}.gz -a
-file               format  type  num_seqs    sum_len  min_len  avg_len  max_len   Q1   Q2   Q3  sum_gap  N50  N50_num  Q20(%)  Q30(%)  AvgQual  GC(%)
-hairpin.fa.gz      FASTA   RNA     28,645  2,949,871       39      103    2,354   76   91  111        0  101      380       0       0        0  45.77
-mature.fa.gz       FASTA   RNA     35,828    781,222       15     21.8       34   21   22   22        0   22       12       0       0        0   47.6
-Illimina1.8.fq.gz  FASTQ   DNA     10,000  1,500,000      150      150      150  150  150  150        0  150        1   96.16   89.71    24.82  49.91
-nanopore.fq.gz     FASTQ   DNA      4,000  1,798,723      153    449.7    6,006  271  318  391        0  395      585   40.79   12.63     9.48  46.66
-reads_1.fq.gz      FASTQ   DNA      2,500    567,516      226      227      229  227  227  227        0  227        3   91.24   86.62    15.45  53.63
-reads_2.fq.gz      FASTQ   DNA      2,500    560,002      223      224      225  224  224  224        0  224        2   91.06   87.66    14.62  54.77
+file               format  type  num_seqs    sum_len  min_len  avg_len  max_len   Q1   Q2   Q3  sum_gap  N50  N50_num  Q20(%)  Q30(%)  AvgQual  GC(%)  sum_n
+hairpin.fa.gz      FASTA   RNA     28,645  2,949,871       39      103    2,354   76   91  111        0  101      380       0       0        0  45.77    255
+mature.fa.gz       FASTA   RNA     35,828    781,222       15     21.8       34   21   22   22        0   22       12       0       0        0   47.6      0
+Illimina1.8.fq.gz  FASTQ   DNA     10,000  1,500,000      150      150      150  150  150  150        0  150        1   96.16   89.71    24.82  49.91     38
+nanopore.fq.gz     FASTQ   DNA      4,000  1,798,723      153    449.7    6,006  271  318  391        0  395      585   40.79   12.63     9.48  46.66      0
+reads_1.fq.gz      FASTQ   DNA      2,500    567,516      226      227      229  227  227  227        0  227        3   91.24   86.62    15.45  53.63     44
+reads_2.fq.gz      FASTQ   DNA      2,500    560,002      223      224      225  224  224  224        0  224        2   91.06   87.66    14.62  54.77      2
 </code></pre>
 </li>
 <li>
@@ -4274,7 +4275,9 @@ <h2 id="replace">replace</h2>
 
 Special replacement symbols (only for replacing name not sequence):
 
-    {nr}    Record number, starting from 1
+    {fn}    File name
+    {fbn}   File base name
+    {fbne}  File base name without any extension
     {kv}    Corresponding value of the key (captured variable $n) by key-value file,
             n can be specified by flag -I (--key-capt-idx) (default: 1)
 
@@ -4425,6 +4428,14 @@ <h2 id="replace">replace</h2>
 </code></pre>
 </li>
 <li>
+<p>Add file names.</p>
+<pre><code>$ seqkit replace ../tests/hairpin.fa -p '.+' -r '{fn}__{fbn}__{fbne}__{nr}' | seqkit seq -n | head -n 3
+../tests/hairpin.fa__hairpin.fa__hairpin__1
+../tests/hairpin.fa__hairpin.fa__hairpin__2
+../tests/hairpin.fa__hairpin.fa__hairpin__3
+</code></pre>
+</li>
+<li>
 <p>Replace key with value by key-value file</p>
 <pre><code>$ more test.fa
 &gt;seq1 name1