Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add and fix ECLint in pipeline #629

Merged
merged 18 commits into from
May 12, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_size = 4
indent_style = space

[*.{yml,yaml}]
indent_size = 2

# These files are edited upstream in nf-core/modules
[/modules/nf-core/**]
charset = unset
end_of_line = unset
insert_final_newline = unset
trim_trailing_whitespace = unset
indent_style = unset
indent_size = unset

[/assets/email*]
indent_size = unset
4 changes: 2 additions & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/rnas

- [ ] This comment contains a description of changes (with reason).
- [ ] If you've fixed a bug or added code that should be tested, add tests!
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/rnaseq/tree/master/.github/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the nf-core/rnaseq _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/rnaseq/tree/master/.github/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the nf-core/rnaseq _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
- [ ] Make sure your code lints (`nf-core lint .`).
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`).
- [ ] Usage Documentation in `docs/usage.md` is updated.
Expand Down
27 changes: 19 additions & 8 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,17 @@ on:
types: [published]

jobs:

Markdown:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v1
with:
node-version: '10'
node-version: "10"
- name: Install markdownlint
run: npm install -g markdownlint-cli
- name: Run Markdownlint
run: markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml
run: markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.markdownlint.yml

# If the above check failed, post a comment on the PR explaining the failure
- name: Post PR comment
Expand Down Expand Up @@ -47,14 +46,28 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}
allow-repeats: false

EditorConfig:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- uses: actions/setup-node@v1
with:
node-version: "10"

- name: Install editorconfig-checker
run: npm install -g editorconfig-checker

- name: Run ECLint check
run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test)

YAML:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/setup-node@v1
with:
node-version: '10'
node-version: "10"
- name: Install yaml-lint
run: npm install -g yaml-lint
- name: Run yaml-lint
Expand Down Expand Up @@ -88,7 +101,6 @@ jobs:
nf-core:
runs-on: ubuntu-latest
steps:

- name: Check out pipeline code
uses: actions/checkout@v2

Expand All @@ -101,8 +113,8 @@ jobs:

- uses: actions/setup-python@v1
with:
python-version: '3.6'
architecture: 'x64'
python-version: "3.6"
architecture: "x64"

- name: Install dependencies
run: |
Expand All @@ -129,4 +141,3 @@ jobs:
lint_log.txt
lint_results.md
PR_number.txt

4 changes: 4 additions & 0 deletions .github/markdownlint.yml → .markdownlint.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Markdownlint configuration file
default: true
line-length: false
ul-indent:
indent: 4
no-duplicate-header:
siblings_only: true
no-inline-html:
Expand All @@ -10,3 +12,5 @@ no-inline-html:
- kbd
- details
- summary
single-title:
level: 2
3 changes: 3 additions & 0 deletions .nf-core-lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@ files_unchanged:
- assets/email_template.html
- assets/email_template.txt
- assets/multiqc_config.yaml
- assets/sendmail_template.txt
- lib/NfcoreSchema.groovy
- docs/README.md
files_exist:
- .github/markdownlint.yml
- bin/markdown_to_html.py
actions_awsfulltest: False
67 changes: 34 additions & 33 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### :warning: Major enhancements

* Samplesheet format has changed from `group,replicate,fastq_1,fastq_2,strandedness` to `sample,fastq_1,fastq_2,strandedness`
* This gives users the flexibility to name their samples however they wish (see [#550](https://github.com/nf-core/rnaseq/issues/550]))
* PCA generated by DESeq2 will now be monochrome and will not be grouped by using the replicate id
* This gives users the flexibility to name their samples however they wish (see [#550](https://github.com/nf-core/rnaseq/issues/550]))
* PCA generated by DESeq2 will now be monochrome and will not be grouped by using the replicate id
* Updated Nextflow version to `v21.04.0` (see [nextflow#572](https://github.com/nextflow-io/nextflow/issues/1964))
* Restructure pipeline scripts into `modules/`, `subworkflows/` and `workflows/` directories

Expand All @@ -34,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* [[#604](https://github.com/nf-core/rnaseq/issues/604)] - Additional fasta with GENCODE annotation results in biotype error
* [[#610](https://github.com/nf-core/rnaseq/issues/610)] - save R objects as RDS
* [[#619](https://github.com/nf-core/rnaseq/issues/619)] - implicit declaration of the workflow in main
* [[#629](https://github.com/nf-core/modules/pull/629)] - Add and fix EditorConfig linting in entire pipeline
* [[nf-core/modules#423](https://github.com/nf-core/modules/pull/423)] - Replace `publish_by_id` module option to `publish_by_meta`
* [[nextflow#2060](https://github.com/nextflow-io/nextflow/issues/2060)] - Pipeline execution hang when native task fail to be submitted

Expand All @@ -50,9 +51,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
| | `--validate_params` |
| `--clusterOptions` | |

> **NB:** Parameter has been __updated__ if both old and new parameter information is present.
> **NB:** Parameter has been __added__ if just the new parameter information is present.
> **NB:** Parameter has been __removed__ if parameter information isn't present.
> **NB:** Parameter has been __updated__ if both old and new parameter information is present.
> **NB:** Parameter has been __added__ if just the new parameter information is present.
> **NB:** Parameter has been __removed__ if parameter information isn't present.

### Software dependencies

Expand All @@ -64,20 +65,20 @@ Note, since the pipeline is now using Nextflow DSL2, each process will be run wi
| `multiqc` | 1.9 | 1.10.1 |
| `preseq` | 2.0.3 | 3.1.2 |

> **NB:** Dependency has been __updated__ if both old and new version information is present.
> **NB:** Dependency has been __added__ if just the new version information is present.
> **NB:** Dependency has been __removed__ if version information isn't present.
> **NB:** Dependency has been __updated__ if both old and new version information is present.
> **NB:** Dependency has been __added__ if just the new version information is present.
> **NB:** Dependency has been __removed__ if version information isn't present.

## [[3.0](https://github.com/nf-core/rnaseq/releases/tag/3.0)] - 2020-12-15

### :warning: Major enhancements

* You will need to install Nextflow `>=20.11.0-edge` to run the pipeline. If you are using Singularity, then features introduced in that release now enable the pipeline to directly download Singularity images hosted by Biocontainers as opposed to performing a conversion from Docker images (see [#496](https://github.com/nf-core/rnaseq/issues/496)).
* The previous default of aligning BAM files using STAR and quantifying using featureCounts (`--aligner star`) has been removed. The new default is to align with STAR and quantify using Salmon (`--aligner star_salmon`).
* This decision was made primarily because of the limitations of featureCounts to appropriately quantify gene expression data. Please see [Zhao et al., 2015](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141910#pone-0141910-t001) and [Soneson et al., 2015](https://f1000research.com/articles/4-1521/v1)).
* This decision was made primarily because of the limitations of featureCounts to appropriately quantify gene expression data. Please see [Zhao et al., 2015](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141910#pone-0141910-t001) and [Soneson et al., 2015](https://f1000research.com/articles/4-1521/v1)).
* For similar reasons, **quantification will not be performed** if using `--aligner hisat2` due to the lack of an appropriate option to calculate accurate expression estimates from HISAT2 derived genomic alignments.
* This pipeline option is still available for those who have a preference for the alignment, QC and other types of downstream analysis compatible with the output of HISAT2. No gene-level quantification results will be generated.
* In a future release we hope to add back quantitation for HISAT2 using different tools.
* This pipeline option is still available for those who have a preference for the alignment, QC and other types of downstream analysis compatible with the output of HISAT2. No gene-level quantification results will be generated.
* In a future release we hope to add back quantitation for HISAT2 using different tools.

### Enhancements & fixes

Expand Down Expand Up @@ -108,9 +109,9 @@ Note, since the pipeline is now using Nextflow DSL2, each process will be run wi
|  | `--singularity_pull_docker_container` |
| `--skip_featurecounts` |  |

> **NB:** Parameter has been __updated__ if both old and new parameter information is present.
> **NB:** Parameter has been __added__ if just the new parameter information is present.
> **NB:** Parameter has been __removed__ if parameter information isn't present.
> **NB:** Parameter has been __updated__ if both old and new parameter information is present.
> **NB:** Parameter has been __added__ if just the new parameter information is present.
> **NB:** Parameter has been __removed__ if parameter information isn't present.

### Software dependencies

Expand All @@ -126,9 +127,9 @@ Note, since the pipeline is now using Nextflow DSL2, each process will be run wi
| `ucsc-bedclip` | | 377 |
| `umi_tools` | 1.0.1 | 1.1.1 |

> **NB:** Dependency has been __updated__ if both old and new version information is present.
> **NB:** Dependency has been __added__ if just the new version information is present.
> **NB:** Dependency has been __removed__ if version information isn't present.
> **NB:** Dependency has been __updated__ if both old and new version information is present.
> **NB:** Dependency has been __added__ if just the new version information is present.
> **NB:** Dependency has been __removed__ if version information isn't present.

## [[2.0](https://github.com/nf-core/rnaseq/releases/tag/2.0)] - 2020-11-12

Expand Down Expand Up @@ -244,10 +245,10 @@ Note, since the pipeline is now using Nextflow DSL2, each process will be run wi
* `--skipEdgeR` qc not performed by DESeq2 instead
* `--star_memory` in favour of updating centrally on nf-core/modules if required
* Strandedness is now specified at the sample-level via the input samplesheet
* `--forwardStranded`
* `--reverseStranded`
* `--unStranded`
* `--pico`
* `--forwardStranded`
* `--reverseStranded`
* `--unStranded`
* `--pico`

### Software dependencies

Expand Down Expand Up @@ -292,9 +293,9 @@ Note, since the pipeline is now using Nextflow DSL2, each process will be run wi
| `r-gplots` | - | - |
| `r-markdown` | - | - |

> **NB:** Dependency has been __updated__ if both old and new version information is present.
> **NB:** Dependency has been __added__ if just the new version information is present.
> **NB:** Dependency has been __removed__ if version information isn't present.
> **NB:** Dependency has been __updated__ if both old and new version information is present.
> **NB:** Dependency has been __added__ if just the new version information is present.
> **NB:** Dependency has been __removed__ if version information isn't present.

## [[1.4.2](https://github.com/nf-core/rnaseq/releases/tag/1.4.2)] - 2019-10-18

Expand All @@ -321,9 +322,9 @@ Major novel changes include:
* Several improvements in `featureCounts` handling of types other than `exon`. It is possible now to handle nuclearRNAseq data. Nuclear RNA has un-spliced RNA, and the whole transcript, including the introns, needs to be counted, e.g. by specifying `--fc_count_type transcript`.
* Support for [outputting unaligned data](https://github.com/nf-core/rnaseq/issues/277) to results folders.
* Added options to skip several steps
* Skip trimming using `--skipTrimming`
* Skip BiotypeQC using `--skipBiotypeQC`
* Skip Alignment using `--skipAlignment` to only use pseudo-alignment using Salmon
* Skip trimming using `--skipTrimming`
* Skip BiotypeQC using `--skipBiotypeQC`
* Skip Alignment using `--skipAlignment` to only use pseudo-alignment using Salmon

### Documentation updates

Expand Down Expand Up @@ -434,7 +435,7 @@ Major novel changes include:
* Remove legacy rseqc `strandRule` config code. [#119](https://github.com/nf-core/rnaseq/issues/119)
* Added STRINGTIE ballgown output to results folder [#125](https://github.com/nf-core/rnaseq/issues/125)
* HiSAT index build now requests `200GB` memory, enough to use the exons / splice junction option for building.
* Added documentation about the `--hisatBuildMemory` option.
* Added documentation about the `--hisatBuildMemory` option.
* BAM indices are stored and re-used between processes [#71](https://github.com/nf-core/rnaseq/issues/71)

### Bug Fixes
Expand Down Expand Up @@ -474,25 +475,25 @@ In summary, the main changes are:
* Rebranding and renaming throughout the pipeline to nf-core
* Updating many parts of the pipeline config and style to meet nf-core standards
* Support for GFF files in addition to GTF files
* Just use `--gff` instead of `--gtf` when specifying a file path
* Just use `--gff` instead of `--gtf` when specifying a file path
* New command line options to skip various quality control steps
* More safety checks when launching a pipeline
* Several new sanity checks - for example, that the specified reference genome exists
* Several new sanity checks - for example, that the specified reference genome exists
* Improved performance with memory usage (especially STAR and Picard)
* New BigWig file outputs for plotting coverage across the genome
* Refactored gene body coverage calculation, now much faster and using much less memory
* Bugfixes in the MultiQC process to avoid edge cases where it wouldn't run
* MultiQC report now automatically attached to the email sent when the pipeline completes
* New testing method, with data on GitHub
* Now run pipeline with `-profile test` instead of using bash scripts
* Now run pipeline with `-profile test` instead of using bash scripts
* Rewritten continuous integration tests with Travis CI
* New explicit support for Singularity containers
* Improved MultiQC support for DupRadar and featureCounts
* Now works for all users instead of just NGI Stockholm
* Now works for all users instead of just NGI Stockholm
* New configuration for use on AWS batch
* Updated config syntax to support latest versions of Nextflow
* Built-in support for a number of new local HPC systems
* CCGA, GIS, UCT HEX, updates to UPPMAX, CFC, BINAC, Hebbe, c3se
* CCGA, GIS, UCT HEX, updates to UPPMAX, CFC, BINAC, Hebbe, c3se
* Slightly improved documentation (more updates to come)
* Updated software packages

Expand Down
Loading