`npm query` Command & Dependency Selector Syntax

Summary

Introduce a new npm query commmand which exposes a new dependency selector syntax (informed by & respecting many aspects of the CSS Selectors 4 Spec).

Motivation

Standardize the shape of, & querying of, dependency graphs with a robust object model, metadata & selector syntax
Leverage existing, known language syntax & operators from CSS to make disperate package information broadly accessible
Unlock the ability to anwser complex, multi-faceted questions about dependencies, their relationships & associative metadata
Consolidate redundant logic of similar query commands in npm (ex. npm fund, npm ls, npm outdated, npm audit ...)

Detailed Explanation

This RFC's spec & implementation should closely mimic the capabilities of existing CSS Selector specifications. Notably, we'll introduce limited net-new classes, states & syntax to ensure the widest adoption & understanding of paradigms. When deviating, we'll explicitely state why & how.

Implementation

Arborist's Node Class will have a new .querySelectorAll() method
- this method will return a filtered, flattened dependency Arborist Node list based on a valid query selector
Introduce a new command, npm query, which will take a dependency selector & output a flattened dependency Node list (output is in json by default, but configurable)

Dependency Selector Syntax

Overview:

there is no "type" or "tag" selectors (ex. div, h1, a) as a dependency/target is the only type of Node that can be queried
the term "dependencies" is in reference to any Node found in the idealTree returned by Arborist

Selectors

* universal selector
#<name> dependency selector (equivalent to [name="..."])
#<name>@<version> (equivalent to [name=<name>]:semver(<version>))
, selector list delimiter
. class selector
: pseudo class selector
> direct decendent/child selector
~ sibling selector

Attribute Selectors

[] attribute selector (ie. existence of attribute)
[attribute=value] attribute value is equivalant...
[attribute~=value] attribute value contains word...
[attribute*=value] attribute value contains string...
[attribute|=value] attribute value is equal to or starts with...
[attribute^=value] attribute value begins with...
[attribute$=value] attribute value ends with...

Pseudo Selectors

:not(<selector>)
:has(<selector>)
:where(<selector list>)
:is(<selector list>)
:root matches the root node/dependency
:scope matches node/dependency it was queried against
:empty when a dependency has no dependencies
:private when a dependency is private
:link when a dependency is linked
:deduped when a dependency has been deduped
:override when a dependency is an override
:extraneous when a dependency could be but is not deduped
:outdated when a dependency is not latest
:vulnerable when a dependency has a CVE
:semver(<spec>) matching a valid node-semver spec
:path(<path>) glob matching based on dependencies path relative to the project
:realpath(<path>) glob matching based on dependencies realpath
:type(<type>) based on currently recognized types

Generic Pseudo Selectors:

A standardized pseudo selector pattern will be used for Objects, Arrays or Arrays of Objects accessible via Arborist's Node.package metadata. Pseudo classes generated in this way will allow for iterative attribute selection.

Arrays specifically use a special value keyword in place of a typical attribute name. Arrays also support exact value matching when a String is passed to the selector. See examples below:

Example of an `Object`:

/* return dependencies that have a `scripts.test` containing `"tap"` */
*:scripts([test~=tap])

Example of an `Array` Attribute Selection:

/* return dependencies that have a keyword that begins with "react" */
*:keywords([value^="react"])

Example of an `Array` matching directly to a value:

/* return dependencies that have the exact keyword "react" */
/* this is equivalent to `*:keywords([value="react"])` */
*:keywords("react")

Example of an `Array` of `Object`s:

/* returns */
*:contributors([email="ruyadorno@github.com"])`

Classes

Given that Arborist will control our understanding of the DOM (Dependency Object Model), claseses are predefined by their relationships to their ancestors & are specific to their dependency type. Dependencies may have, & are allowed to have, multiple classifications (ex. a workspace may also be a dev dependency).

.prod
.dev
.optional
.peer
.bundled
.workspace

Attributes

There are several known attributes that are normalized & queryable living in Node.package (aka. package.json).

[name]
[version]
[description]
[homepage]
[bugs]
[author]
[license]
[funding]
[files]
[main]
[browser]
[bin]
[man]
[directories]
[repository]
[scripts]
[config]
[workspaces]
[dependencies]
[devDependencies]
[optionalDependencies]
[bundledDependencies]
[peerDependencies]
[peerDependenciesMeta]
[engines]
[os]
[cpu]
[keywords]

Command

`npm query "<selector>"` (alias `q`)

Options:

query-output
- Default: json
- Type: json, list, explain, outdated, funding, audit, duplicates, file

Response Output

an array of dependency objects is returned which can contain multiple copies of the same package which may or may not have been linked or deduped

[
  {
    "name": "",
    "version": "",
    "description": "",
    "homepage": "",
    "bugs": {},
    "author": {},
    "license": {},
    "funding": {},
    "files": [],
    "main": "",
    "browser": "",
    "bin": {},
    "man": [],
    "directories": {},
    "repository": {},
    "scripts": {},
    "config": {},
    "dependencies": {},
    "devDependencies": {},
    "optionalDependencies": {},
    "bundledDependencies": {},
    "peerDependencies": {},
    "peerDependenciesMeta": {},
    "engines": {},
    "os": [],
    "cpu": [],
    "workspaces": {},
    "keywords": [],
    "ancenstry": "",
    "path": "",
    "realpath": "",
    "parent": "",
    "vulnerabilities": [],
    "cwe": []
  },
  ...

Usage

npm query ":root > .workspace > *"  # get all workspace direct deps

Extended Example Queries & Use Cases

// all deps
*

// all direct deps
:root > *

// direct production deps
:root > .prod

// direct development deps
:root > .dev 

// any peer dep of a direct deps
:root > * > .peer  
  
// any workspace dep
.workspace

// all workspaces that depend on another workspace
.workspace > .workspace
    
// all workspaces that have peer deps
.workspace:has(*.peer)

// any dep named "lodash"
// equivalent to [name="lodash"]
#lodash 
  
// any deps named "lodash" & within semver range ^"1.2.3"
#lodash@^1.2.3
// equivalent to...
[name="lodash"]:semver(^1.2.3)

// get the hoisted node for a given semver range
#lodash@^1.2.3:not(:deduped)

// querying deps with a specific version
#lodash@2.1.5
// equivalent to...
[name="lodash"][version="2.1.5"]
  
// all deps living alongside vulnerable deps
*:vulnerable ~ *:not(:vulnerable)

// has any deps
*:has(*)

// has any vulnerable deps
*:has(*:vulnerable)

// deps with no other deps (ie. "leaf" nodes)
*:empty

// all vulnerable deps that aren't dev deps & that aren't vulnerable to CWE-1333
*:vulnerable:not(.dev:cwe(1333))
  
// manually querying git dependencies
*[repository^="github:"],
*[repository^="git:"],
*[repository^="https://github.com"],
*[repository^="http://github.com"],
*[repository^="https://github.com"],
*[repository^="+git:..."]
  
// querying for all git dependencies
*:type(git)

// find all references to "install" scripts
*[scripts=install], 
*[scripts=postinstall],
*[scripts=preinstall]

// get production dependencies that aren't also dev deps
.prod:not(.dev)
    
// get dependencies with specific licenses
*[license="MIT"], *[license="ISC"]

// find all packages that have @ruyadorno as a contributor
*:contributors([email=ruyadorno@github.com])

Next Steps

Command Mapping to `query-output`

Previous commands with similar behaivours will now be able to utilize Aborist Node.querySelectorAll() under-the-hood & will fast-follow the npm query implementation.

`npm list`

npm list # equivalent to...
npm query ":root > *"

npm list --all # equivalent to...
npm query "*" --query-output list

npm list <pkg> # equivalent to...
npm query "#<pkg>" --query-output list

`npm explain`

npm explain <pkg> # equivalent to...
npm query "#<pkg>" --query-output explain

`npm outdated`

npm outdated # equivalent to...
npm query ":root > *:outdated" --query-output outdated

npm outdated --all # equivalent to...
npm query "*:outdated" --query-output outdated

`npm fund`

npm fund # equivalent to...
npm query ":root > *[funding]" --query-output funding

npm fund --all # equivalent to...
npm query "*[funding]" --query-output funding

npm fund <pkg> # equivalent to...
npm query "#<pkg>" --query-output funding

`npm audit`

npm audit # equivalent to...
npm query ":root > *:vulnerable" --query-output audit

npm audit --all # equivalent to...
npm query "*:vulnerable" --query-output audit

npm audit <pkg> # equivalent to...
npm query "#<pkg>" --query-output audit

`npm find-dupes`

npm find-dupes # equivalent to...
npm query "*:deduped" --query-output duplicates

`npm view`

npm view # equivalent to...
npm query ":root" --query-output view

npm view <pkg> # equivalent to...
npm query "#<pkg>" --query-output view

Commands could read from `stdin`

In a future RFC, & major version bump, npm could begin reading from stdin to chain commands together with a common understand of a dependency object. All of the below commands would add the ability to execute as if they were passed package specs (our current defulat representation of packages/dependencies).

audit, bugs, ci, config, deprecate, diff, dist-tag, docs, edit, exec, 
explain, explore, find-dupes, fund, get, install, install-ci-test, 
install-test, link, list, outdated, pack, pkg, prune, publish, rebuild, 
repo, restart, run-script, set, set-script, shrinkwrap, star, stars,
start, stop, team, test, token, uninstall, unpublish, unstar, update,
version, view

Example of piping from `npm query` to other commands

# list workspaces w/ peer deps
npm query ".workspace:has(.peer)" | npm ls

# list outdated direct dev deps
npm query ":root > .dev:outdated" | npm outdated

# install the same dev deps across all workspaces
npm query ":root > .dev" | npm install --workspaces

# show audit details for dependencies with a specific vulnerability/CWE
npm query "*:cwe(1333)" | npm audit

# show audit details for vulnerable deps that aren't ReDoS dev deps
npm query "*:vulnerable:not(.dev:cwe(1333))" | npm audit

Prior Art

HTML & DOM Specifications
CSS, Selectors & Pseudo Class Specifications
AST Selector Libraries/Parsers
pnpm's --filter Flag
Gzemnid Package Querying

F.A.Q.

Q. Is there any such thing as a bare specifier?
- A. No. Unlike CSS Selectors, there's no "element", or "dependency" in this context, equivalent. The selector syntax presuposes all entities are packages.
Q. Should this syntax cover all possible queries of a dependency graph?
- A. No. This spec is meant to provide a sufficently mature mechanism/building block for answering the majority of questions end-users have about their depndencies (re. 80/20 rule applies here)

Unresolved Questions & Bikeshedding

N/A

Files

0000-dependency-selector-syntax.md

Latest commit

History

0000-dependency-selector-syntax.md

File metadata and controls

npm query Command & Dependency Selector Syntax