Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tooling: Ensure consistent formatting in all scripts #30795

Closed
gziolo opened this issue Apr 13, 2021 · 5 comments
Closed

Tooling: Ensure consistent formatting in all scripts #30795

gziolo opened this issue Apr 13, 2021 · 5 comments
Labels
Good First Issue An issue that's suitable for someone looking to contribute for the first time Needs Dev Ready for, and needs developer efforts [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Build Tooling Issues or PRs related to build tooling [Type] Code Quality Issues or PRs that relate to code quality

Comments

@gziolo
Copy link
Member

gziolo commented Apr 13, 2021

What problem does this address?

Follow-up for #30714.

I tested npm run fixtures:regenerate with all the formatting applied and it looks like it uses different formatting so definitely something to look at:

Screen Shot 2021-04-13 at 10 45 25

It is not only ranges that is formatted differently but it also uses spaces for indentation ...

Follow-up for #30715.

npx prettier --write **/*.md doesn't play nicely with npm run docs:build. @nosolosw, is it possible to include formatting in the process of generating automated docs with @wordpress/docgen or is better to run code formatting as the post-processing step inside the shell script we use?

What is your proposed solution?

It looks like npm run fixtures:regenerate command conflicts with what npx prettier --write **/*.json does for all JSON files. We should update npm run fixtures:regenerate to produce output that is aligned with Prettier formatting.

npx prettier --write **/*.md doesn't play nicely with npm run docs:build. We should update @wordpress/docgen to align both.

@gziolo gziolo added Needs Dev Ready for, and needs developer efforts Good First Issue An issue that's suitable for someone looking to contribute for the first time [Type] Code Quality Issues or PRs that relate to code quality [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Build Tooling Issues or PRs related to build tooling labels Apr 13, 2021
@oandregal
Copy link
Member

👋 What would be some instructions to reproduce the issue?

Doing npx prettier --format **/*.md results in a console output that I don't know how to interpret:

# Managing Packages

This repository uses lerna to manage WordPress modules and publish them as packages to npm.

Creating a New Package

When creating a new package, you need to provide at least the following:

  1. package.json based on the template:

    {
    	"name": "@wordpress/package-name",
    	"version": "1.0.0-prerelease",
    	"description": "Package description.",
    	"author": "The WordPress Contributors",
    	"license": "GPL-2.0-or-later",
    	"keywords": [ "wordpress" ],
    	"homepage": "https://github.com/WordPress/gutenberg/tree/HEAD/packages/package-name/README.md",
    	"repository": {
    		"type": "git",
    		"url": "https://github.com/WordPress/gutenberg.git"
    	},
    	"bugs": {
    		"url": "https://github.com/WordPress/gutenberg/issues"
    	},
    	"main": "build/index.js",
    	"module": "build-module/index.js",
    	"react-native": "src/index",
    	"dependencies": {
    		"@babel/runtime": "^7.13.10"
    	},
    	"publishConfig": {
    		"access": "public"
    	}
    }

    This assumes that your code is located in the src folder and will be transpiled with Babel.

  2. .npmrc file which disables creating package-lock.json file for the package:

    package-lock=false
    
  3. README.md file containing at least:

    • Package name
    • Package description
    • Installation details
    • Usage example
    • API documentation, if applicable (more info)
    • Code is Poetry logo (<br/><br/><p align="center"><img src="https://s.w.org/style/images/codeispoetry.png?1" alt="Code is Poetry." /></p>)
  4. CHANGELOG.md file containing at least:

    <!-- Learn how to maintain this file at https://github.com/WordPress/gutenberg/tree/HEAD/packages#maintaining-changelogs. -->
    
    ## Unreleased
    
    Initial release.
    

Managing Dependencies

There are two types of dependencies that you might want to add to one of the existing WordPress packages.

Production Dependencies

Production dependencies are stored in the dependencies section of the package’s package.json file.

Adding New Dependencies

The simplest way to add a production dependency to one of the packages is to run a very convenient lerna add command from the root of the project.

Example:

lerna add lodash packages/a11y

This command adds the latest version of lodash as a dependency to the @wordpress/a11y package, which is located in packages/a11y folder.

Removing Existing Dependencies

Removing a dependency from one of the WordPress packages requires some manual work. You need to remove the line in the corresponding dependencies section of the package.json file.

Example:

+++ b/packages/scripts/package.json
@@ -43,7 +43,6 @@
                "check-node-version": "^4.1.0",
                "cross-spawn": "^5.1.0",
                "eslint": "^7.1.0",
-               "jest": "^26.6.3",
                "minimist": "^1.2.0",
                "npm-package-json-lint": "^3.6.0",

Next, you need to run npm install in the root of the project to ensure that package-lock.json file gets properly regenerated.

Updating Existing Dependencies

This is the most confusing part of working with lerna which causes a lot of hassles for contributors. The most successful strategy so far is to do the following:

  1. First, remove the existing dependency as described in the previous section.
  2. Next, add the same dependency back as described in the first section of this chapter. This time it wil get the latest version applied unless you enforce a different version explicitly.

Development Dependencies

In contrast to production dependencies, development dependencies shouldn't be stored in individual WordPress packages. Instead they should be installed in the project's package.json file using the usual npm install command. In effect, all development tools are configured to work with every package at the same time to ensure they share the same characteristics and integrate correctly with each other.

Example:

npm install glob --save-dev

This commands adds the latest version of glob as a development dependency to the package.json file. It has to be executed from the root of the project.

Maintaining API documentation

Each public API change needs to be reflected in the corresponding API documentation. To ensure that code and documentation are in sync automatically, Gutenberg has developed a few utilities.

Packages can add the following HTML comment within their top-level README.md:

<!-- START TOKEN(Autogenerated API docs) -->

Content within the HTML comment will be replaced by the generated documentation.

<!-- END TOKEN(Autogenerated API docs) -->`.

Each time there is a commit to the public API of the package the README.md will be updated and kept in sync.

The above snippet within the package's README.md signals the Gutenberg utilities to go to src/index.js and extract the JSDoc comments of the export statements into a more friendly format.

Packages may want to use a different source file or add the exports of many files into the same README.md (see packages/core-data/README.md as an example); they can do so by adding the relative path to be used as source into the HTML comment:

<!-- START TOKEN(Autogenerated API docs|src/actions.js) -->

Content within the HTML comment will be replaced by the generated documentation.

<!-- END TOKEN(Autogenerated API docs|src/actions.js) -->`.

Maintaining Changelogs

In maintaining dozens of npm packages, it can be tough to keep track of changes. To simplify the release process, each package includes a CHANGELOG.md file which details all published releases and the unreleased ("Unreleased") changes, if any exist.

For each pull request, you should always include relevant changes in a "Unreleased" heading at the top of the file. You should add the heading if it doesn't already exist.

Example:

<!-- Learn how to maintain this file at https://github.com/WordPress/gutenberg/tree/HEAD/packages#maintaining-changelogs. -->

## Unreleased

### Bug Fix

-   Fixed an off-by-one error with the `sum` function.

There are a number of common release subsections you can follow. Each is intended to align to a specific meaning in the context of the Semantic Versioning (semver) specification the project adheres to. It is important that you describe your changes accurately, since this is used in the packages release process to help determine the version of the next release.

  • "Breaking Change" - A backwards-incompatible change which requires specific attention of the impacted developers to reconcile (requires a major version bump).
  • "New Feature" - The addition of a new backwards-compatible function or feature to the existing public API (requires a minor version bump).
  • "Enhancement" - Backwards-compatible improvements to existing functionality (requires a minor version bump).
  • "Bug Fix" - Resolutions to existing buggy behavior (requires a patch version bump).
  • "Internal" - Changes which do not have an impact on the public interface or behavior of the module (requires a patch version bump).

While other section naming can be used when appropriate, it's important that are expressed clearly to avoid confusion for both the packages releaser and third-party consumers.

When in doubt, refer to Semantic Versioning specification.

If you are publishing new versions of packages, note that there are versioning recommendations outlined in the Gutenberg Release Process document which prescribe minimum version bumps for specific types of releases. The chosen version should be the greater of the two between the semantic versioning and Gutenberg release minimum version bumps.

Releasing Packages

Lerna automatically releases all outdated packages. To check which packages are outdated and will be released, type npm run publish:check from the branch that keeps the code for a given wp/X.Y release type (example wp/5.7).

If you have the ability to publish packages, you must have 2FA enabled on your npm account.

Before Releasing

Confirm that you're logged in to npm, by running npm whoami. If you're not logged in, run npm adduser to login.

If you're publishing a new package, ensure that its package.json file contains the correct publishConfig settings:

{
	"publishConfig": {
		"access": "public"
	}
}

You can check your package configs by running npm run lint-pkg-json.

Development Release

Run the following commands from the trunk branch to publish to npm (with a next dist tag) a development version of the packages.

npm install
./bin/plugin/cli.js npm-next

See more details in Development Releases section of the Gutenberg release process documentation.

Production Release

To release a production version for the outdated packages, run the following commands from the trunk branch:

npm install
./bin/plugin/cli.js npm-latest

See more details in Synchronizing WordPress Trunk section of the Gutenberg release process documentation.

Legacy Patch Release

To release a patch for the older major or minor version of packages, run the following commands from the corresponding wp/X.Y (example wp/5.7) release branch:

npm install
npm run publish:patch

This is usually necessary when adding bug fixes or security patches to the earlier versions of WordPress. This will publish only a patch version of the built packages. This is useful for backpublishing certain packages to WordPress.

See more details in Minor WordPress Releases section of the Gutenberg release process documentation.

TypeScript

The TypeScript language is a typed superset of JavaScript that compiles to plain JavaScript.
Gutenberg does not use the TypeScript language, however TypeScript has powerful tooling that can be applied to JavaScript projects.

Gutenberg uses TypeScript for several reasons, including:

  • Powerful editor integrations improve developer experience.
  • Type system can detect some issues and lead to more robust software.
  • Type declarations can be produced to allow other projects to benefit from these advantages as well.

Using TypeScript

Gutenberg uses TypeScript by running the TypeScript compiler (tsc) on select packages.
These packages benefit from type checking and produced type declarations in the published packages.

To opt-in to TypeScript tooling, packages should include a tsconfig.json file in the package root and add an entry to the root tsconfig.json references.
The changes will indicate that the package has opted-in and will be included in the TypeScript build process.

A tsconfig.json file should look like the following (comments are not necessary):

{
	// Extends a base configuration common to most packages
	"extends": "../../tsconfig.base.json",

	// Options for the TypeScript compiler
	// We'll usually set our `rootDir` and `declarationDir` as follows, which is specific
	// to each project.
	"compilerOptions": {
		"rootDir": "src",
		"declarationDir": "build-types"
	},

	// Which source files should be included
	"include": [ "src/**/*" ],

	// Other WordPress package dependencies that have opted-in to TypeScript should be listed
	// here. In this case, our package depends on `@wordpress/dom-ready`.
	"references": [ { "path": "../dom-ready" } ]
}

Type declarations will be produced in the build-types which should be included in the published package.
For consumers to use the published type declarations, we'll set the types field in package.json:

{
	"main": "build/index.js",
	"main-module": "build-module/index.js",
	"types": "build-types"
}

Ensure that the build-types directory will be included in the published package, for example if a files field is declared.

Optimizing for bundlers

In order for bundlers to tree-shake packages effectively, they often need to know whether a package includes side effects in its code. This is done through the sideEffects field in the package's package.json.

If your package has no side effects, simply set the field to false:

{
	"name": "package",
	"sideEffects": false
}

If your package includes a few files with side effects, you can list them instead:

{
	"name": "package",
	"sideEffects": [
		"file-with-side-effects.js",
		"another-file-with-side-effects.js"
	]
}

Please consult the side effects documentation for more information on identifying and declaring side effects.

Side effects

What are side effects?

Many @wordpress packages, such as UI-focused ones that register blocks or data stores, make use of side effects in their code. A side effect, in an ES module context, is code that performs some externally-visible behavior (that is, behavior which is visible outside the module) when the module is loaded.

Here is an example:

import { registerStore } from '@wordpress/data';

const store = registerStore( STORE_NAME, {
	// ...
} );

registerStore is being called at the top level, which means that it will run as soon as the module first gets imported. These changes are visible externally, since things are being modified in an external store, that lives in another module. Other examples of side effects include setting globals on window, or adding browser behavior through polyfills.

However, if this were to happen inside of an init function that doesn't get called on module load, then that would no longer be a side effect:

import { registerStore } from '@wordpress/data';

export function init() {
	const store = registerStore( STORE_NAME, {
		// ...
	} );
}

// `init` doesn't get called at the top level of the module,
// therefore importing the module doesn't cause side effects.

Declaring a variable or performing any modification at the top level that only affects the current module isn't a side effect either, since it's contained to the module:

import list from './list';

// Not a side effect.
let localVariable = [];
// Not a side effect, either.
for ( const entry of list ) {
	localVariable.push( processListEntry( entry ) );
}

The influence of side effects on bundling

Modern bundlers have the concept of tree-shaking, where unused code is removed from the final bundles, as it's not necessary. This becomes important in libraries that offer a lot of different functionality, since consumers of that library may only be using a small portion of it, and don't want their bundles to be larger than necessary.

These libraries should thus take steps to ensure they can indeed be correctly tree-shaken, and @wordpress packages are no exception.

This brings us back to side effects. As we've seen, side effects are code that runs simply by virtue of importing a module, and has an external influence of some sort. This means that the code cannot be tree-shaken away; it needs to run, because it changes things outside of the module that may be needed elsewhere.

Unfortunately, side effects are hard to determine automatically, and some bundlers err on the side of caution, assuming that every module potentially has side effects. This becomes a problem for index modules which re-export things from other modules, as that effectively means everything in there must now be bundled together:

// index.js

export { a, b } from './module1';
export { c, d, e } from './module2';
export { f } from './module3';

// Nothing can be tree-shaken away, because the bundler doesn't know if
// this or the re-exported modules have any side effects.

Telling bundlers about side effects

Since bundlers can't figure out side effects for themselves, we need to explicitly declare them. That's done in a package's package.json. For example, if a package has no side effects, it can simply set sideEffects to false:

{
	"name": "package",
	"sideEffects": false
}

If it has a few files with side effects, it can list them:

{
	"name": "package",
	"sideEffects": [ "dist/store/index.js", "dist/polyfill/index.js" ]
}

This allows the bundler to assume that only the modules that were declared have side effects, and nothing else does. Of course, this means that we need to be careful to include everything that does have side effects, or problems can arise in applications that make use of the package.

Storybook

Storybook is an open-source tool that provides a sandbox to develop and visualize components in isolation. See the Storybook site for more information about the tool.

The Gutenberg project uses Storybook to view and work with the UI components developed in the WordPress packages.

View online at: https://wordpress.github.io/gutenberg/

Run locally in your development environment running: npm run storybook:dev from the top-level Gutenberg directory.

@gziolo
Copy link
Member Author

gziolo commented Apr 13, 2021

It should be npx prettier --write **/*.md, sorry about that 🙈

@oandregal
Copy link
Member

oandregal commented Apr 15, 2021

Ok, took a look at this, my braindump:

  • Running npx prettier --write **/**/*.md gives a list of files modified. packages/i18n/CHANGELOG.md is unrelated to docgen, but the others are (the tools have different opinions about spacing and I learned that prettier formats code within markdown code blocks as well ― that's cool!).
  • I understand we want to introduce this formatting step automatically in CI and another tooling, hence, it's problematic that tools have different opinions.
  • A brute force approach is adding && prettier --write **/**/*.md to the docs:build npm script => if we use this in CI and other tools (haven't check it) everything will be fine.
  • A bit more sensible approach is to modify this to call prettier on modified files.

Does this help?

@gziolo
Copy link
Member Author

gziolo commented Apr 15, 2021

Thank you @nosolosw. It looks like we can collect updated files and run Prettier on all of them in one call or individually after each file gets updated 👍🏻

@gziolo
Copy link
Member Author

gziolo commented Sep 15, 2021

Now that #33498 and #33502 landed, the issue no longer exists 🎉

@gziolo gziolo closed this as completed Sep 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Good First Issue An issue that's suitable for someone looking to contribute for the first time Needs Dev Ready for, and needs developer efforts [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Build Tooling Issues or PRs related to build tooling [Type] Code Quality Issues or PRs that relate to code quality
Projects
None yet
Development

No branches or pull requests

2 participants