diff --git a/.gitignore b/.gitignore index 84617d1de..d54c8bae1 100644 --- a/.gitignore +++ b/.gitignore @@ -4,8 +4,6 @@ /coverage/ /dist/ /docs/.observablehq/dist/ -/docs/theme/*.md -/docs/themes.md /test/build/ /test/output/**/*-changed.* /test/output/build/**/*-changed/ diff --git a/docs/config.md b/docs/config.md index 1063f5a4e..d5174af75 100644 --- a/docs/config.md +++ b/docs/config.md @@ -155,6 +155,10 @@ The pages list should _not_ include the home page (`/`) as this is automatically Whether to show the previous & next links in the footer; defaults to true. The pages are linked in the same order as they appear in the sidebar. +## dynamicPaths + +The list of [parameterized pages](./params) and [dynamic pages](./page-loaders) to generate, either as a (synchronous) iterable of strings, or a function that returns an async iterable of strings if you wish to load the list of dynamic pages asynchronously. + ## head An HTML fragment to add to the head. Defaults to the empty string. If specified as a function, receives an object with the page’s `title`, (front-matter) `data`, and `path`, and must return a string. @@ -224,7 +228,7 @@ These additional results may also point to external links if the **path** is spe ```js run=false export default { search: { - async* index() { + async *index() { yield { path: "https://example.com", title: "Example", @@ -237,7 +241,7 @@ export default { ## interpreters -The **interpreters** option specifies additional interpreted languages for data loaders, indicating the file extension and associated interpreter. (See [loader routing](./loaders#routing) for more.) The default list of interpreters is: +The **interpreters** option specifies additional interpreted languages for data loaders, indicating the file extension and associated interpreter. (See [loader routing](./data-loaders#routing) for more.) The default list of interpreters is: ```js run=false { diff --git a/docs/convert.md b/docs/convert.md index db194acfb..354a52647 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -384,7 +384,7 @@ The `convert` command only supports code cell modes: Markdown, JavaScript, HTML, ## Databases -Database connectors can be replaced by [data loaders](./loaders). +Database connectors can be replaced by [data loaders](./data-loaders). ## Secrets diff --git a/docs/data-loaders.md b/docs/data-loaders.md new file mode 100644 index 000000000..50b482bd3 --- /dev/null +++ b/docs/data-loaders.md @@ -0,0 +1,324 @@ +--- +keywords: server-side rendering, ssr +--- + +# Data loaders + +**Data loaders** generate static snapshots of data during build. For example, a data loader might query a database and output CSV data, or server-side render a chart and output a PNG image. + +Why static snapshots? Performance is critical for dashboards: users don’t like to wait, and dashboards only create value if users look at them. Data loaders practically force your app to be fast because data is precomputed and thus can be served instantly — you don’t need to run queries separately for each user on load. Furthermore, data can be highly optimized (and aggregated and anonymized), minimizing what you send to the client. And since data loaders run only during build, your users don’t need direct access to your data warehouse, making your dashboards more secure and robust. + +
Data loaders are optional. You can use fetch or WebSocket if you prefer to load data at runtime, or you can store data in static files.
+
You can use continuous deployment to rebuild data as often as you like, ensuring that data is always up-to-date.
+ +Data loaders can be written in any programming language. They can even invoke binary executables such as ffmpeg or DuckDB. For convenience, Framework has built-in support for common languages: JavaScript, TypeScript, Python, and R. Naturally you can use any third-party library or SDK for these languages, too. + +A data loader can be as simple as a shell script that invokes [curl](https://curl.se/) to fetch recent earthquakes from the [USGS](https://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php): + +```sh +curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson +``` + +Data loaders use [file-based routing](#routing), so assuming this shell script is named `quakes.json.sh`, a `quakes.json` file is then generated at build time. You can access this file from the client using [`FileAttachment`](./files): + +```js echo +FileAttachment("quakes.json").json() +``` + +A data loader can transform data to perfectly suit the needs of a dashboard. The JavaScript data loader below uses [D3](./lib/d3) to output [CSV](./lib/csv) with three columns representing the _magnitude_, _longitude_, and _latitude_ of each earthquake. + +```js run=false echo +import {csvFormat} from "d3-dsv"; + +// Fetch GeoJSON from the USGS. +const response = await fetch("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson"); +if (!response.ok) throw new Error(`fetch failed: ${response.status}`); +const collection = await response.json(); + +// Convert to an array of objects. +const features = collection.features.map((f) => ({ + magnitude: f.properties.mag, + longitude: f.geometry.coordinates[0], + latitude: f.geometry.coordinates[1] +})); + +// Output CSV. +process.stdout.write(csvFormat(features)); +``` + +Assuming the loader above is named `quakes.csv.js`, you can access its output from the client as `quakes.csv`: + +```js echo +const quakes = FileAttachment("quakes.csv").csv({typed: true}); +``` + +Now you can display the earthquakes in a map using [Observable Plot](./lib/plot): + +```js +const world = await fetch(import.meta.resolve("npm:world-atlas/land-110m.json")).then((response) => response.json()); +const land = topojson.feature(world, world.objects.land); +``` + +```js echo +Plot.plot({ + projection: { + type: "orthographic", + rotate: [110, -30] + }, + marks: [ + Plot.graticule(), + Plot.sphere(), + Plot.geo(land, {stroke: "var(--theme-foreground-faint)"}), + Plot.dot(quakes, {x: "longitude", y: "latitude", r: "magnitude", stroke: "#f43f5e"}) + ] +}) +``` + +During preview, the preview server automatically runs the data loader the first time its output is needed and [caches](#caching) the result; if you edit the data loader, the preview server will automatically run it again and push the new result to the client. + +## Archives + +Data loaders can generate multi-file archives such as ZIP files; individual files can then be pulled from archives using `FileAttachment`. This allows a data loader to output multiple (often related) files from the same source data in one go. Framework also supports _implicit_ data loaders, _extractors_, that extract referenced files from static archives. So whether an archive is static or generated dynamically by a data loader, you can use `FileAttachment` to pull files from it. + +The following archive extensions are supported: + +- `.zip` - for the [ZIP]() archive format +- `.tar` - for [tarballs]() +- `.tar.gz` and `.tgz` - for [compressed tarballs](https://en.wikipedia.org/wiki/Gzip) + +Here’s an example of loading an image from `lib/muybridge.zip`: + +```js echo +FileAttachment("lib/muybridge/deer.jpeg").image({width: 320, alt: "A deer"}) +``` + +You can do the same with static HTML: + +A deer + +```html run=false +A deer +``` + +Below is a TypeScript data loader `quakes.zip.ts` that uses [JSZip](https://stuk.github.io/jszip/) to generate a ZIP archive of two files, `metadata.json` and `features.csv`. Note that the data loader is responsible for serializing the `metadata` and `features` objects to appropriate format corresponding to the file extension (`.json` and `.csv`); data loaders are responsible for doing their own serialization. + +```js run=false +import {csvFormat} from "d3-dsv"; +import JSZip from "jszip"; + +// Fetch GeoJSON from the USGS. +const response = await fetch("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson"); +if (!response.ok) throw new Error(`fetch failed: ${response.status}`); +const collection = await response.json(); + +// Convert to an array of objects. +const features = collection.features.map((f) => ({ + magnitude: f.properties.mag, + longitude: f.geometry.coordinates[0], + latitude: f.geometry.coordinates[1] +})); + +// Output a ZIP archive to stdout. +const zip = new JSZip(); +zip.file("metadata.json", JSON.stringify(collection.metadata, null, 2)); +zip.file("features.csv", csvFormat(features)); +zip.generateNodeStream().pipe(process.stdout); +``` + +To load data in the browser, use `FileAttachment`: + +```js run=false +const metadata = FileAttachment("quakes/metadata.json").json(); +const features = FileAttachment("quakes/features.csv").csv({typed: true}); +``` + +The ZIP file itself can be also referenced as a whole — for example if the names of the files are not known in advance — with [`file.zip`](./lib/zip): + +```js echo +const zip = FileAttachment("quakes.zip").zip(); +const metadata = zip.then((zip) => zip.file("metadata.json").json()); +``` + +Like with any other file, files from generated archives are live in preview (refreshing automatically if the corresponding data loader is edited), and are added to the build only if [statically referenced](./files#static-analysis) by `FileAttachment`. + +## Routing + +Data loaders live in the source root (typically `src`) alongside your other source files. When a file is referenced from JavaScript via `FileAttachment`, if the file does not exist, Framework will look for a file of the same name with a double extension to see if there is a corresponding data loader. By default, the following second extensions are checked, in order, with the corresponding language and interpreter: + +- `.js` - JavaScript (`node`) +- `.ts` - TypeScript (`tsx`) +- `.py` - Python (`python3`) +- `.R` - R (`Rscript`) +- `.rs` - Rust (`rust-script`) +- `.go` - Go (`go run`) +- `.java` — Java (`java`; requires Java 11+ and [single-file programs](https://openjdk.org/jeps/330)) +- `.jl` - Julia (`julia`) +- `.php` - PHP (`php`) +- `.sh` - shell script (`sh`) +- `.exe` - arbitrary executable + +
The interpreters configuration option can be used to extend the list of supported extensions.
+ +For example, for the file `quakes.csv`, the following data loaders are considered: `quakes.csv.js`, `quakes.csv.ts`, `quakes.csv.py`, _etc._ The first match is used. + +## Execution + +To use an interpreted data loader (anything other than `.exe`), the corresponding interpreter must be installed and available on your `$PATH`. Any additional modules, packages, libraries, _etc._, must also be installed. Some interpreters are not available on all platforms; for example `sh` is only available on Unix-like systems. + +
+ +You can use a virtual environment in Python, such as [venv](https://docs.python.org/3/tutorial/venv.html) or [uv](https://github.com/astral-sh/uv), to install libraries locally to the project. This is useful when working in multiple projects, and when collaborating; you can also track dependencies in a `requirements.txt` file. + +To create a virtual environment with venv: + +```sh +python3 -m venv .venv +``` + +Or with uv: + +```sh +uv venv +``` + +To activate the virtual environment on macOS or Linux: + +```sh +source .venv/bin/activate +``` + +Or on Windows: + +```sh +.venv\Scripts\activate +``` + +To install required packages: + +```sh +pip install -r requirements.txt +``` + +You can then run the `observable preview` or `observable build` (or `npm run dev` or `npm run build`) commands as usual; data loaders will run within the virtual environment. Run the `deactivate` command or use Control-D to exit the virtual environment. + +
+ +Data loaders are run in the same working directory in which you run the `observable build` or `observable preview` command, which is typically the project root. In Node, you can access the current working directory by calling `process.cwd()`, and the data loader’s source location with [`import.meta.url`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta). To compute the path of a file relative to the data loader source (rather than relative to the current working directory), use [`import.meta.resolve`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve). For example, a data loader in `src/summary.txt.js` could read the file `src/table.txt` as: + +```js run=false +import {readFile} from "node:fs/promises"; +import {fileURLToPath} from "node:url"; + +const table = await readFile(fileURLToPath(import.meta.resolve("./table.txt")), "utf-8"); +``` + +Executable (`.exe`) data loaders are run directly and must have the executable bit set. This is typically done via [`chmod`](https://en.wikipedia.org/wiki/Chmod). For example: + +```sh +chmod +x src/quakes.csv.exe +``` + +While a `.exe` data loader may be any binary executable (_e.g.,_ compiled from C), it is often convenient to specify another interpreter using a [shebang](). For example, to write a data loader in Perl: + +```perl +#!/usr/bin/env perl + +print("Hello World\n"); +``` + +If multiple requests are made concurrently for the same data loader, the data loader will only run once; each concurrent request will receive the same response. + +## Output + +Data loaders must output to [standard output](). The first extension (such as `.csv`) does not affect the generated snapshot; the data loader is solely responsible for producing the expected output (such as CSV). If you wish to log additional information from within a data loader, be sure to log to standard error, say by using [`console.warn`](https://developer.mozilla.org/en-US/docs/Web/API/console/warn) or `process.stderr`; otherwise the logs will be included in the output file and sent to the client. + +## Building + +Data loaders generate files at build time that live alongside other [static files](./files) in the `_file` directory of the output root. For example, to generate a `quakes.json` file at build time by fetching and caching data from the USGS, you could write a data loader in a shell script like so: + +```ini +. +├─ src +│ ├─ index.md +│ └─ quakes.json.sh +└─ ... +``` + +Where `quakes.json.sh` is: + +```sh +curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson +``` + +This will produce the following output root: + +```ini +. +├─ dist +│ ├─ _file +│ │ └─ quakes.99da78d9.json +│ ├─ _observablehq +│ │ └─ ... # additional assets for serving the site +│ └─ index.html +└─ ... +``` + +As another example, say you have a `quakes.zip` archive that includes yearly files for observed earthquakes. If you reference `FileAttachment("quakes/2021.csv")`, Framework will pull the `2021.csv` from `quakes.zip`. So this source root: + +```ini +. +├─ src +│ ├─ index.md +│ └─ quakes.zip +└─ ... +``` + +Becomes this output: + +```ini +. +├─ dist +│ ├─ _file +│ │ └─ quakes +│ │ └─ 2021.e5f2eb94.csv +│ ├─ _observablehq +│ │ └─ ... # additional assets for serving the site +│ └─ index.html +└─ ... +``` + +A data loader is only run during build if its corresponding output file is referenced in at least one page. Framework does not scour the source root (typically `src`) for data loaders. + +## Caching + +When a data loader runs successfully, its output is saved to a cache which lives in `.observablehq/cache` within the source root (typically `src`). + +During preview, Framework considers the cache “fresh” if the modification time of the cached output is newer than the modification time of the corresponding data loader source. If you edit a data loader or update its modification time with `touch`, the cache is invalidated; when previewing a page that uses the data loader, the preview server will detect that the data loader was modified and automatically run it, pushing the new data down to the client and re-evaluating any referencing code — no reload required! + +During build, Framework ignores modification times and only runs a data loader if its output is not cached. Continuous integration caches typically don’t preserve modification times, so this design makes it easier to control which data loaders to run by selectively populating the cache. + +To purge the data loader cache and force all data loaders to run on the next build, delete the entire cache. For example: + +```sh +rm -rf src/.observablehq/cache +``` + +To force a specific data loader to run on the next build instead, delete its corresponding output from the cache. For example, to rebuild `src/quakes.csv`: + +```sh +rm -f src/.observablehq/cache/quakes.csv +``` + +See [Automated deploys: Caching](./deploying#caching) for more on caching during CI. + +## Errors + +When a data loader fails, it _must_ return a non-zero [exit code](https://en.wikipedia.org/wiki/Exit_status). If a data loader produces a zero exit code, Framework will assume that it was successful and will cache and serve the output to the client. Empty output is not by itself considered an error; however, a warning is displayed in the preview server and build logs. + +During preview, data loader errors will be shown in the preview server log, and a 500 HTTP status code will be returned to the client that attempted to load the corresponding file. This typically results in an error such as: + +``` +RuntimeError: Unable to load file: quakes.csv +``` + +When any data loader fails, the entire build fails. diff --git a/docs/files.md b/docs/files.md index 8b66f2991..b019f577c 100644 --- a/docs/files.md +++ b/docs/files.md @@ -4,7 +4,7 @@ keywords: file, fileattachment, attachment # Files -Load files — whether static or generated dynamically by a [data loader](./loaders) — using the built-in `FileAttachment` function. This is available by default in Markdown, but you can import it explicitly like so: +Load files — whether static or generated dynamically by a [data loader](./data-loaders) — using the built-in `FileAttachment` function. This is available by default in Markdown, but you can import it explicitly like so: ```js echo import {FileAttachment} from "npm:@observablehq/stdlib"; @@ -16,7 +16,7 @@ The `FileAttachment` function takes a path and returns a file handle. This handl FileAttachment("volcano.json") ``` -Like a [local import](./imports#local-imports), the path is relative to the calling code’s source file: either the page’s Markdown file or the imported local JavaScript module. To load a remote file, use [`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API), or use a [data loader](./loaders) to download the file at build time. +Like a [local import](./imports#local-imports), the path is relative to the calling code’s source file: either the page’s Markdown file or the imported local JavaScript module. To load a remote file, use [`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API), or use a [data loader](./data-loaders) to download the file at build time. Calling `FileAttachment` doesn’t actually load the file; the contents are only loaded when you invoke a [file contents method](#supported-formats). For example, to load a JSON file: @@ -32,7 +32,7 @@ volcano ## Static analysis -The `FileAttachment` function can _only_ be passed a static string literal; constructing a dynamic path such as `FileAttachment("my" + "file.csv")` is invalid syntax. Static analysis is used to invoke [data loaders](./loaders) at build time, and ensures that only referenced files are included in the generated output during build. This also allows a content hash in the file name for cache breaking during deploy. +The `FileAttachment` function can _only_ be passed a static string literal; constructing a dynamic path such as FileAttachment(\`frame$\{i}.png\`) is invalid syntax. Static analysis is used to invoke [data loaders](./data-loaders) at build time, and ensures that only referenced files are included in the generated output during build. This also allows a content hash in the file name for cache breaking during deploy. If you have multiple files, you can enumerate them explicitly like so: diff --git a/docs/getting-started.md b/docs/getting-started.md index 1477753f1..568bd4e81 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -295,7 +295,7 @@ If this barfs a bunch of JSON in the terminal, it’s working as intended. 😅 ### File attachments -Framework uses [file-based routing](./loaders#routing) for data loaders: the data loader forecast.json.js serves the file forecast.json. To load this file from src/weather.md we use the relative path ./data/forecast.json. In effect, data loaders are simply a naming convention for generating “static” files — a big advantage of which is that you can edit a data loader and the changes immediately propagate to the live preview without needing a reload. +Framework uses [file-based routing](./data-loaders#routing) for data loaders: the data loader forecast.json.js serves the file forecast.json. To load this file from src/weather.md we use the relative path ./data/forecast.json. In effect, data loaders are simply a naming convention for generating “static” files — a big advantage of which is that you can edit a data loader and the changes immediately propagate to the live preview without needing a reload. To load a file in JavaScript, use the built-in [`FileAttachment`](./files). In `weather.md`, replace the contents of the JavaScript code block (the parts inside the triple backticks ```) with the following code: @@ -557,7 +557,7 @@ forecast = requests.get(station["properties"]["forecastHourly"]).json() json.dump(forecast, sys.stdout) ``` -To write the data loader in R, name it forecast.json.R. Or as shell script, forecast.json.sh. You get the idea. See [Data loaders: Routing](./loaders#routing) for more. The beauty of this approach is that you can leverage the strengths (and libraries) of multiple languages, and still get instant updates in the browser as you develop. +To write the data loader in R, name it forecast.json.R. Or as shell script, forecast.json.sh. You get the idea. See [Data loaders: Routing](./data-loaders#routing) for more. The beauty of this approach is that you can leverage the strengths (and libraries) of multiple languages, and still get instant updates in the browser as you develop. ### Deploy automatically diff --git a/docs/index.md b/docs/index.md index de4077c5c..0d64abb44 100644 --- a/docs/index.md +++ b/docs/index.md @@ -141,7 +141,7 @@ index: false **Observable Framework** is an [open-source](https://github.com/observablehq/framework) static site generator for data apps, dashboards, reports, and more. Framework includes a preview server for local development, and a command-line interface for automating builds & deploys. -You write simple [Markdown](./markdown) pages — with interactive charts and inputs in [reactive JavaScript](./javascript), and with data snapshots generated by [loaders](./loaders) in _any_ programming language (SQL, Python, R, and more) — and Framework compiles it into a static site with instant page loads for a great user experience. Since everything is just files, you can use your preferred editor and source control, write unit tests, share code with other apps, integrate with CI/CD, and host projects anywhere. +You write simple [Markdown](./markdown) pages — with interactive charts and inputs in [reactive JavaScript](./javascript), and with data snapshots generated by [data loaders](./data-loaders) in _any_ programming language (SQL, Python, R, and more) — and Framework compiles it into a static site with instant page loads for a great user experience. Since everything is just files, you can use your preferred editor and source control, write unit tests, share code with other apps, integrate with CI/CD, and host projects anywhere. Framework includes thoughtfully-designed [themes](./themes), [grids](./markdown#grids), and [libraries](./imports) to help you build displays of data that look great on any device, including [Observable Plot](./lib/plot), [D3](./lib/d3), [Mosaic](./lib/mosaic), [Vega-Lite](./lib/vega-lite), [Graphviz](./lib/dot), [Mermaid](./lib/mermaid), [Leaflet](./lib/leaflet), [KaTeX](./lib/tex), and myriad more. And for working with data in the client, there’s [DuckDB](./lib/duckdb), [Arquero](./lib/arquero), [SQLite](./lib/sqlite), and more, too. diff --git a/docs/lib/mosaic.md b/docs/lib/mosaic.md index 6b89cc25f..f8b159ad5 100644 --- a/docs/lib/mosaic.md +++ b/docs/lib/mosaic.md @@ -7,7 +7,7 @@ sql: [Mosaic](https://uwdata.github.io/mosaic/) is a system for linking data visualizations, tables, and inputs, leveraging [DuckDB](./duckdb) for scalable processing. Mosaic includes an interactive grammar of graphics, [Mosaic vgplot](https://uwdata.github.io/mosaic/vgplot/), built on [Observable Plot](./plot). With vgplot, you can interactively visualize and explore millions — even billions — of data points. -The example below shows the pickup and dropoff locations of one million taxi rides in New York City from Jan 1–3, 2010. The dataset is stored in a 8MB [Apache Parquet](./arrow#apache-parquet) file, generated with a [data loader](../loaders). +The example below shows the pickup and dropoff locations of one million taxi rides in New York City from Jan 1–3, 2010. The dataset is stored in a 8MB [Apache Parquet](./arrow#apache-parquet) file, generated with a [data loader](../data-loaders). ${maps} diff --git a/docs/lib/sqlite.md b/docs/lib/sqlite.md index 722f5419f..b5259410d 100644 --- a/docs/lib/sqlite.md +++ b/docs/lib/sqlite.md @@ -26,7 +26,7 @@ const db = SQLiteDatabaseClient.open(FileAttachment("chinook.db")); (Note that unlike [`DuckDBClient`](./duckdb), a `SQLiteDatabaseClient` takes a single argument representing _all_ of the tables in the database; that’s because a SQLite file stores multiple tables, whereas DuckDB typically uses separate Apache Parquet, CSV, or JSON files for each table.) -Using `FileAttachment` means that referenced files are automatically copied to `dist` during build, and you can even generate SQLite files using [data loaders](../loaders). But if you want to “hot” load a live file from an external server, pass a string to `SQLiteDatabaseClient.open`: +Using `FileAttachment` means that referenced files are automatically copied to `dist` during build, and you can even generate SQLite files using [data loaders](../data-loaders). But if you want to “hot” load a live file from an external server, pass a string to `SQLiteDatabaseClient.open`: ```js run=false const db = SQLiteDatabaseClient.open("https://static.observableusercontent.com/files/b3711cfd9bdf50cbe4e74751164d28e907ce366cd4bf56a39a980a48fdc5f998c42a019716a8033e2b54defdd97e4a55ebe4f6464b4f0678ea0311532605a115"); diff --git a/docs/lib/zip.md b/docs/lib/zip.md index 92a6052c2..4d984af95 100644 --- a/docs/lib/zip.md +++ b/docs/lib/zip.md @@ -24,7 +24,7 @@ To pull out a single file from the archive, use the `archive.file` method. It re muybridge.file("deer.jpeg").image({width: 320, alt: "A deer"}) ``` -That said, if you know the name of the file within the ZIP archive statically, you don’t need to load the ZIP archive; you can simply request the [file within the archive](../loaders#archives) directly. The specified file is then extracted from the ZIP archive at build time. +That said, if you know the name of the file within the ZIP archive statically, you don’t need to load the ZIP archive; you can simply request the [file within the archive](../data-loaders#archives) directly. The specified file is then extracted from the ZIP archive at build time. ```js echo FileAttachment("muybridge/deer.jpeg").image({width: 320, alt: "A deer"}) @@ -38,7 +38,7 @@ For images and other media, you can simply use static HTML. A deer ``` -One reason to load a ZIP archive is that you don’t know the files statically — maybe there are lots of files and you don’t want to enumerate them statically, or maybe you expect them to change over time and the ZIP archive is generated by a [data loader](../loaders). For example, maybe you want to display an arbitrary collection of images. +One reason to load a ZIP archive is that you don’t know the files statically — maybe there are lots of files and you don’t want to enumerate them statically, or maybe you expect them to change over time and the ZIP archive is generated by a [data loader](../data-loaders). For example, maybe you want to display an arbitrary collection of images. ```js echo Gallery(await Promise.all(muybridge.filenames.map((f) => muybridge.file(f).image()))) diff --git a/docs/loaders.md b/docs/loaders.md index aeb7f33e8..75bb3ef78 100644 --- a/docs/loaders.md +++ b/docs/loaders.md @@ -1,320 +1,3 @@ -# Data loaders + -**Data loaders** generate static snapshots of data during build. For example, a data loader might query a database and output CSV data, or server-side render a chart and output a PNG image. - -Why static snapshots? Performance is critical for dashboards: users don’t like to wait, and dashboards only create value if users look at them. Data loaders practically force your app to be fast because data is precomputed and thus can be served instantly — you don’t need to run queries separately for each user on load. Furthermore, data can be highly optimized (and aggregated and anonymized), minimizing what you send to the client. And since data loaders run only during build, your users don’t need direct access to your data warehouse, making your dashboards more secure and robust. - -
Data loaders are optional. You can use fetch or WebSocket if you prefer to load data at runtime, or you can store data in static files.
-
You can use continuous deployment to rebuild data as often as you like, ensuring that data is always up-to-date.
- -Data loaders can be written in any programming language. They can even invoke binary executables such as ffmpeg or DuckDB. For convenience, Framework has built-in support for common languages: JavaScript, TypeScript, Python, and R. Naturally you can use any third-party library or SDK for these languages, too. - -A data loader can be as simple as a shell script that invokes [curl](https://curl.se/) to fetch recent earthquakes from the [USGS](https://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php): - -```sh -curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson -``` - -Data loaders use [file-based routing](#routing), so assuming this shell script is named `quakes.json.sh`, a `quakes.json` file is then generated at build time. You can access this file from the client using [`FileAttachment`](./files): - -```js echo -FileAttachment("quakes.json").json() -``` - -A data loader can transform data to perfectly suit the needs of a dashboard. The JavaScript data loader below uses [D3](./lib/d3) to output [CSV](./lib/csv) with three columns representing the _magnitude_, _longitude_, and _latitude_ of each earthquake. - -```js run=false echo -import {csvFormat} from "d3-dsv"; - -// Fetch GeoJSON from the USGS. -const response = await fetch("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson"); -if (!response.ok) throw new Error(`fetch failed: ${response.status}`); -const collection = await response.json(); - -// Convert to an array of objects. -const features = collection.features.map((f) => ({ - magnitude: f.properties.mag, - longitude: f.geometry.coordinates[0], - latitude: f.geometry.coordinates[1] -})); - -// Output CSV. -process.stdout.write(csvFormat(features)); -``` - -Assuming the loader above is named `quakes.csv.js`, you can access its output from the client as `quakes.csv`: - -```js echo -const quakes = FileAttachment("quakes.csv").csv({typed: true}); -``` - -Now you can display the earthquakes in a map using [Observable Plot](./lib/plot): - -```js -const world = await fetch(import.meta.resolve("npm:world-atlas/land-110m.json")).then((response) => response.json()); -const land = topojson.feature(world, world.objects.land); -``` - -```js echo -Plot.plot({ - projection: { - type: "orthographic", - rotate: [110, -30] - }, - marks: [ - Plot.graticule(), - Plot.sphere(), - Plot.geo(land, {stroke: "var(--theme-foreground-faint)"}), - Plot.dot(quakes, {x: "longitude", y: "latitude", r: "magnitude", stroke: "#f43f5e"}) - ] -}) -``` - -During preview, the preview server automatically runs the data loader the first time its output is needed and [caches](#caching) the result; if you edit the data loader, the preview server will automatically run it again and push the new result to the client. - -## Archives - -Data loaders can generate multi-file archives such as ZIP files; individual files can then be pulled from archives using `FileAttachment`. This allows a data loader to output multiple (often related) files from the same source data in one go. Framework also supports _implicit_ data loaders, _extractors_, that extract referenced files from static archives. So whether an archive is static or generated dynamically by a data loader, you can use `FileAttachment` to pull files from it. - -The following archive extensions are supported: - -- `.zip` - for the [ZIP]() archive format -- `.tar` - for [tarballs]() -- `.tar.gz` and `.tgz` - for [compressed tarballs](https://en.wikipedia.org/wiki/Gzip) - -Here’s an example of loading an image from `lib/muybridge.zip`: - -```js echo -FileAttachment("lib/muybridge/deer.jpeg").image({width: 320, alt: "A deer"}) -``` - -You can do the same with static HTML: - -A deer - -```html run=false -A deer -``` - -Below is a TypeScript data loader `quakes.zip.ts` that uses [JSZip](https://stuk.github.io/jszip/) to generate a ZIP archive of two files, `metadata.json` and `features.csv`. Note that the data loader is responsible for serializing the `metadata` and `features` objects to appropriate format corresponding to the file extension (`.json` and `.csv`); data loaders are responsible for doing their own serialization. - -```js run=false -import {csvFormat} from "d3-dsv"; -import JSZip from "jszip"; - -// Fetch GeoJSON from the USGS. -const response = await fetch("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson"); -if (!response.ok) throw new Error(`fetch failed: ${response.status}`); -const collection = await response.json(); - -// Convert to an array of objects. -const features = collection.features.map((f) => ({ - magnitude: f.properties.mag, - longitude: f.geometry.coordinates[0], - latitude: f.geometry.coordinates[1] -})); - -// Output a ZIP archive to stdout. -const zip = new JSZip(); -zip.file("metadata.json", JSON.stringify(collection.metadata, null, 2)); -zip.file("features.csv", csvFormat(features)); -zip.generateNodeStream().pipe(process.stdout); -``` - -To load data in the browser, use `FileAttachment`: - -```js run=false -const metadata = FileAttachment("quakes/metadata.json").json(); -const features = FileAttachment("quakes/features.csv").csv({typed: true}); -``` - -The ZIP file itself can be also referenced as a whole — for example if the names of the files are not known in advance — with [`file.zip`](./lib/zip): - -```js echo -const zip = FileAttachment("quakes.zip").zip(); -const metadata = zip.then((zip) => zip.file("metadata.json").json()); -``` - -Like with any other file, files from generated archives are live in preview (refreshing automatically if the corresponding data loader is edited), and are added to the build only if [statically referenced](./files#static-analysis) by `FileAttachment`. - -## Routing - -Data loaders live in the source root (typically `src`) alongside your other source files. When a file is referenced from JavaScript via `FileAttachment`, if the file does not exist, Framework will look for a file of the same name with a double extension to see if there is a corresponding data loader. By default, the following second extensions are checked, in order, with the corresponding language and interpreter: - -- `.js` - JavaScript (`node`) -- `.ts` - TypeScript (`tsx`) -- `.py` - Python (`python3`) -- `.R` - R (`Rscript`) -- `.rs` - Rust (`rust-script`) -- `.go` - Go (`go run`) -- `.java` — Java (`java`; requires Java 11+ and [single-file programs](https://openjdk.org/jeps/330)) -- `.jl` - Julia (`julia`) -- `.php` - PHP (`php`) -- `.sh` - shell script (`sh`) -- `.exe` - arbitrary executable - -
The interpreters configuration option can be used to extend the list of supported extensions.
- -For example, for the file `quakes.csv`, the following data loaders are considered: `quakes.csv.js`, `quakes.csv.ts`, `quakes.csv.py`, _etc._ The first match is used. - -## Execution - -To use an interpreted data loader (anything other than `.exe`), the corresponding interpreter must be installed and available on your `$PATH`. Any additional modules, packages, libraries, _etc._, must also be installed. Some interpreters are not available on all platforms; for example `sh` is only available on Unix-like systems. - -
- -You can use a virtual environment in Python, such as [venv](https://docs.python.org/3/tutorial/venv.html) or [uv](https://github.com/astral-sh/uv), to install libraries locally to the project. This is useful when working in multiple projects, and when collaborating; you can also track dependencies in a `requirements.txt` file. - -To create a virtual environment with venv: - -```sh -python3 -m venv .venv -``` - -Or with uv: - -```sh -uv venv -``` - -To activate the virtual environment on macOS or Linux: - -```sh -source .venv/bin/activate -``` - -Or on Windows: - -```sh -.venv\Scripts\activate -``` - -To install required packages: - -```sh -pip install -r requirements.txt -``` - -You can then run the `observable preview` or `observable build` (or `npm run dev` or `npm run build`) commands as usual; data loaders will run within the virtual environment. Run the `deactivate` command or use Control-D to exit the virtual environment. - -
- -Data loaders are run in the same working directory in which you run the `observable build` or `observable preview` command, which is typically the project root. In Node, you can access the current working directory by calling `process.cwd()`, and the data loader’s source location with [`import.meta.url`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta). To compute the path of a file relative to the data loader source (rather than relative to the current working directory), use [`import.meta.resolve`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve). For example, a data loader in `src/summary.txt.js` could read the file `src/table.txt` as: - -```js run=false -import {readFile} from "node:fs/promises"; -import {fileURLToPath} from "node:url"; - -const table = await readFile(fileURLToPath(import.meta.resolve("./table.txt")), "utf-8"); -``` - -Executable (`.exe`) data loaders are run directly and must have the executable bit set. This is typically done via [`chmod`](https://en.wikipedia.org/wiki/Chmod). For example: - -```sh -chmod +x src/quakes.csv.exe -``` - -While a `.exe` data loader may be any binary executable (_e.g.,_ compiled from C), it is often convenient to specify another interpreter using a [shebang](). For example, to write a data loader in Perl: - -```perl -#!/usr/bin/env perl - -print("Hello World\n"); -``` - -If multiple requests are made concurrently for the same data loader, the data loader will only run once; each concurrent request will receive the same response. - -## Output - -Data loaders must output to [standard output](). The first extension (such as `.csv`) does not affect the generated snapshot; the data loader is solely responsible for producing the expected output (such as CSV). If you wish to log additional information from within a data loader, be sure to log to stderr, say by using [`console.warn`](https://developer.mozilla.org/en-US/docs/Web/API/console/warn); otherwise the logs will be included in the output file and sent to the client. - -## Building - -Data loaders generate files at build time that live alongside other [static files](./files) in the `_file` directory of the output root. For example, to generate a `quakes.json` file at build time by fetching and caching data from the USGS, you could write a data loader in a shell script like so: - -```ini -. -├─ src -│ ├─ index.md -│ └─ quakes.json.sh -└─ ... -``` - -Where `quakes.json.sh` is: - -```sh -curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson -``` - -This will produce the following output root: - -```ini -. -├─ dist -│ ├─ _file -│ │ └─ quakes.99da78d9.json -│ ├─ _observablehq -│ │ └─ ... # additional assets for serving the site -│ └─ index.html -└─ ... -``` - -As another example, say you have a `quakes.zip` archive that includes yearly files for observed earthquakes. If you reference `FileAttachment("quakes/2021.csv")`, Framework will pull the `2021.csv` from `quakes.zip`. So this source root: - -```ini -. -├─ src -│ ├─ index.md -│ └─ quakes.zip -└─ ... -``` - -Becomes this output: - -```ini -. -├─ dist -│ ├─ _file -│ │ └─ quakes -│ │ └─ 2021.e5f2eb94.csv -│ ├─ _observablehq -│ │ └─ ... # additional assets for serving the site -│ └─ index.html -└─ ... -``` - -A data loader is only run during build if its corresponding output file is referenced in at least one page. Framework does not scour the source root (typically `src`) for data loaders. - -## Caching - -When a data loader runs successfully, its output is saved to a cache which lives in `.observablehq/cache` within the source root (typically `src`). - -During preview, Framework considers the cache “fresh” if the modification time of the cached output is newer than the modification time of the corresponding data loader source. If you edit a data loader or update its modification time with `touch`, the cache is invalidated; when previewing a page that uses the data loader, the preview server will detect that the data loader was modified and automatically run it, pushing the new data down to the client and re-evaluating any referencing code — no reload required! - -During build, Framework ignores modification times and only runs a data loader if its output is not cached. Continuous integration caches typically don’t preserve modification times, so this design makes it easier to control which data loaders to run by selectively populating the cache. - -To purge the data loader cache and force all data loaders to run on the next build, delete the entire cache. For example: - -```sh -rm -rf src/.observablehq/cache -``` - -To force a specific data loader to run on the next build instead, delete its corresponding output from the cache. For example, to rebuild `src/quakes.csv`: - -```sh -rm -f src/.observablehq/cache/quakes.csv -``` - -See [Automated deploys: Caching](./deploying#caching) for more on caching during CI. - -## Errors - -When a data loader fails, it _must_ return a non-zero [exit code](https://en.wikipedia.org/wiki/Exit_status). If a data loader produces a zero exit code, Framework will assume that it was successful and will cache and serve the output to the client. Empty output is not by itself considered an error; however, a warning is displayed in the preview server and build logs. - -During preview, data loader errors will be shown in the preview server log, and a 500 HTTP status code will be returned to the client that attempted to load the corresponding file. This typically results in an error such as: - -``` -RuntimeError: Unable to load file: quakes.csv -``` - -When any data loader fails, the entire build fails. +Moved to [Data loaders](./data-loaders). diff --git a/docs/page-loaders.md.js b/docs/page-loaders.md.js new file mode 100644 index 000000000..7006c85d9 --- /dev/null +++ b/docs/page-loaders.md.js @@ -0,0 +1,60 @@ +process.stdout.write(`--- +keywords: server-side rendering, ssr +--- + +# Page loaders + +Page loaders are a special type of [data loader](./data-loaders) for dynamically generating (or “server-side rendering”) pages. Page loaders are programs that emit [Markdown](./markdown) to standard out, and have a double extension starting with \`.md\`, such as \`.md.js\` for a JavaScript page loader or \`.md.py\` for a Python page loader. + +By “baking” dynamically-generated content into static Markdown, you can further improve the performance of pages since the content exists on page load rather than waiting for JavaScript to run. You may even be able to avoid loading additional assets and JavaScript libraries. + +For example, to render a map of recent earthquakes into static inline SVG using D3, you could use a JavaScript page loader as \`quakes.md.js\`: + +~~~js run=false +import * as d3 from "d3-geo"; +import * as topojson from "topojson-client"; + +const quakes = await (await fetch("https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson")).json(); +const world = await (await fetch("https://cdn.jsdelivr.net/npm/world-atlas@2.0.2/land-110m.json")).json(); +const land = topojson.feature(world, world.objects.land); + +const projection = d3.geoOrthographic().rotate([110, -40]).fitExtent([[2, 2], [638, 638]], {type: "Sphere"}); +const path = d3.geoPath(projection); + +process.stdout.write(\`# Recent quakes + + + + + + + +\`); +~~~ + +See the [data loaders](./data-loaders) documentation for more on execution, routing, and caching. + +
+ +Page loaders often use [parameterized routes](./params) to generate multiple pages from a single program. + +
+ +
+ +When using page loaders, keep an eye on the generated page size, particularly with complex maps and data visualizations in SVG. To keep the page size small, consider server-side rendering a low-fidelity placeholder and then replacing it with the full graphic using JavaScript on the client. + +
+ +
+ +To allow importing of a JavaScript page loader without running it, have the page loader check whether \`process.argv[1]\` is the same as \`import.meta.url\` before running: + +~~~js run=false +if (process.argv[1] === fileURLToPath(import.meta.url)) { + process.stdout.write(\`# Hello page\`); +} +~~~ + +
+`); diff --git a/docs/params.md b/docs/params.md new file mode 100644 index 000000000..c3f907043 --- /dev/null +++ b/docs/params.md @@ -0,0 +1,157 @@ +# Parameterized routes + +Parameterized routes allow a single [Markdown](./markdown) source file or [page loader](./page-loaders) to generate many pages, or a single [data loader](./data-loaders) to generate many files. + +A parameterized route is denoted by square brackets, such as `[param]`, in a file or directory name. For example, the following project structure could be used to generate a page for many products: + +``` +. +├─ src +│ ├─ index.md +│ └─ products +│ └─ [product].md +└─ ⋯ +``` + +(File and directory names can also be partially parameterized such as `prefix-[param].md` or `[param]-suffix.md`, or contain multiple parameters such as `[year]-[month]-[day].md`.) + +The [**dynamicPaths** config option](./config#dynamicPaths) would then specify the list of product pages: + +```js run=false +export default { + dynamicPaths: [ + "/products/100736", + "/products/221797", + "/products/399145", + "/products/475651", + … + ] +}; +``` + +Rather than hard-coding the list of paths as above, you’d more commonly use code to enumerate them, say by querying a database for products. In this case, you can either use [top-level await](https://v8.dev/features/top-level-await) or specify the **dynamicPaths** config option as a function that returns an [async iterable](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_async_iterator_and_async_iterable_protocols). For example, using [Postgres.js](https://github.com/porsager/postgres/blob/master/README.md#usage) you might say: + +```js run=false +import postgres from "postgres"; + +const sql = postgres(); // Note: uses psql environment variables + +export default { + async *dynamicPaths() { + for await (const {id} of sql`SELECT id FROM products`.cursor()) { + yield `/products/${id}`; + } + } +}; +``` + +## Params in JavaScript + +Within a parameterized page, observable.params.param exposes the value of the parameter param to JavaScript [fenced code blocks](./javascript#fenced-code-blocks) and [inline expressions](./javascript#inline-expressions), and likewise for any imported [local modules](./imports#local-imports) with parameterized routes. For example, to display the value of the `product` parameter in Markdown: + +```md run=false +The current product is ${observable.params.product}. +``` + +Since parameter values are known statically at build time, you can reference parameter values in calls to `FileAttachment`. For example, to load the JSON file `/products/[product].json` for the corresponding product from the page `/products/[product].md`, you could say: + +```js run=false +const info = FileAttachment(`${observable.params.product}.json`).json(); +``` + +This is an exception: otherwise `FileAttachment` only accepts a static string literal as an argument since Framework uses [static analysis](./files#static-analysis) to find referenced files. If you need more flexibility, consider using a [page loader](./page-loaders) to generate the page. + +## Params in data loaders + +Parameter values are passed as command-line arguments such as `--product=42` to parameterized [data loaders](./data-loaders). In a JavaScript data loader, you can use [`parseArgs`](https://nodejs.org/api/util.html#utilparseargsconfig) from `node:util` to parse command-line arguments. + +For example, here is a parameterized data loader `sales-[product].csv.js` that generates a CSV of daily sales totals for a particular product by querying a PostgreSQL database: + +```js run=false +import {parseArgs} from "node:util"; +import {csvFormat} from "d3-dsv"; +import postgres from "postgres"; + +const sql = postgres(); // Note: uses psql environment variables + +const { + values: {product} +} = parseArgs({ + options: {product: {type: "string"}} +}); + +const sales = await sql` + SELECT + DATE(sale_date) AS sale_day, + SUM(quantity) AS total_quantity_sold, + SUM(total_amount) AS total_sales_amount + FROM + sales + WHERE + product_id = ${product} + GROUP BY + DATE(sale_date) + ORDER BY + sale_day +`; + +process.stdout.write(csvFormat(sales)); + +await sql.end(); +``` + +Using the above data loader, you could then load `sales-42.csv` to get the daily sales data for product 42. + +## Params in page loaders + +As with data loaders, parameter values are passed as command-line arguments such as `--product=42` to parameterized [page loaders](./page-loaders). In a JavaScript page loader, you can use [`parseArgs`](https://nodejs.org/api/util.html#utilparseargsconfig) from `node:util` to parse command-line arguments. You can then bake parameter values into the resulting page code, or reference them dynamically [in client-side JavaScript](#params-in-java-script) using `observable.params`. + +For example, here is a parameterized page loader `sales-[product].md.js` that renders a chart with daily sales numbers for a particular product, loading the data from the parameterized data loader `sales-[product].csv.js` shown above: + +~~~~js run=false +import {parseArgs} from "node:util"; + +const { + values: {product} +} = parseArgs({ + options: {product: {type: "string"}} +}); + +process.stdout.write(`# Sales of product ${product} + +~~~js +const sales = FileAttachment(\`sales-${product}.csv\`).csv({typed: true}); +~~~ + +~~~js +Plot.plot({ + x: {interval: "day", label: null}, + y: {grid: true}, + marks: [ + Plot.barY(sales, {x: "sale_day", y: "total_sales_amount", tip: true}), + Plot.ruleY([0]) + ] +}) +~~~ + +`); +~~~~ + +In a page generated by a JavaScript page loader, you typically don’t reference `observable.params`; instead, bake the current parameter values directly into the generated code. (You can still reference `observable.params` in the generated client-side JavaScript if you want to.) Framework’s [theme previews](./themes) are implemented as parameterized page loaders; see [their source](https://github.com/observablehq/framework/blob/main/docs/theme/%5Btheme%5D.md.ts) for a practical example. + +## Precedence + +If multiple sources match a particular route, Framework choses the most-specific match. Exact matches are preferred over parameterized matches, and higher directories (closer to the root) are given priority over lower directories. + +For example, for the page `/product/42`, the following sources might be considered: + +* `/product/42.md` (exact match on static file) +* `/product/42.md.js` (exact match on page loader) +* `/product/[id].md` (parameterized static file) +* `/product/[id].md.js` (parameterized page loader) +* `/[category]/42.md` (static file in parameterized directory) +* `/[category]/42.md.js` (page loader in parameterized directory) +* `/[category]/[product].md` (etc.) +* `/[category]/[product].md.js` + +(For brevity, only JavaScript page loaders are shown above; in practice Framework will consider all registered interpreters when checking for page loaders. [Archive data loaders](./data-loaders#archives) are also not shown.) diff --git a/docs/project-structure.md b/docs/project-structure.md index 7b4000655..bd535c872 100644 --- a/docs/project-structure.md +++ b/docs/project-structure.md @@ -38,7 +38,7 @@ This is the “source root” — where your source files live. It doesn’t hav #### `src/.observablehq/cache` -This is where the [data loader](./loaders) cache lives. You don’t typically have to worry about this since it’s autogenerated when the first data loader is referenced. You can `rm -rf src/.observablehq/cache` to clean the cache and force data loaders to re-run. +This is where the [data loader](./data-loaders) cache lives. You don’t typically have to worry about this since it’s autogenerated when the first data loader is referenced. You can `rm -rf src/.observablehq/cache` to clean the cache and force data loaders to re-run. #### `src/.observablehq/deploy.json` @@ -50,7 +50,7 @@ You can put shared [JavaScript modules](./imports) anywhere in your source root, #### `src/data` -You can put [data loaders](./loaders) or static files anywhere in your source root, but we recommend putting them here. +You can put [data loaders](./data-loaders) or static files anywhere in your source root, but we recommend putting them here. #### `src/index.md` @@ -69,10 +69,10 @@ Framework uses file-based routing: each page in your project has a corresponding ├─ src │ ├─ hello.md │ └─ index.md -└─ ... +└─ ⋯ ``` - + When the site is built, the output root (`dist`) will contain two corresponding static HTML pages (`hello.html` and `index.html`), along with a few additional assets needed for the site to work. @@ -80,12 +80,18 @@ When the site is built, the output root (`dist`) will contain two corresponding . ├─ dist │ ├─ _observablehq -│ │ └─ ... # additional assets for serving the site +│ │ └─ ⋯ # additional assets for serving the site │ ├─ hello.html │ └─ index.html -└─ ... +└─ ⋯ ``` +
+ +While normally a Markdown file generates only a single page, Framework also supports [parameterized pages](./params) (also called _dynamic routes_), allowing a Markdown file to generate many pages with different data. + +
+ For this site, routes map to files as: ``` @@ -103,11 +109,11 @@ Pages can live in folders. For example: . ├─ src │ ├─ missions -| | ├─ index.md -| | ├─ apollo.md +│ │ ├─ index.md +│ │ ├─ apollo.md │ │ └─ gemini.md │ └─ index.md -└─ ... +└─ ⋯ ``` With this setup, routes are served as: @@ -125,11 +131,11 @@ As a variant of the above structure, you can move the `missions/index.md` up to . ├─ src │ ├─ missions -| | ├─ apollo.md +│ │ ├─ apollo.md │ │ └─ gemini.md │ ├─ missions.md │ └─ index.md -└─ ... +└─ ⋯ ``` Now routes are served as: diff --git a/docs/sql.md b/docs/sql.md index 86167f608..05361025e 100644 --- a/docs/sql.md +++ b/docs/sql.md @@ -5,9 +5,9 @@ sql: # SQL -
This page covers client-side SQL using DuckDB. To run a SQL query on a remote database such as PostgreSQL or Snowflake, use a data loader.
+
This page covers client-side SQL using DuckDB. To run a SQL query on a remote database such as PostgreSQL or Snowflake, use a data loader.
-Framework includes built-in support for client-side SQL powered by [DuckDB](./lib/duckdb). You can use SQL to query data from [CSV](./lib/csv), [TSV](./lib/csv), [JSON](./files#json), [Apache Arrow](./lib/arrow), [Apache Parquet](./lib/arrow#apache-parquet), and DuckDB database files, which can either be static or generated by [data loaders](./loaders). +Framework includes built-in support for client-side SQL powered by [DuckDB](./lib/duckdb). You can use SQL to query data from [CSV](./lib/csv), [TSV](./lib/csv), [JSON](./files#json), [Apache Arrow](./lib/arrow), [Apache Parquet](./lib/arrow#apache-parquet), and DuckDB database files, which can either be static or generated by [data loaders](./data-loaders). To use SQL, first register the desired tables in the page’s [front matter](./markdown#front-matter) using the **sql** option. Each key is a table name, and each value is the path to the corresponding data file. For example, to register a table named `gaia` from a Parquet file: @@ -27,7 +27,7 @@ sql: --- ``` -
For performance and reliability, we recommend using local files rather than loading data from external servers at runtime. You can use a data loader to take a snapshot of a remote data during build if needed.
+
For performance and reliability, we recommend using local files rather than loading data from external servers at runtime. You can use a data loader to take a snapshot of a remote data during build if needed.
You can also register tables via code (say to have sources that are defined dynamically via user input) by defining the `sql` symbol with [DuckDBClient.sql](./lib/duckdb). diff --git a/docs/theme/generate-themes.ts b/docs/theme/generate-themes.ts deleted file mode 100644 index 4f3a7e700..000000000 --- a/docs/theme/generate-themes.ts +++ /dev/null @@ -1,25 +0,0 @@ -import {writeFile} from "node:fs/promises"; -import {faint} from "../../src/tty.js"; -import renderIndex, {themes} from "../themes.md.js"; -import renderTheme from "./[theme].md.js"; - -async function generateFile(path: string, contents: string): Promise { - console.log(`${faint("generating")} ${path}`); - await writeFile(path, contents); -} - -await generateFile("./docs/themes.md", renderIndex()); - -for (const theme of themes.light) { - await generateFile(`./docs/theme/${theme}.md`, renderTheme(theme)); -} -for (const theme of themes.dark) { - await generateFile(`./docs/theme/${theme}.md`, renderTheme(theme)); -} - -await generateFile("./docs/theme/light.md", renderTheme("light")); -await generateFile("./docs/theme/light-alt.md", renderTheme("[light, alt]")); -await generateFile("./docs/theme/dark.md", renderTheme("dark")); -await generateFile("./docs/theme/dark-alt.md", renderTheme("[dark, alt]")); -await generateFile("./docs/theme/wide.md", renderTheme("wide")); -await generateFile("./docs/theme/dashboard.md", renderTheme("dashboard")); diff --git a/observablehq.config.ts b/observablehq.config.ts index 739658701..abf285df8 100644 --- a/observablehq.config.ts +++ b/observablehq.config.ts @@ -2,6 +2,7 @@ import {existsSync} from "node:fs"; import {readFile, readdir, stat} from "node:fs/promises"; import {join} from "node:path/posix"; import {formatPrefix} from "d3-format"; +import {themes} from "./docs/themes.md.ts"; let stargazers_count: number; try { @@ -24,10 +25,12 @@ export default { {name: "Reactivity", path: "/reactivity"}, {name: "JSX", path: "/jsx"}, {name: "Imports", path: "/imports"}, - {name: "Data loaders", path: "/loaders"}, + {name: "Data loaders", path: "/data-loaders"}, {name: "Files", path: "/files"}, {name: "SQL", path: "/sql"}, {name: "Themes", path: "/themes"}, + {name: "Page loaders", path: "/page-loaders"}, + {name: "Parameterized routes", path: "/params"}, {name: "Configuration", path: "/config"}, {name: "Deploying", path: "/deploying"}, {name: "Examples", path: "https://github.com/observablehq/framework/tree/main/examples"}, @@ -85,6 +88,18 @@ export default { {name: "Converting notebooks", path: "/convert"}, {name: "Contributing", path: "/contributing", pager: false} ], + dynamicPaths: [ + "/page-loaders", + "/theme/dark", + "/theme/dark-alt", + "/theme/dashboard", + "/theme/light", + "/theme/light-alt", + "/theme/wide", + "/themes", + ...themes.dark.map((theme) => `/theme/${theme}`), + ...themes.light.map((theme) => `/theme/${theme}`) + ], base: "/framework", globalStylesheets: [ "https://fonts.googleapis.com/css2?family=Inter:ital,opsz,wght@0,14..32,100..900;1,14..32,100..900&family=Spline+Sans+Mono:ital,wght@0,300..700;1,300..700&display=swap" @@ -125,14 +140,17 @@ export default { footer: `© ${new Date().getUTCFullYear()} Observable, Inc.`, style: "style.css", search: { - async* index() { + async *index() { for (const name of await readdir("examples")) { const root = join("examples", name); if ((await stat(root)).isDirectory() && existsSync(join(root, "README.md"))) { const source = await readFile(join(root, "README.md"), "utf-8"); yield { path: `https://observablehq.observablehq.cloud/framework-example-${name}/`, - title: source.split("\n").find((line) => line.startsWith("# "))?.slice(2), + title: source + .split("\n") + .find((line) => line.startsWith("# ")) + ?.slice(2), text: source }; } diff --git a/package.json b/package.json index acd31089e..d8ade74f8 100644 --- a/package.json +++ b/package.json @@ -19,10 +19,9 @@ "observable": "dist/bin/observable.js" }, "scripts": { - "dev": "rimraf --glob docs/themes.md docs/theme/*.md && (tsx watch docs/theme/generate-themes.ts & tsx watch --ignore docs --no-warnings=ExperimentalWarning ./src/bin/observable.ts preview --no-open)", - "docs:themes": "rimraf --glob docs/themes.md docs/theme/*.md && tsx docs/theme/generate-themes.ts", - "docs:build": "yarn docs:themes && tsx --no-warnings=ExperimentalWarning ./src/bin/observable.ts build", - "docs:deploy": "yarn docs:themes && tsx --no-warnings=ExperimentalWarning ./src/bin/observable.ts deploy", + "dev": "tsx watch --ignore docs --no-warnings=ExperimentalWarning ./src/bin/observable.ts preview --no-open", + "docs:build": "tsx --no-warnings=ExperimentalWarning ./src/bin/observable.ts build", + "docs:deploy": "tsx --no-warnings=ExperimentalWarning ./src/bin/observable.ts deploy", "build": "rimraf dist && node build.js --outdir=dist --outbase=src \"src/**/*.{ts,js,css}\" --ignore \"**/*.d.ts\"", "test": "concurrently npm:test:mocha npm:test:tsc npm:test:lint npm:test:prettier", "test:coverage": "c8 --check-coverage --lines 80 --per-file yarn test:mocha", @@ -70,6 +69,7 @@ "esbuild": "^0.20.1", "fast-array-diff": "^1.1.0", "fast-deep-equal": "^3.1.3", + "glob": "^10.3.10", "gray-matter": "^4.0.3", "he": "^1.2.0", "highlight.js": "^11.8.0", @@ -120,7 +120,6 @@ "eslint-config-prettier": "^9.1.0", "eslint-import-resolver-typescript": "^3.6.1", "eslint-plugin-import": "^2.29.0", - "glob": "^10.3.10", "mocha": "^10.2.0", "prettier": "^3.0.3 <3.1", "react": "^18.2.0", diff --git a/src/build.ts b/src/build.ts index d88c2b3e9..807a37235 100644 --- a/src/build.ts +++ b/src/build.ts @@ -1,15 +1,13 @@ import {createHash} from "node:crypto"; -import {existsSync} from "node:fs"; -import {access, constants, copyFile, readFile, rm, stat, writeFile} from "node:fs/promises"; +import {copyFile, readFile, rm, stat, writeFile} from "node:fs/promises"; import {basename, dirname, extname, join} from "node:path/posix"; import type {Config} from "./config.js"; import {CliError, isEnoent} from "./error.js"; -import {getClientPath, prepareOutput, visitMarkdownFiles} from "./files.js"; -import {getLocalModuleHash, getModuleHash, readJavaScript} from "./javascript/module.js"; +import {getClientPath, prepareOutput} from "./files.js"; +import {findModule, getLocalModuleHash, getModuleHash, readJavaScript} from "./javascript/module.js"; import {transpileModule} from "./javascript/transpile.js"; import type {Logger, Writer} from "./logger.js"; import type {MarkdownPage} from "./markdown.js"; -import {parseMarkdown} from "./markdown.js"; import {populateNpmCache, resolveNpmImport, rewriteNpmImports} from "./npm.js"; import {isAssetPath, isPathImport, relativePath, resolvePath, within} from "./path.js"; import {renderPage} from "./render.js"; @@ -57,15 +55,6 @@ export async function build( const {root, loaders} = config; Telemetry.record({event: "build", step: "start"}); - // Make sure all files are readable before starting to write output files. - let pageCount = 0; - for (const sourceFile of visitMarkdownFiles(root)) { - await access(join(root, sourceFile), constants.R_OK); - pageCount++; - } - if (!pageCount) throw new CliError(`Nothing to build: no page files found in your ${root} directory.`); - effects.logger.log(`${faint("found")} ${pageCount} ${faint(`page${pageCount === 1 ? "" : "s"} in`)} ${root}`); - // Prepare for build (such as by emptying the existing output root). await effects.prepare(); @@ -75,29 +64,31 @@ export async function build( const localImports = new Set(); // e.g., "/components/foo.js" const globalImports = new Set(); // e.g., "/_observablehq/search.js" const stylesheets = new Set(); // e.g., "/style.css" - for (const sourceFile of visitMarkdownFiles(root)) { - const sourcePath = join(root, sourceFile); - const path = join("/", dirname(sourceFile), basename(sourceFile, ".md")); - const options = {path, ...config}; - effects.output.write(`${faint("parse")} ${sourcePath} `); + for await (const path of config.paths()) { + effects.output.write(`${faint("parse")} ${path} `); const start = performance.now(); - const source = await readFile(sourcePath, "utf8"); - const page = parseMarkdown(source, options); + const options = {path, ...config}; + const page = await loaders.loadPage(path, options, effects); if (page.data.draft) { effects.logger.log(faint("(skipped)")); continue; } - const resolvers = await getResolvers(page, {path: sourceFile, ...config}); + const resolvers = await getResolvers(page, options); const elapsed = Math.floor(performance.now() - start); - for (const f of resolvers.assets) files.add(resolvePath(sourceFile, f)); - for (const f of resolvers.files) files.add(resolvePath(sourceFile, f)); - for (const i of resolvers.localImports) localImports.add(resolvePath(sourceFile, i)); - for (let i of resolvers.globalImports) if (isPathImport((i = resolvers.resolveImport(i)))) globalImports.add(resolvePath(sourceFile, i)); // prettier-ignore - for (const s of resolvers.stylesheets) stylesheets.add(/^\w+:/.test(s) ? s : resolvePath(sourceFile, s)); + for (const f of resolvers.assets) files.add(resolvePath(path, f)); + for (const f of resolvers.files) files.add(resolvePath(path, f)); + for (const i of resolvers.localImports) localImports.add(resolvePath(path, i)); + for (let i of resolvers.globalImports) if (isPathImport((i = resolvers.resolveImport(i)))) globalImports.add(resolvePath(path, i)); // prettier-ignore + for (const s of resolvers.stylesheets) stylesheets.add(/^\w+:/.test(s) ? s : resolvePath(path, s)); effects.output.write(`${faint("in")} ${(elapsed >= 100 ? yellow : faint)(`${elapsed}ms`)}\n`); - pages.set(sourceFile, {page, resolvers}); + pages.set(path, {page, resolvers}); } + // Check that there’s at least one page. + const pageCount = pages.size; + if (!pageCount) throw new CliError(`Nothing to build: no page files found in your ${root} directory.`); + effects.logger.log(`${faint("built")} ${pageCount} ${faint(`page${pageCount === 1 ? "" : "s"} in`)} ${root}`); + // For cache-breaking we rename most assets to include content hashes. const aliases = new Map(); const cacheRoot = join(root, ".observablehq", "cache"); @@ -164,21 +155,14 @@ export async function build( // Copy over referenced files, accumulating hashed aliases. for (const file of files) { - let sourcePath = join(root, file); - effects.output.write(`${faint("copy")} ${sourcePath} ${faint("→")} `); - if (!existsSync(sourcePath)) { - const loader = loaders.find(join("/", file), {useStale: true}); - if (!loader) { - effects.logger.error(red("error: missing referenced file")); - continue; - } - try { - sourcePath = join(root, await loader.load(effects)); - } catch (error) { - if (!isEnoent(error)) throw error; - effects.logger.error(red("error: missing referenced file")); - continue; - } + effects.output.write(`${faint("copy")} ${join(root, file)} ${faint("→")} `); + let sourcePath: string; + try { + sourcePath = join(root, await loaders.loadFile(join("/", file), {useStale: true}, effects)); + } catch (error) { + if (!isEnoent(error)) throw error; + effects.logger.error(red("error: missing referenced file")); + continue; } const contents = await readFile(sourcePath); const hash = createHash("sha256").update(contents).digest("hex").slice(0, 8); @@ -237,8 +221,13 @@ export async function build( return applyHash(join("/_import", path), hash); }; for (const path of localImports) { - const sourcePath = join(root, path); - const importPath = join("_import", path); + const module = findModule(root, path); + if (!module) { + effects.logger.error(red(`error: import not found: ${path}`)); + continue; + } + const sourcePath = join(root, module.path); + const importPath = join("_import", module.path); effects.output.write(`${faint("copy")} ${sourcePath} ${faint("→")} `); const resolveImport = getModuleResolver(root, path); let input: string; @@ -252,6 +241,7 @@ export async function build( const contents = await transpileModule(input, { root, path, + params: module.params, async resolveImport(specifier) { let resolution: string; if (isPathImport(specifier)) { @@ -272,10 +262,9 @@ export async function build( } // Wrap the resolvers to apply content-hashed file names. - for (const [sourceFile, page] of pages) { - const path = join("/", dirname(sourceFile), basename(sourceFile, ".md")); + for (const [path, page] of pages) { const {resolvers} = page; - pages.set(sourceFile, { + pages.set(path, { ...page, resolvers: { ...resolvers, @@ -305,14 +294,11 @@ export async function build( // Render pages! const buildManifest: BuildManifest = {pages: []}; - for (const [sourceFile, {page, resolvers}] of pages) { - const outputPath = join(dirname(sourceFile), basename(sourceFile, ".md") + ".html"); - const path = join("/", dirname(sourceFile), basename(sourceFile, ".md")); + for (const [path, {page, resolvers}] of pages) { effects.output.write(`${faint("render")} ${path} ${faint("→")} `); const html = await renderPage(page, {...config, path, resolvers}); - await effects.writeFile(outputPath, html); - const urlPath = config.normalizePath("/" + outputPath); - buildManifest.pages.push({path: urlPath, title: page.title}); + await effects.writeFile(`${path}.html`, html); + buildManifest.pages.push({path: config.normalizePath(path), title: page.title}); } // Write the build manifest. @@ -328,11 +314,9 @@ export async function build( }` ); } else { - const [sourceFile, {resolvers}] = node.data!; - const outputPath = join(dirname(sourceFile), basename(sourceFile, ".md") + ".html"); - const path = join("/", dirname(sourceFile), basename(sourceFile, ".md")); + const [path, {resolvers}] = node.data!; const resolveOutput = (name: string) => join(config.output, resolvePath(path, name)); - const pageSize = (await stat(join(config.output, outputPath))).size; + const pageSize = (await stat(join(config.output, `${path}.html`))).size; const importSize = await accumulateSize(resolvers.staticImports, resolvers.resolveImport, resolveOutput); const fileSize = (await accumulateSize(resolvers.files, resolvers.resolveFile, resolveOutput)) + diff --git a/src/client/preview.js b/src/client/preview.js index 9e94f90f5..215d9f914 100644 --- a/src/client/preview.js +++ b/src/client/preview.js @@ -8,7 +8,7 @@ export * from "./index.js"; let minReopenDelay = 1000; let maxReopenDelay = 30000; let reopenDelay = minReopenDelay; -let reopenDecay = 1.1; // exponential backoff factor +let reopenDecay = 1.5; // exponential backoff factor export function open({hash, eval: compile} = {}) { let opened = false; @@ -21,7 +21,6 @@ export function open({hash, eval: compile} = {}) { socket.onopen = () => { console.info("socket open"); opened = true; - reopenDelay = minReopenDelay; send({type: "hello", path: location.pathname, hash}); }; @@ -29,6 +28,10 @@ export function open({hash, eval: compile} = {}) { const message = JSON.parse(event.data); console.info("↓", message); switch (message.type) { + case "welcome": { + reopenDelay = minReopenDelay; // reset on successful connection + break; + } case "reload": { location.reload(); break; diff --git a/src/config.ts b/src/config.ts index e2891105b..f90aeae43 100644 --- a/src/config.ts +++ b/src/config.ts @@ -7,12 +7,13 @@ import {cwd} from "node:process"; import {pathToFileURL} from "node:url"; import type MarkdownIt from "markdown-it"; import wrapAnsi from "wrap-ansi"; -import {LoaderResolver} from "./dataloader.js"; import {visitMarkdownFiles} from "./files.js"; import {formatIsoDate, formatLocaleDate} from "./format.js"; import type {FrontMatter} from "./frontMatter.js"; +import {LoaderResolver} from "./loader.js"; import {createMarkdownIt, parseMarkdownMetadata} from "./markdown.js"; import {isAssetPath, parseRelativeUrl, resolvePath} from "./path.js"; +import {isParameterizedPath} from "./route.js"; import {resolveTheme} from "./theme.js"; import {bold, yellow} from "./tty.js"; @@ -80,6 +81,7 @@ export interface Config { sidebar: boolean; // defaults to true if pages isn’t empty pages: (Page | Section)[]; pager: boolean; // defaults to true + paths: () => AsyncIterable; // defaults to static Markdown files scripts: Script[]; // deprecated; defaults to empty array head: PageFragmentFunction | string | null; // defaults to null header: PageFragmentFunction | string | null; // defaults to null @@ -111,6 +113,7 @@ export interface ConfigSpec { title?: unknown; pages?: unknown; pager?: unknown; + dynamicPaths?: unknown; toc?: unknown; linkify?: unknown; typographer?: unknown; @@ -180,7 +183,7 @@ function readPages(root: string, md: MarkdownIt): Page[] { const files: {file: string; source: string}[] = []; const hash = createHash("sha256"); for (const file of visitMarkdownFiles(root)) { - if (file === "index.md" || file === "404.md") continue; + if (isParameterizedPath(file) || file === "index.md" || file === "404.md") continue; const source = readFileSync(join(root, file), "utf8"); files.push({file, source}); hash.update(file).update(source); @@ -239,6 +242,7 @@ export function normalizeConfig(spec: ConfigSpec = {}, defaultRoot?: string, wat const title = spec.title === undefined ? undefined : String(spec.title); const pages = spec.pages === undefined ? undefined : normalizePages(spec.pages); const pager = spec.pager === undefined ? true : Boolean(spec.pager); + const dynamicPaths = normalizeDynamicPaths(spec.dynamicPaths); const toc = normalizeToc(spec.toc as any); const sidebar = spec.sidebar === undefined ? undefined : Boolean(spec.sidebar); const scripts = spec.scripts === undefined ? [] : normalizeScripts(spec.scripts); @@ -247,6 +251,18 @@ export function normalizeConfig(spec: ConfigSpec = {}, defaultRoot?: string, wat const footer = pageFragment(spec.footer === undefined ? defaultFooter() : spec.footer); const search = spec.search == null || spec.search === false ? null : normalizeSearch(spec.search as any); const interpreters = normalizeInterpreters(spec.interpreters as any); + const normalizePath = getPathNormalizer(spec.cleanUrls); + + // If this path ends with a slash, then add an implicit /index to the + // end of the path. Otherwise, remove the .html extension (we use clean + // paths as the internal canonical representation; see normalizePage). + function normalizePagePath(pathname: string): string { + pathname = normalizePath(pathname); + if (pathname.endsWith("/")) pathname = join(pathname, "index"); + else pathname = pathname.replace(/\.html$/, ""); + return pathname; + } + const config: Config = { root, output, @@ -255,6 +271,14 @@ export function normalizeConfig(spec: ConfigSpec = {}, defaultRoot?: string, wat sidebar: sidebar!, // see below pages: pages!, // see below pager, + async *paths() { + for await (const path of getDefaultPaths(root)) { + yield normalizePagePath(path); + } + for await (const path of dynamicPaths()) { + yield normalizePagePath(path); + } + }, scripts, head, header, @@ -264,7 +288,7 @@ export function normalizeConfig(spec: ConfigSpec = {}, defaultRoot?: string, wat globalStylesheets, search, md, - normalizePath: getPathNormalizer(spec.cleanUrls), + normalizePath, loaders: new LoaderResolver({root, interpreters}), watchPath }; @@ -274,6 +298,18 @@ export function normalizeConfig(spec: ConfigSpec = {}, defaultRoot?: string, wat return config; } +function getDefaultPaths(root: string): string[] { + return Array.from(visitMarkdownFiles(root)) + .filter((path) => !isParameterizedPath(path)) + .map((path) => join("/", dirname(path), basename(path, ".md"))); +} + +function normalizeDynamicPaths(spec: unknown): Config["paths"] { + if (typeof spec === "function") return spec as () => AsyncIterable; + const paths = Array.from((spec ?? []) as ArrayLike, String); + return async function* () { yield* paths; }; // prettier-ignore +} + function getPathNormalizer(spec: unknown = true): (path: string) => string { const cleanUrls = Boolean(spec); return (path) => { diff --git a/src/error.ts b/src/error.ts index 4eb673a6d..8823d8900 100644 --- a/src/error.ts +++ b/src/error.ts @@ -17,6 +17,10 @@ export class HttpError extends Error { } } +export function enoent(path: string): NodeJS.ErrnoException { + return Object.assign(new Error(`Not found: ${path}`), {code: "ENOENT"}); +} + export function isEnoent(error: unknown): error is NodeJS.ErrnoException { return isSystemError(error) && error.code === "ENOENT"; } diff --git a/src/fileWatchers.ts b/src/fileWatchers.ts index 194c2d938..c1dcdf15e 100644 --- a/src/fileWatchers.ts +++ b/src/fileWatchers.ts @@ -1,8 +1,8 @@ import type {FSWatcher} from "node:fs"; import {watch} from "node:fs"; -import type {LoaderResolver} from "./dataloader.js"; import {isEnoent} from "./error.js"; import {maybeStat} from "./files.js"; +import type {LoaderResolver} from "./loader.js"; import {resolvePath} from "./path.js"; export class FileWatchers { diff --git a/src/javascript/globals.ts b/src/javascript/globals.ts index 32fb8d650..cd3326943 100644 --- a/src/javascript/globals.ts +++ b/src/javascript/globals.ts @@ -47,6 +47,7 @@ export const defaultGlobals = new Set([ "Number", "navigator", "Object", + "observable", // for observable.params.foo "parseFloat", "parseInt", "performance", diff --git a/src/javascript/module.ts b/src/javascript/module.ts index 237673c5e..5b3e4dc14 100644 --- a/src/javascript/module.ts +++ b/src/javascript/module.ts @@ -1,14 +1,16 @@ import type {Hash} from "node:crypto"; import {createHash} from "node:crypto"; -import {accessSync, constants, existsSync, readFileSync, statSync} from "node:fs"; +import {accessSync, constants, readFileSync, statSync} from "node:fs"; import {readFile} from "node:fs/promises"; -import {join} from "node:path/posix"; +import {extname, join} from "node:path/posix"; import type {Program} from "acorn"; import {transform, transformSync} from "esbuild"; import {resolveNodeImport} from "../node.js"; import {resolveNpmImport} from "../npm.js"; import {resolvePath} from "../path.js"; import {builtins} from "../resolvers.js"; +import type {RouteResult} from "../route.js"; +import {route} from "../route.js"; import {findFiles} from "./files.js"; import {findImports, parseImports} from "./imports.js"; import {parseProgram} from "./parse.js"; @@ -118,10 +120,12 @@ export async function getLocalModuleHash(root: string, path: string): Promise { - const jsxPath = resolveJsx(path); - if (jsxPath !== null) { - const source = await readFile(jsxPath, "utf-8"); +export async function readJavaScript(sourcePath: string): Promise { + const source = await readFile(sourcePath, "utf-8"); + if (sourcePath.endsWith(".jsx")) { const {code} = await transform(source, { loader: "jsx", jsx: "automatic", jsxImportSource: "npm:react", - sourcefile: jsxPath + sourcefile: sourcePath }); return code; } - return await readFile(path, "utf-8"); + return source; } -export function readJavaScriptSync(path: string): string { - const jsxPath = resolveJsx(path); - if (jsxPath !== null) { - const source = readFileSync(jsxPath, "utf-8"); +export function readJavaScriptSync(sourcePath: string): string { + const source = readFileSync(sourcePath, "utf-8"); + if (sourcePath.endsWith(".jsx")) { const {code} = transformSync(source, { loader: "jsx", jsx: "automatic", jsxImportSource: "npm:react", - sourcefile: jsxPath + sourcefile: sourcePath }); return code; } - return readFileSync(path, "utf-8"); + return source; } diff --git a/src/javascript/params.ts b/src/javascript/params.ts new file mode 100644 index 000000000..ca543f924 --- /dev/null +++ b/src/javascript/params.ts @@ -0,0 +1,52 @@ +import type {Identifier, Literal, MemberExpression, Node} from "acorn"; +import {simple} from "acorn-walk"; +import type {Params} from "../route.js"; +import {findReferences} from "./references.js"; +import {syntaxError} from "./syntaxError.js"; + +export type ParamReference = MemberExpression & {property: Identifier | (Literal & {value: string}); value: string}; + +function getParamName(param: ParamReference): string { + return param.property.type === "Identifier" ? param.property.name : param.property.value; +} + +// Note: mutates node by materializing the values of the param references it contains +export function checkParams(node: Node, input: string, params: Params): void { + for (const [name, param] of findParams(node, params, input)) { + param.value = params[name]; + } +} + +export function findParams(body: Node, params: Params, input: string): [name: string, node: ParamReference][] { + const matches: [string, ParamReference][] = []; + const references = findReferences(body, {filterReference: ({name}) => name === "observable"}); + + simple(body, { + MemberExpression(node) { + if (isParamReference(node)) { + const name = getParamName(node); + if (!(name in params)) throw syntaxError(`undefined parameter: ${name}`, node, input); + matches.push([name, node]); + } + } + }); + + function isParamReference(node: MemberExpression): node is ParamReference { + if ( + node.object.type !== "MemberExpression" || + node.object.object.type !== "Identifier" || + node.object.object.name !== "observable" || + node.object.property.type !== "Identifier" || + node.object.property.name !== "params" || + !references.includes(node.object.object) + ) { + return false; + } + if (node.property.type !== "Identifier" && node.property.type !== "Literal") { + throw syntaxError("invalid param reference", node, input); + } + return true; + } + + return matches; +} diff --git a/src/javascript/parse.ts b/src/javascript/parse.ts index 5e16c68de..aef288568 100644 --- a/src/javascript/parse.ts +++ b/src/javascript/parse.ts @@ -1,5 +1,6 @@ import {Parser, tokTypes} from "acorn"; import type {Expression, Identifier, Options, Program} from "acorn"; +import type {Params} from "../route.js"; import {checkAssignments} from "./assignments.js"; import {findAwaits} from "./awaits.js"; import {findDeclarations} from "./declarations.js"; @@ -7,6 +8,7 @@ import type {FileExpression} from "./files.js"; import {findFiles} from "./files.js"; import type {ImportReference} from "./imports.js"; import {findExports, findImports} from "./imports.js"; +import {checkParams} from "./params.js"; import {findReferences} from "./references.js"; import {syntaxError} from "./syntaxError.js"; @@ -15,6 +17,8 @@ export interface ParseOptions { path: string; /** If true, require the input to be an expresssion. */ inline?: boolean; + /** Any dynamic route parameters for observable.params. */ + params?: Params; } export const acornOptions: Options = { @@ -38,7 +42,7 @@ export interface JavaScriptNode { * the specified inline JavaScript expression. */ export function parseJavaScript(input: string, options: ParseOptions): JavaScriptNode { - const {inline = false, path} = options; + const {inline = false, path, params} = options; let expression = maybeParseExpression(input); // first attempt to parse as expression if (expression?.type === "ClassExpression" && expression.id) expression = null; // treat named class as program if (expression?.type === "FunctionExpression" && expression.id) expression = null; // treat named function as program @@ -47,6 +51,7 @@ export function parseJavaScript(input: string, options: ParseOptions): JavaScrip const exports = findExports(body); if (exports.length) throw syntaxError("Unexpected token 'export'", exports[0], input); // disallow exports const references = findReferences(body); + if (params) checkParams(body, input, params); checkAssignments(body, references, input); return { body, @@ -60,8 +65,10 @@ export function parseJavaScript(input: string, options: ParseOptions): JavaScrip }; } -export function parseProgram(input: string): Program { - return Parser.parse(input, acornOptions); +export function parseProgram(input: string, params?: Params): Program { + const body = Parser.parse(input, acornOptions); + if (params) checkParams(body, input, params); + return body; } /** diff --git a/src/javascript/references.ts b/src/javascript/references.ts index d5d85da9c..56fd19426 100644 --- a/src/javascript/references.ts +++ b/src/javascript/references.ts @@ -48,9 +48,11 @@ export function findReferences( node: Node, { globals = defaultGlobals, + filterReference = (identifier: Identifier) => !globals.has(identifier.name), filterDeclaration = () => true }: { globals?: Set; + filterReference?: (identifier: Identifier) => any; filterDeclaration?: (identifier: {name: string}) => any; } = {} ): Identifier[] { @@ -149,7 +151,7 @@ export function findReferences( return; } } - if (!globals.has(name)) { + if (filterReference(node)) { references.push(node); } } diff --git a/src/javascript/source.ts b/src/javascript/source.ts index 648f3e7f5..e905231a3 100644 --- a/src/javascript/source.ts +++ b/src/javascript/source.ts @@ -1,10 +1,6 @@ -import type {Literal, Node, TemplateLiteral} from "acorn"; +import type {BinaryExpression, Literal, MemberExpression, Node, TemplateLiteral} from "acorn"; -export type StringLiteral = ( - | {type: "Literal"; value: string} - | {type: "TemplateLiteral"; quasis: {value: {cooked: string}}[]} -) & - Node; +export type StringLiteral = (Literal & {value: string}) | TemplateLiteral | BinaryExpression; export function isLiteral(node: Node): node is Literal { return node.type === "Literal"; @@ -15,9 +11,42 @@ export function isTemplateLiteral(node: Node): node is TemplateLiteral { } export function isStringLiteral(node: Node): node is StringLiteral { - return isLiteral(node) ? /^['"]/.test(node.raw!) : isTemplateLiteral(node) ? node.expressions.length === 0 : false; + return isLiteral(node) + ? /^['"]/.test(node.raw!) + : isTemplateLiteral(node) + ? node.expressions.every(isStringLiteral) + : isBinaryExpression(node) + ? node.operator === "+" && isStringLiteral(node.left) && isStringLiteral(node.right) + : isMemberExpression(node) + ? "value" in node // param + : false; } export function getStringLiteralValue(node: StringLiteral): string { - return node.type === "Literal" ? node.value : node.quasis[0].value.cooked; + return node.type === "TemplateLiteral" + ? getTemplateLiteralValue(node) + : node.type === "BinaryExpression" + ? getBinaryExpressionValue(node) + : node.value; // Literal or ParamReference +} + +function getTemplateLiteralValue(node: TemplateLiteral): string { + let value = node.quasis[0].value.cooked!; + for (let i = 0; i < node.expressions.length; ++i) { + value += getStringLiteralValue(node.expressions[i] as StringLiteral); + value += node.quasis[i + 1].value.cooked!; + } + return value; +} + +function getBinaryExpressionValue(node: BinaryExpression): string { + return getStringLiteralValue(node.left as StringLiteral) + getStringLiteralValue(node.right as StringLiteral); +} + +function isMemberExpression(node: Node): node is MemberExpression { + return node.type === "MemberExpression"; +} + +function isBinaryExpression(node: Node): node is BinaryExpression { + return node.type === "BinaryExpression"; } diff --git a/src/javascript/transpile.ts b/src/javascript/transpile.ts index b264ca94f..3838e1b46 100644 --- a/src/javascript/transpile.ts +++ b/src/javascript/transpile.ts @@ -4,11 +4,13 @@ import type {ImportDeclaration, ImportDefaultSpecifier, ImportNamespaceSpecifier import {simple} from "acorn-walk"; import {isPathImport, relativePath, resolvePath, resolveRelativePath} from "../path.js"; import {getModuleResolver} from "../resolvers.js"; +import type {Params} from "../route.js"; import {Sourcemap} from "../sourcemap.js"; import type {FileExpression} from "./files.js"; import {findFiles} from "./files.js"; import type {ExportNode, ImportNode} from "./imports.js"; import {hasImportDeclaration, isImportMetaResolve} from "./imports.js"; +import {findParams} from "./params.js"; import type {JavaScriptNode} from "./parse.js"; import {parseProgram} from "./parse.js"; import type {StringLiteral} from "./source.js"; @@ -17,11 +19,15 @@ import {getStringLiteralValue, isStringLiteral} from "./source.js"; export interface TranspileOptions { id: string; path: string; + params?: Params; mode?: string; resolveImport?: (specifier: string) => string; } -export function transpileJavaScript(node: JavaScriptNode, {id, path, mode, resolveImport}: TranspileOptions): string { +export function transpileJavaScript( + node: JavaScriptNode, + {id, path, params, mode, resolveImport}: TranspileOptions +): string { let async = node.async; const inputs = Array.from(new Set(node.references.map((r) => r.name))); const outputs = Array.from(new Set(node.declarations?.map((r) => r.name))); @@ -29,6 +35,7 @@ export function transpileJavaScript(node: JavaScriptNode, {id, path, mode, resol if (display) inputs.push("display"), (async = true); if (hasImportDeclaration(node.body)) async = true; const output = new Sourcemap(node.input).trim(); + if (params) rewriteParams(output, node.body, params, node.input); rewriteImportDeclarations(output, node.body, resolveImport); rewriteImportExpressions(output, node.body, resolveImport); rewriteFileExpressions(output, node.files, path); @@ -46,20 +53,23 @@ export function transpileJavaScript(node: JavaScriptNode, {id, path, mode, resol export interface TranspileModuleOptions { root: string; path: string; + params?: Params; resolveImport?: (specifier: string) => Promise; } /** Rewrites import specifiers and FileAttachment calls in the specified ES module. */ export async function transpileModule( input: string, - {root, path, resolveImport = getModuleResolver(root, path)}: TranspileModuleOptions + {root, path, params, resolveImport = getModuleResolver(root, path)}: TranspileModuleOptions ): Promise { const servePath = `/${join("_import", path)}`; - const body = parseProgram(input); // TODO ignore syntax error? + const body = parseProgram(input, params); // TODO ignore syntax error? const output = new Sourcemap(input); const imports: (ImportNode | ExportNode)[] = []; const calls: CallExpression[] = []; + if (params) rewriteParams(output, body, params, input); + simple(body, { ImportDeclaration: rewriteImport, ImportExpression: rewriteImport, @@ -211,3 +221,9 @@ function isNotNamespaceSpecifier( ): node is ImportSpecifier | ImportDefaultSpecifier { return node.type !== "ImportNamespaceSpecifier"; } + +export function rewriteParams(output: Sourcemap, body: Node, params: Params, input: string): void { + for (const [name, node] of findParams(body, params, input)) { + output.replaceLeft(node.start, node.end, JSON.stringify(params[name])); + } +} diff --git a/src/dataloader.ts b/src/loader.ts similarity index 55% rename from src/dataloader.ts rename to src/loader.ts index c86e9bc6d..df11f1620 100644 --- a/src/dataloader.ts +++ b/src/loader.ts @@ -1,18 +1,23 @@ import {createHash} from "node:crypto"; -import type {WriteStream} from "node:fs"; -import {createReadStream, existsSync, statSync} from "node:fs"; +import type {FSWatcher, WatchListener, WriteStream} from "node:fs"; +import {createReadStream, existsSync, statSync, watch} from "node:fs"; import {open, readFile, rename, unlink} from "node:fs/promises"; -import {dirname, extname, join, relative} from "node:path/posix"; +import {dirname, extname, join} from "node:path/posix"; import {createGunzip} from "node:zlib"; import {spawn} from "cross-spawn"; import JSZip from "jszip"; import {extract} from "tar-stream"; +import {enoent} from "./error.js"; import {maybeStat, prepareOutput} from "./files.js"; import {FileWatchers} from "./fileWatchers.js"; import {formatByteSize} from "./format.js"; import type {FileInfo} from "./javascript/module.js"; import {getFileInfo} from "./javascript/module.js"; import type {Logger, Writer} from "./logger.js"; +import type {MarkdownPage, ParseOptions} from "./markdown.js"; +import {parseMarkdown} from "./markdown.js"; +import type {Params} from "./route.js"; +import {route} from "./route.js"; import {cyan, faint, green, red, yellow} from "./tty.js"; const runningCommands = new Map>(); @@ -42,9 +47,14 @@ const defaultEffects: LoadEffects = { output: process.stdout }; +export interface LoadOptions { + useStale?: boolean; +} + export interface LoaderOptions { root: string; path: string; + params?: Params; targetPath: string; useStale: boolean; } @@ -63,60 +73,148 @@ export class LoaderResolver { } /** - * Finds the loader for the specified target path, relative to the specified - * source root, if it exists. If there is no such loader, returns undefined. + * Loads the file at the specified path, returning a promise to the path to + * the (possibly generated) file relative to the source root. + */ + async loadFile(path: string, options?: LoadOptions, effects?: LoadEffects): Promise { + const loader = this.find(path, options); + if (!loader) throw enoent(path); + return await loader.load(effects); + } + + /** + * Loads the page at the specified path, returning a promise to the parsed + * page object. + */ + async loadPage(path: string, options: LoadOptions & ParseOptions, effects?: LoadEffects): Promise { + const loader = this.find(`${path}.md`); + if (!loader) throw enoent(path); + const source = await readFile(join(this.root, await loader.load(effects)), "utf8"); + return parseMarkdown(source, {params: loader.params, ...options}); + } + + /** + * Returns a watcher for the page at the specified path. + */ + watchPage(path: string, listener: WatchListener): FSWatcher { + const loader = this.find(`${path}.md`); + if (!loader) throw enoent(path); + return watch(join(this.root, loader.path), listener); + } + + /** + * Finds the loader for the specified target path, relative to the source + * root, if the loader exists. If there is no such loader, returns undefined. * For files within archives, we find the first parent folder that exists, but * abort if we find a matching folder or reach the source root; for example, * if src/data exists, we won’t look for a src/data.zip. */ - find(targetPath: string, {useStale = false} = {}): Loader | undefined { - const exact = this.findExact(targetPath, {useStale}); - if (exact) return exact; - let dir = dirname(targetPath); - for (let parent: string; true; dir = parent) { - parent = dirname(dir); - if (parent === dir) return; // reached source root - if (existsSync(join(this.root, dir))) return; // found folder - if (existsSync(join(this.root, parent))) break; // found parent - } - for (const [ext, Extractor] of extractors) { - const archive = dir + ext; - if (existsSync(join(this.root, archive))) { - return new Extractor({ - preload: async () => archive, - inflatePath: targetPath.slice(archive.length - ext.length + 1), - path: join(this.root, archive), - root: this.root, - targetPath, - useStale - }); - } - const archiveLoader = this.findExact(archive, {useStale}); - if (archiveLoader) { + find(path: string, {useStale = false}: LoadOptions = {}): Loader | undefined { + return this.findFile(path, {useStale}) ?? this.findArchive(path, {useStale}); + } + + // Finding a file: + // - /path/to/file.csv + // - /path/to/file.csv.js + // - /path/to/[param].csv + // - /path/to/[param].csv.js + // - /path/[param]/file.csv + // - /path/[param]/file.csv.js + // - /path/[param1]/[param2].csv + // - /path/[param1]/[param2].csv.js + // - /[param]/to/file.csv + // - /[param]/to/file.csv.js + // - /[param1]/to/[param2].csv + // - /[param1]/to/[param2].csv.js + // - /[param1]/[param2]/file.csv + // - /[param1]/[param2]/file.csv.js + // - /[param1]/[param2]/[param3].csv + // - /[param1]/[param2]/[param3].csv.js + private findFile(targetPath: string, {useStale}: {useStale: boolean}): Loader | undefined { + const ext = extname(targetPath); + const exts = [ext, ...Array.from(this.interpreters.keys(), (iext) => ext + iext)]; + const found = route(this.root, targetPath.slice(0, -ext.length), exts); + if (!found) return; + const {path, params, ext: fext} = found; + if (fext === ext) return new StaticLoader({root: this.root, path, params}); + const commandPath = join(this.root, path); + const [command, ...args] = this.interpreters.get(fext.slice(ext.length))!; + if (command != null) args.push(commandPath); + return new CommandLoader({ + command: command ?? commandPath, + args: params ? args.concat(defineParams(params)) : args, + path, + params, + root: this.root, + targetPath, + useStale + }); + } + + // Finding a file in an archive: + // - /path/to.zip + // - /path/to.tgz + // - /path/to.zip.js + // - /path/to.tgz.js + // - /path/[param].zip + // - /path/[param].tgz + // - /path/[param].zip.js + // - /path/[param].tgz.js + // - /[param]/to.zip + // - /[param]/to.tgz + // - /[param]/to.zip.js + // - /[param]/to.tgz.js + // - /[param1]/[param2].zip + // - /[param1]/[param2].tgz + // - /[param1]/[param2].zip.js + // - /[param1]/[param2].tgz.js + // - /path.zip + // - /path.tgz + // - /path.zip.js + // - /path.tgz.js + // - /[param].zip + // - /[param].tgz + // - /[param].zip.js + // - /[param].tgz.js + private findArchive(targetPath: string, {useStale}: {useStale: boolean}): Loader | undefined { + const exts = this.getArchiveExtensions(); + for (let dir = dirname(targetPath), parent: string; (parent = dirname(dir)) !== dir; dir = parent) { + const found = route(this.root, dir, exts); + if (!found) continue; + const {path, params, ext: fext} = found; + const inflatePath = targetPath.slice(dir.length + 1); // file.jpeg + if (extractors.has(fext)) { + const Extractor = extractors.get(fext)!; return new Extractor({ - preload: async (options) => archiveLoader.load(options), - inflatePath: targetPath.slice(archive.length - ext.length + 1), - path: archiveLoader.path, + preload: async () => path, // /path/to.zip + inflatePath, + path, + params, root: this.root, - targetPath, + targetPath, // /path/to/file.jpg useStale }); } - } - } - - private findExact(targetPath: string, {useStale}): Loader | undefined { - for (const [ext, [command, ...args]] of this.interpreters) { - if (!existsSync(join(this.root, targetPath + ext))) continue; - if (extname(targetPath) === "") { - console.warn(`invalid data loader path: ${targetPath + ext}`); - return; - } - const path = join(this.root, targetPath + ext); - return new CommandLoader({ - command: command ?? path, - args: command == null ? args : [...args, path], + const iext = extname(fext); + const commandPath = join(this.root, path); + const [command, ...args] = this.interpreters.get(iext)!; + if (command != null) args.push(commandPath); + const eext = fext.slice(0, -iext.length); // .zip + const loader = new CommandLoader({ + command: command ?? commandPath, + args: params ? args.concat(defineParams(params)) : args, path, + params, + root: this.root, + targetPath: dir + eext, // /path/to.zip + useStale + }); + const Extractor = extractors.get(eext)!; + return new Extractor({ + preload: async (options) => loader.load(options), // /path/to.zip.js + inflatePath, + path: loader.path, + params, root: this.root, targetPath, useStale @@ -124,14 +222,27 @@ export class LoaderResolver { } } + // .zip, .tar, .tgz, .zip.js, .zip.py, etc. + getArchiveExtensions(): string[] { + const exts = Array.from(extractors.keys()); + for (const e of extractors.keys()) for (const i of this.interpreters.keys()) exts.push(e + i); + return exts; + } + + /** + * Returns the path to watch, relative to the current working directory, for + * the specified source path, relative to the source root. + */ getWatchPath(path: string): string | undefined { const exactPath = join(this.root, path); if (existsSync(exactPath)) return exactPath; if (exactPath.endsWith(".js")) { const jsxPath = exactPath + "x"; if (existsSync(jsxPath)) return jsxPath; + return; // loaders aren’t supported for .js } - return this.find(path)?.path; + const foundPath = this.find(path)?.path; + if (foundPath) return join(this.root, foundPath); } watchFiles(path: string, watchPaths: Iterable, callback: (name: string) => void) { @@ -139,27 +250,26 @@ export class LoaderResolver { } /** - * Returns the path to the backing file during preview, which is the source - * file for the associated data loader if the file is generated by a loader. + * Returns the path to the backing file during preview, relative to the source + * root, which is the source file for the associated data loader if the file + * is generated by a loader. */ - private getSourceFilePath(name: string): string { - let path = name; + private getSourceFilePath(path: string): string { if (!existsSync(join(this.root, path))) { const loader = this.find(path); - if (loader) path = relative(this.root, loader.path); + if (loader) return loader.path; } return path; } /** - * Returns the path to the backing file during build, which is the cached - * output file if the file is generated by a loader. + * Returns the path to the backing file during build, relative to the source + * root, which is the cached output file if the file is generated by a loader. */ - private getOutputFilePath(name: string): string { - let path = name; + private getOutputFilePath(path: string): string { if (!existsSync(join(this.root, path))) { const loader = this.find(path); - if (loader) path = join(".observablehq", "cache", name); + if (loader) return join(".observablehq", "cache", path); } return path; } @@ -192,19 +302,58 @@ export class LoaderResolver { } } -export abstract class Loader { +function defineParams(params: Params): string[] { + return Object.entries(params) + .filter(([name]) => /^[a-z0-9_]+$/i.test(name)) // ignore non-ASCII parameters + .map(([name, value]) => `--${name}=${value}`); +} + +export interface Loader { /** * The source root relative to the current working directory, such as src. */ readonly root: string; /** - * The path to the loader script or executable relative to the current working - * directory. This is exposed so that clients can check which file to watch to - * see if the loader is edited (and in which case it needs to be re-run). + * The path to the loader script or executable relative to the source root. + * This is exposed so that clients can check which file to watch to see if the + * loader is edited (and in which case it needs to be re-run). */ readonly path: string; + /** TODO */ + readonly params: Params | undefined; + + /** + * Runs this loader, returning the path to the generated output file relative + * to the source root; this is typically within the .observablehq/cache folder + * within the source root. + */ + load(effects?: LoadEffects): Promise; +} + +/** Used by LoaderResolver.find to represent a static file resolution. */ +class StaticLoader implements Loader { + readonly root: string; + readonly path: string; + readonly params: Params | undefined; + + constructor({root, path, params}: Omit) { + this.root = root; + this.path = path; + this.params = params; + } + + async load() { + return this.path; + } +} + +abstract class AbstractLoader implements Loader { + readonly root: string; + readonly path: string; + readonly params: Params | undefined; + /** * The path to the loader script’s output relative to the destination root. * This is where the loader’s output is served, but the loader generates the @@ -213,30 +362,27 @@ export abstract class Loader { readonly targetPath: string; /** - * Should the loader use a stale cache. true when building. + * Whether the loader should use a stale cache; true when building. */ readonly useStale?: boolean; - constructor({root, path, targetPath, useStale}: LoaderOptions) { + constructor({root, path, params, targetPath, useStale}: LoaderOptions) { this.root = root; this.path = path; + this.params = params; this.targetPath = targetPath; this.useStale = useStale; } - /** - * Runs this loader, returning the path to the generated output file relative - * to the source root; this is within the .observablehq/cache folder within - * the source root. - */ async load(effects = defaultEffects): Promise { + const loaderPath = join(this.root, this.path); const key = join(this.root, this.targetPath); let command = runningCommands.get(key); if (!command) { command = (async () => { const outputPath = join(".observablehq", "cache", this.targetPath); const cachePath = join(this.root, outputPath); - const loaderStat = await maybeStat(this.path); + const loaderStat = await maybeStat(loaderPath); const cacheStat = await maybeStat(cachePath); if (!cacheStat) effects.output.write(faint("[missing] ")); else if (cacheStat.mtimeMs < loaderStat!.mtimeMs) { @@ -268,7 +414,7 @@ export abstract class Loader { command.finally(() => runningCommands.delete(key)).catch(() => {}); runningCommands.set(key, command); } - effects.output.write(`${cyan("load")} ${this.path} ${faint("→")} `); + effects.output.write(`${cyan("load")} ${this.targetPath} ${faint("→")} `); const start = performance.now(); command.then( (path) => { @@ -294,7 +440,7 @@ interface CommandLoaderOptions extends LoaderOptions { args: string[]; } -class CommandLoader extends Loader { +class CommandLoader extends AbstractLoader { /** * The command to run, such as "node" for a JavaScript loader, "tsx" for * TypeScript, and "sh" for a shell script. "noop" when we only need to @@ -327,16 +473,16 @@ class CommandLoader extends Loader { } } -interface ZipExtractorOptions extends LoaderOptions { +interface ExtractorOptions extends LoaderOptions { preload: Loader["load"]; inflatePath: string; } -class ZipExtractor extends Loader { +class ZipExtractor extends AbstractLoader { private readonly preload: Loader["load"]; private readonly inflatePath: string; - constructor({preload, inflatePath, ...options}: ZipExtractorOptions) { + constructor({preload, inflatePath, ...options}: ExtractorOptions) { super(options); this.preload = preload; this.inflatePath = inflatePath; @@ -345,19 +491,17 @@ class ZipExtractor extends Loader { async exec(output: WriteStream, effects?: LoadEffects): Promise { const archivePath = join(this.root, await this.preload(effects)); const file = (await JSZip.loadAsync(await readFile(archivePath))).file(this.inflatePath); - if (!file) throw Object.assign(new Error("file not found"), {code: "ENOENT"}); + if (!file) throw enoent(this.inflatePath); const pipe = file.nodeStream().pipe(output); await new Promise((resolve, reject) => pipe.on("error", reject).on("finish", resolve)); } } -interface TarExtractorOptions extends LoaderOptions { - preload: Loader["load"]; - inflatePath: string; +interface TarExtractorOptions extends ExtractorOptions { gunzip?: boolean; } -class TarExtractor extends Loader { +class TarExtractor extends AbstractLoader { private readonly preload: Loader["load"]; private readonly inflatePath: string; private readonly gunzip: boolean; @@ -383,7 +527,7 @@ class TarExtractor extends Loader { entry.resume(); } } - throw Object.assign(new Error("file not found"), {code: "ENOENT"}); + throw enoent(this.inflatePath); } } @@ -393,12 +537,12 @@ class TarGzExtractor extends TarExtractor { } } -const extractors = [ +const extractors = new Map Loader>([ [".zip", ZipExtractor], [".tar", TarExtractor], [".tar.gz", TarGzExtractor], [".tgz", TarGzExtractor] -] as const; +]); function formatElapsed(start: number): string { const elapsed = performance.now() - start; diff --git a/src/markdown.ts b/src/markdown.ts index 5be039275..394700d22 100644 --- a/src/markdown.ts +++ b/src/markdown.ts @@ -19,6 +19,7 @@ import type {JavaScriptNode} from "./javascript/parse.js"; import {parseJavaScript} from "./javascript/parse.js"; import {isAssetPath, relativePath} from "./path.js"; import {parsePlaceholder} from "./placeholder.js"; +import type {Params} from "./route.js"; import {transpileSql} from "./sql.js"; import {transpileTag} from "./tag.js"; import {InvalidThemeError} from "./theme.js"; @@ -39,6 +40,7 @@ export interface MarkdownPage { data: FrontMatter; style: string | null; code: MarkdownCode[]; + params?: Params; } interface ParseContext { @@ -46,6 +48,7 @@ interface ParseContext { startLine: number; currentLine: number; path: string; + params?: Params; } function uniqueCodeId(context: ParseContext, content: string): string { @@ -108,7 +111,7 @@ function getLiveSource(content: string, tag: string, attributes: Record { - const {path} = context; + const {path, params} = context; const token = tokens[idx]; const {tag, attributes} = parseInfo(token.info); token.info = tag; @@ -119,7 +122,7 @@ function makeFenceRenderer(baseRenderer: RenderRule): RenderRule { if (source != null) { const id = uniqueCodeId(context, source); // TODO const sourceLine = context.startLine + context.currentLine; - const node = parseJavaScript(source, {path}); + const node = parseJavaScript(source, {path, params}); context.code.push({id, node, mode: tag === "jsx" ? "jsx" : "block"}); html += `
${ node.expression ? "" : "" @@ -180,12 +183,12 @@ const transformPlaceholderCore: RuleCore = (state) => { function makePlaceholderRenderer(): RenderRule { return (tokens, idx, options, context: ParseContext) => { - const {path} = context; + const {path, params} = context; const token = tokens[idx]; const id = uniqueCodeId(context, token.content); try { // TODO sourceLine: context.startLine + context.currentLine - const node = parseJavaScript(token.content, {path, inline: true}); + const node = parseJavaScript(token.content, {path, params, inline: true}); context.code.push({id, node, mode: "inline"}); return ``; } catch (error) { @@ -212,6 +215,7 @@ export interface ParseOptions { head?: Config["head"]; header?: Config["header"]; footer?: Config["footer"]; + params?: Params; } export function createMarkdownIt({ @@ -237,10 +241,10 @@ export function createMarkdownIt({ } export function parseMarkdown(input: string, options: ParseOptions): MarkdownPage { - const {md, path} = options; + const {md, path, params} = options; const {content, data} = readFrontMatter(input); const code: MarkdownCode[] = []; - const context: ParseContext = {code, startLine: 0, currentLine: 0, path}; + const context: ParseContext = {code, startLine: 0, currentLine: 0, path, params}; const tokens = md.parse(content, context); const body = md.renderer.render(tokens, md.options, context); // Note: mutates code! const title = data.title !== undefined ? data.title : findTitle(tokens); @@ -252,7 +256,8 @@ export function parseMarkdown(input: string, options: ParseOptions): MarkdownPag data, title, style: getStyle(data, options), - code + code, + params }; } diff --git a/src/preview.ts b/src/preview.ts index 3922544d5..bd0bb62e5 100644 --- a/src/preview.ts +++ b/src/preview.ts @@ -1,7 +1,7 @@ import {createHash} from "node:crypto"; import {watch} from "node:fs"; import type {FSWatcher, WatchEventType} from "node:fs"; -import {access, constants, readFile} from "node:fs/promises"; +import {access, constants} from "node:fs/promises"; import {createServer} from "node:http"; import type {IncomingMessage, RequestListener, Server, ServerResponse} from "node:http"; import {basename, dirname, join, normalize} from "node:path/posix"; @@ -16,15 +16,14 @@ import type {WebSocket} from "ws"; import {WebSocketServer} from "ws"; import type {Config} from "./config.js"; import {readConfig} from "./config.js"; -import type {LoaderResolver} from "./dataloader.js"; -import {HttpError, isEnoent, isHttpError, isSystemError} from "./error.js"; +import {enoent, isEnoent, isHttpError, isSystemError} from "./error.js"; import {getClientPath} from "./files.js"; import type {FileWatchers} from "./fileWatchers.js"; import {isComment, isElement, isText, parseHtml, rewriteHtml} from "./html.js"; import type {FileInfo} from "./javascript/module.js"; -import {readJavaScript} from "./javascript/module.js"; +import {findModule, readJavaScript} from "./javascript/module.js"; import {transpileJavaScript, transpileModule} from "./javascript/transpile.js"; -import {parseMarkdown} from "./markdown.js"; +import type {LoaderResolver} from "./loader.js"; import type {MarkdownCode, MarkdownPage} from "./markdown.js"; import {populateNpmCache} from "./npm.js"; import {isPathImport, resolvePath} from "./path.js"; @@ -32,6 +31,8 @@ import {renderPage} from "./render.js"; import type {Resolvers} from "./resolvers.js"; import {getResolvers} from "./resolvers.js"; import {bundleStyles, rollupClient} from "./rollup.js"; +import type {Params} from "./route.js"; +import {route} from "./route.js"; import {searchIndex} from "./search.js"; import {Telemetry} from "./telemetry.js"; import {bold, faint, green, link} from "./tty.js"; @@ -114,9 +115,9 @@ export class PreviewServer { const config = await this._readConfig(); const {root, loaders} = config; if (this._verbose) console.log(faint(req.method!), req.url); + const url = new URL(req.url!, "http://localhost"); + let pathname = decodeURI(url.pathname); try { - const url = new URL(req.url!, "http://localhost"); - let pathname = decodeURI(url.pathname); let match: RegExpExecArray | null; if (pathname === "/_observablehq/client.js") { end(req, res, await rollupClient(getClientPath("preview.js"), root, pathname), "text/javascript"); @@ -137,50 +138,32 @@ export class PreviewServer { send(req, pathname, {root: join(root, ".observablehq", "cache")}).pipe(res); } else if (pathname.startsWith("/_import/")) { const path = pathname.slice("/_import".length); - const filepath = join(root, path); - try { - if (pathname.endsWith(".css")) { - await access(filepath, constants.R_OK); - end(req, res, await bundleStyles({path: filepath}), "text/css"); + if (pathname.endsWith(".css")) { + const module = route(root, path.slice(0, -".css".length), [".css"]); + if (module) { + const sourcePath = join(root, path); + await access(sourcePath, constants.R_OK); + end(req, res, await bundleStyles({path: sourcePath}), "text/css"); return; - } else if (pathname.endsWith(".js")) { - const input = await readJavaScript(join(root, path)); - const output = await transpileModule(input, {root, path}); + } + } else if (pathname.endsWith(".js")) { + const module = findModule(root, path); + if (module) { + const input = await readJavaScript(join(root, module.path)); + const output = await transpileModule(input, {root, path, params: module.params}); end(req, res, output, "text/javascript"); return; } - } catch (error) { - if (!isEnoent(error)) throw error; } - throw new HttpError(`Not found: ${pathname}`, 404); + throw enoent(path); } else if (pathname.startsWith("/_file/")) { - const path = pathname.slice("/_file".length); - const filepath = join(root, path); - try { - await access(filepath, constants.R_OK); - send(req, pathname.slice("/_file".length), {root}).pipe(res); - return; - } catch (error) { - if (!isEnoent(error)) throw error; - } - - // Look for a data loader for this file. - const loader = loaders.find(path); - if (loader) { - try { - send(req, await loader.load(), {root}).pipe(res); - return; - } catch (error) { - if (!isEnoent(error)) throw error; - } - } - throw new HttpError(`Not found: ${pathname}`, 404); + send(req, await loaders.loadFile(pathname.slice("/_file".length)), {root}).pipe(res); } else { if ((pathname = normalize(pathname)).startsWith("..")) throw new Error("Invalid path: " + pathname); // Normalize the pathname (e.g., adding ".html" if cleanUrls is false, // dropping ".html" if cleanUrls is true) and redirect if necessary. - const normalizedPathname = config.normalizePath(pathname); + const normalizedPathname = encodeURI(config.normalizePath(pathname)); if (url.pathname !== normalizedPathname) { res.writeHead(302, {Location: normalizedPathname + url.search}); res.end(); @@ -195,29 +178,28 @@ export class PreviewServer { // Lastly, serve the corresponding Markdown file, if it exists. // Anything else should 404; static files should be matched above. - try { - const options = {...config, path: pathname, preview: true}; - const source = await readFile(join(root, pathname + ".md"), "utf8"); - const parse = parseMarkdown(source, options); - const html = await renderPage(parse, options); - end(req, res, html, "text/html"); - } catch (error) { - if (!isEnoent(error)) throw error; // internal error - throw new HttpError("Not found", 404); - } + const options = {...config, path: pathname, preview: true}; + const parse = await loaders.loadPage(pathname, options); + end(req, res, await renderPage(parse, options), "text/html"); } } catch (error) { - if (isHttpError(error)) { + if (isEnoent(error)) { + res.statusCode = 404; + } else if (isHttpError(error)) { res.statusCode = error.statusCode; } else { res.statusCode = 500; console.error(error); } if (req.method === "GET" && res.statusCode === 404) { + if (req.url?.startsWith("/_file/") || req.url?.startsWith("/_import/")) { + res.setHeader("Content-Type", "text/plain; charset=utf-8"); + res.end("File not found"); + return; + } try { const options = {...config, path: "/404", preview: true}; - const source = await readFile(join(root, "404.md"), "utf8"); - const parse = parseMarkdown(source, options); + const parse = await loaders.loadPage("/404", options); const html = await renderPage(parse, options); end(req, res, html, "text/html"); return; @@ -298,20 +280,20 @@ function handleWatch(socket: WebSocket, req: IncomingMessage, configPromise: Pro let tables: Map | null = null; let stylesheets: string[] | null = null; let configWatcher: FSWatcher | null = null; - let markdownWatcher: FSWatcher | null = null; + let loaderWatcher: FSWatcher | null = null; let attachmentWatcher: FileWatchers | null = null; let emptyTimeout: ReturnType | null = null; console.log(faint("socket open"), req.url); async function watcher(event: WatchEventType, force = false) { - if (!path || !config) throw new Error("not initialized"); - const {root, loaders} = config; + if (path === null || config === null) throw new Error("not initialized"); + const {loaders} = config; switch (event) { case "rename": { - markdownWatcher?.close(); + loaderWatcher?.close(); try { - markdownWatcher = watch(join(root, path), (event) => watcher(event)); + loaderWatcher = loaders.watchPage(path, (event) => watcher(event)); } catch (error) { if (!isEnoent(error)) throw error; console.error(`file no longer exists: ${path}`); @@ -322,8 +304,14 @@ function handleWatch(socket: WebSocket, req: IncomingMessage, configPromise: Pro break; } case "change": { - const source = await readFile(join(root, path), "utf8"); - const page = parseMarkdown(source, {path, ...config}); + let page: MarkdownPage; + try { + page = await loaders.loadPage(path, {path, ...config}); + } catch (error) { + console.error(error); + socket.terminate(); + return; + } // delay to avoid a possibly-empty file if (!force && page.body === "") { if (!emptyTimeout) { @@ -368,17 +356,17 @@ function handleWatch(socket: WebSocket, req: IncomingMessage, configPromise: Pro } async function hello({path: initialPath, hash: initialHash}: {path: string; hash: string}): Promise { - if (markdownWatcher || configWatcher || attachmentWatcher) throw new Error("already watching"); + if (loaderWatcher || configWatcher || attachmentWatcher) throw new Error("already watching"); path = decodeURI(initialPath); - if (!(path = normalize(path)).startsWith("/")) throw new Error("Invalid path: " + initialPath); + if (!(path = normalize(path)).startsWith("/")) throw new Error(`Invalid path: ${initialPath}`); if (path.endsWith("/")) path += "index"; - path = join(dirname(path), basename(path, ".html") + ".md"); + path = join(dirname(path), basename(path, ".html")); config = await configPromise; const {root, loaders, normalizePath} = config; - const source = await readFile(join(root, path), "utf8"); - const page = parseMarkdown(source, {path, ...config}); + const page = await loaders.loadPage(path, {path, ...config}); const resolvers = await getResolvers(page, {root, path, loaders, normalizePath}); - if (resolvers.hash !== initialHash) return void send({type: "reload"}); + if (resolvers.hash === initialHash) send({type: "welcome"}); + else return void send({type: "reload"}); hash = resolvers.hash; html = getHtml(page, resolvers); code = getCode(page, resolvers); @@ -386,7 +374,7 @@ function handleWatch(socket: WebSocket, req: IncomingMessage, configPromise: Pro tables = getTables(page); stylesheets = Array.from(resolvers.stylesheets, resolvers.resolveStylesheet); attachmentWatcher = await loaders.watchFiles(path, getWatchFiles(resolvers), () => watcher("change")); - markdownWatcher = watch(join(root, path), (event) => watcher(event)); + loaderWatcher = loaders.watchPage(path, (event) => watcher(event)); if (config.watchPath) configWatcher = watch(config.watchPath, () => send({type: "reload"})); } @@ -415,9 +403,9 @@ function handleWatch(socket: WebSocket, req: IncomingMessage, configPromise: Pro attachmentWatcher.close(); attachmentWatcher = null; } - if (markdownWatcher) { - markdownWatcher.close(); - markdownWatcher = null; + if (loaderWatcher) { + loaderWatcher.close(); + loaderWatcher = null; } if (configWatcher) { configWatcher.close(); @@ -447,8 +435,8 @@ function getHtml({body}: MarkdownPage, resolvers: Resolvers): HtmlPart[] { return Array.from(document.body.childNodes, serializeHtml).filter((d): d is HtmlPart => d != null); } -function getCode({code}: MarkdownPage, resolvers: Resolvers): Map { - return new Map(code.map((code) => [code.id, transpileCode(code, resolvers)])); +function getCode({code, params}: MarkdownPage, resolvers: Resolvers): Map { + return new Map(code.map((code) => [code.id, transpileCode(code, resolvers, params)])); } // Including the file has as a comment ensures that the code changes when a @@ -456,10 +444,10 @@ function getCode({code}: MarkdownPage, resolvers: Resolvers): Map { diff --git a/src/render.ts b/src/render.ts index be67483ef..926f24e19 100644 --- a/src/render.ts +++ b/src/render.ts @@ -26,7 +26,7 @@ type RenderInternalOptions = | {preview: true}; // preview export async function renderPage(page: MarkdownPage, options: RenderOptions & RenderInternalOptions): Promise { - const {data} = page; + const {data, params} = page; const {base, path, title, preview} = options; const {loaders, resolvers = await getResolvers(page, options)} = options; const {draft = false, sidebar = options.sidebar} = data; @@ -76,7 +76,7 @@ import ${preview || page.code.length ? `{${preview ? "open, " : ""}define} from : "" }${data?.sql ? `\n${registerTables(data.sql, options)}` : ""} ${preview ? `\nopen({hash: ${JSON.stringify(resolvers.hash)}, eval: (body) => eval(body)});\n` : ""}${page.code - .map(({node, id, mode}) => `\n${transpileJavaScript(node, {id, path, mode, resolveImport})}`) + .map(({node, id, mode}) => `\n${transpileJavaScript(node, {id, path, params, mode, resolveImport})}`) .join("")}`)} ${sidebar ? html`\n${await renderSidebar(options, resolvers)}` : ""}${ toc.show ? html`\n${renderToc(findHeaders(page), toc.label)}` : "" diff --git a/src/resolvers.ts b/src/resolvers.ts index e41cf787c..7f473607b 100644 --- a/src/resolvers.ts +++ b/src/resolvers.ts @@ -1,12 +1,12 @@ import {createHash} from "node:crypto"; import {extname, join} from "node:path/posix"; -import type {LoaderResolver} from "./dataloader.js"; import {findAssets} from "./html.js"; import {defaultGlobals} from "./javascript/globals.js"; import {getFileHash, getModuleHash, getModuleInfo} from "./javascript/module.js"; import {getImplicitDependencies, getImplicitDownloads} from "./libraries.js"; import {getImplicitFileImports, getImplicitInputImports} from "./libraries.js"; import {getImplicitStylesheets} from "./libraries.js"; +import type {LoaderResolver} from "./loader.js"; import type {MarkdownPage} from "./markdown.js"; import {extractNodeSpecifier, resolveNodeImport, resolveNodeImports} from "./node.js"; import {extractNpmSpecifier, populateNpmCache, resolveNpmImport, resolveNpmImports} from "./npm.js"; diff --git a/src/rollup.ts b/src/rollup.ts index a05b370a3..d380630a0 100644 --- a/src/rollup.ts +++ b/src/rollup.ts @@ -1,6 +1,5 @@ import {extname, resolve} from "node:path/posix"; import {nodeResolve} from "@rollup/plugin-node-resolve"; -import type {CallExpression} from "acorn"; import {simple} from "acorn-walk"; import {build} from "esbuild"; import type {AstNode, OutputChunk, Plugin, ResolveIdResult} from "rollup"; @@ -155,7 +154,7 @@ function importMetaResolve(path: string, resolveImport: ImportResolver): Plugin name: "resolve-import-meta-resolve", async transform(code) { const program = this.parse(code); - const resolves: CallExpression[] = []; + const resolves: StringLiteral[] = []; simple(program, { CallExpression(node) { @@ -167,7 +166,7 @@ function importMetaResolve(path: string, resolveImport: ImportResolver): Plugin node.arguments.length === 1 && isStringLiteral(node.arguments[0]) ) { - resolves.push(node); + resolves.push(node.arguments[0]); } } }); @@ -175,9 +174,8 @@ function importMetaResolve(path: string, resolveImport: ImportResolver): Plugin if (!resolves.length) return null; const output = new Sourcemap(code); - for (const node of resolves) { - const source = node.arguments[0]; - const specifier = getStringLiteralValue(source as StringLiteral); + for (const source of resolves) { + const specifier = getStringLiteralValue(source); const resolution = await resolveImport(specifier); if (resolution) output.replaceLeft(source.start, source.end, JSON.stringify(relativePath(path, resolution))); } diff --git a/src/route.ts b/src/route.ts new file mode 100644 index 000000000..320755fb7 --- /dev/null +++ b/src/route.ts @@ -0,0 +1,115 @@ +import {existsSync} from "node:fs"; +import {basename, join} from "node:path/posix"; +import {globSync} from "glob"; + +export type Params = {[name: string]: string}; + +export type RouteResult = {path: string; ext: string; params?: Params}; + +export function isParameterizedPath(path: string): boolean { + return path.split("/").some((name) => /\[.+\]/.test(name)); +} + +/** + * Finds a parameterized file (dynamic route). + * + * When searching for a path, we often want to search for several paths + * simultaneously because the path can be backed by several resources + * (particularly for data loaders). + * + * Finding CSS: + * - /path/to/file.css + * + * Finding JS: + * - /path/to/file.js + * - /path/to/file.jsx + * + * Finding a file: + * - /path/to/file.csv + * - /path/to/file.csv.js + * - /path/to/file.csv.py, etc. + * + * Parameterized files can match with different degrees of specificity. For + * example when searching for /path/to/file.css: + * - /path/to/file.css (exact match) + * - /path/to/[param].css + * - /path/[param]/file.css + * - /path/[param1]/[param2].css + * - /[param]/to/file.css + * - /[param1]/to/[param2].css + * - /[param1]/[param2]/file.css + * - /[param1]/[param2]/[param3].css + * + * We want to return the most-specific match, and the specificity of the match + * takes priority over the list of paths. For example, when searching for JS, + * we’d rather return /path/to/file.jsx than /path/to/[param].js. + * + * For data loaders, we’ll also search within archives, but this is lower + * priority than parameterization. So for example when searching for a file, + * we’d use /path/to/[param].csv over /path/to.zip. + */ +export function route(root: string, path: string, exts: string[]): RouteResult | undefined { + for (const ext of exts) if (!ext) throw new Error("empty extension"); + return routeParams(root, ".", join(".", path).split("/"), exts); +} + +/** + * Finds a parameterized file (dynamic route) recursively, such that the most + * specific match is returned. + */ +function routeParams(root: string, cwd: string, parts: string[], exts: string[]): RouteResult | undefined { + switch (parts.length) { + case 0: + return; + case 1: { + const [first] = parts; + for (const ext of exts) { + if (existsSync(join(root, cwd, first + ext))) { + return {path: join(cwd, first + ext), ext}; + } + } + if (first) { + for (const ext of exts) { + for (const file of globSync(`*\\[?*\\]*${ext}`, {cwd: join(root, cwd), nodir: true})) { + const params = matchParams(basename(file, ext), first); + if (params) return {path: join(cwd, file), params: {...params}, ext}; + } + } + } + return; + } + default: { + const [first, ...rest] = parts; + if (existsSync(join(root, cwd, first))) { + const found = routeParams(root, join(cwd, first), rest, exts); + if (found) return found; + } + if (first) { + for (const dir of globSync("*\\[?*\\]*/", {cwd: join(root, cwd)})) { + const params = matchParams(dir, first); + if (!params) continue; + const found = routeParams(root, join(cwd, dir), rest, exts); + if (found) return {...found, params: {...found.params, ...params}}; + } + } + } + } +} + +function matchParams(file: string, input: string): Params | undefined { + return compilePattern(file).exec(input)?.groups; +} + +function compilePattern(file: string): RegExp { + let pattern = "^"; + let i = 0; + for (let match: RegExpExecArray | null, re = /\[([a-z_]\w*)\]/gi; (match = re.exec(file)); i = re.lastIndex) { + pattern += `${requote(file.slice(i, match.index))}(?<${match[1]}>.+)`; + } + pattern += `${requote(file.slice(i))}$`; + return new RegExp(pattern, "i"); +} + +function requote(text: string): string { + return text.replace(/[\\^$*+?|[\]().{}]/g, "\\$&"); +} diff --git a/src/search.ts b/src/search.ts index f915f6f2b..514ea7abb 100644 --- a/src/search.ts +++ b/src/search.ts @@ -1,11 +1,7 @@ -import {readFile} from "node:fs/promises"; -import {basename, dirname, join} from "node:path/posix"; import he from "he"; import MiniSearch from "minisearch"; import type {Config, SearchResult} from "./config.js"; -import {visitMarkdownFiles} from "./files.js"; -import type {Logger} from "./logger.js"; -import {parseMarkdown} from "./markdown.js"; +import type {Logger, Writer} from "./logger.js"; import {faint, strikethrough} from "./tty.js"; // Avoid reindexing too often in preview. @@ -14,9 +10,13 @@ const reindexDelay = 10 * 60 * 1000; // 10 minutes export interface SearchIndexEffects { logger: Logger; + output: Writer; } -const defaultEffects: SearchIndexEffects = {logger: console}; +const defaultEffects: SearchIndexEffects = { + logger: console, + output: process.stdout +}; const indexOptions = { fields: ["title", "text", "keywords"], // fields to return with search results @@ -57,7 +57,7 @@ export async function searchIndex(config: Config, effects = defaultEffects): Pro } async function* indexPages(config: Config, effects: SearchIndexEffects): AsyncIterable { - const {root, pages} = config; + const {pages, loaders} = config; // Get all the listed pages (which are indexed by default) const pagePaths = new Set(["/index"]); @@ -66,18 +66,15 @@ async function* indexPages(config: Config, effects: SearchIndexEffects): AsyncIt if ("pages" in p) for (const {path} of p.pages) pagePaths.add(path); } - for (const file of visitMarkdownFiles(root)) { - const sourcePath = join(root, file); - const source = await readFile(sourcePath, "utf8"); - const path = `/${join(dirname(file), basename(file, ".md"))}`; - const {body, title, data} = parseMarkdown(source, {...config, path}); + for await (const path of config.paths()) { + const {body, title, data} = await loaders.loadPage(path, {...config, path}); // Skip pages that opt-out of indexing, and skip unlisted pages unless // opted-in. We only log the first case. const listed = pagePaths.has(path); const indexed = data?.index === undefined ? listed : Boolean(data.index); if (!indexed) { - if (listed) effects.logger.log(`${faint("index")} ${strikethrough(sourcePath)} ${faint("(skipped)")}`); + if (listed) effects.logger.log(`${faint("index")} ${strikethrough(path)} ${faint("(skipped)")}`); continue; } @@ -93,7 +90,7 @@ async function* indexPages(config: Config, effects: SearchIndexEffects): AsyncIt .replaceAll(/[\u0300-\u036f]/g, "") .replace(/[^\p{L}\p{N}]/gu, " "); // keep letters & numbers - effects.logger.log(`${faint("index")} ${sourcePath}`); + effects.logger.log(`${faint("index")} ${path}`); yield {path, title, text, keywords: normalizeKeywords(data?.keywords)}; } } diff --git a/src/sourcemap.ts b/src/sourcemap.ts index dd7fb1274..4d2a7c01b 100644 --- a/src/sourcemap.ts +++ b/src/sourcemap.ts @@ -39,6 +39,15 @@ export class Sourcemap { } return lo; } + private _subsume(start: number, end: number): void { + let n = 0; + for (let i = 0; i < this._edits.length; ++i) { + const e = this._edits[i]; + if (start <= e.start && e.end < end) continue; + this._edits[n++] = e; + } + this._edits.length = n; + } insertLeft(index: number, value: string): typeof this { return this.replaceLeft(index, index, value); } @@ -49,10 +58,14 @@ export class Sourcemap { return this.replaceRight(start, end, ""); } replaceLeft(start: number, end: number, value: string): typeof this { - return this._edits.splice(this._bisectLeft(start), 0, {start, end, value}), this; + this._subsume(start, end); + this._edits.splice(this._bisectLeft(start), 0, {start, end, value}); + return this; } replaceRight(start: number, end: number, value: string): typeof this { - return this._edits.splice(this._bisectRight(start), 0, {start, end, value}), this; + this._subsume(start, end); + this._edits.splice(this._bisectRight(start), 0, {start, end, value}); + return this; } translate(position: Position): Position { let index = 0; diff --git a/test/build-test.ts b/test/build-test.ts index c6c860dc5..09698e52d 100644 --- a/test/build-test.ts +++ b/test/build-test.ts @@ -61,12 +61,12 @@ describe("build", () => { // renumber the hashes so they are sequential. This way we don’t have to // update the test snapshots whenever Framework’s client code changes. We // make an exception for minisearch.json because to test the content. - for (const path of findFiles(join(actualDir, "_observablehq"))) { + for (const path of findFiles(join(outputDir, "_observablehq"))) { const match = /^((.+)\.[0-9a-f]{8})\.(\w+)$/.exec(path); if (!match) throw new Error(`no hash found: ${path}`); const [, key, name, ext] = match; - const oldPath = join(actualDir, "_observablehq", path); - const newPath = join(actualDir, "_observablehq", `${name}.${normalizeHash(key)}.${ext}`); + const oldPath = join(outputDir, "_observablehq", path); + const newPath = join(outputDir, "_observablehq", `${name}.${normalizeHash(key)}.${ext}`); if (/^minisearch\.[0-9a-f]{8}\.json$/.test(path)) { await rename(oldPath, newPath); } else { @@ -76,10 +76,10 @@ describe("build", () => { } // Replace any reference to re-numbered files in _observablehq. - for (const path of findFiles(actualDir)) { - const actual = await readFile(join(actualDir, path), "utf8"); + for (const path of findFiles(outputDir)) { + const actual = await readFile(join(outputDir, path), "utf8"); const normalized = actual.replace(/\/_observablehq\/((.+)\.[0-9a-f]{8})\.(\w+)\b/g, (match, key, name, ext) => `/_observablehq/${name}.${normalizeHash(key)}.${ext}`); // prettier-ignore - if (normalized !== actual) await writeFile(join(actualDir, path), normalized); + if (normalized !== actual) await writeFile(join(outputDir, path), normalized); } if (generate) return; diff --git a/test/config-test.ts b/test/config-test.ts index 462fd5031..6de7a1f3b 100644 --- a/test/config-test.ts +++ b/test/config-test.ts @@ -2,12 +2,12 @@ import assert from "node:assert"; import {resolve} from "node:path"; import MarkdownIt from "markdown-it"; import {normalizeConfig as config, mergeToc, readConfig, setCurrentDate} from "../src/config.js"; -import {LoaderResolver} from "../src/dataloader.js"; +import {LoaderResolver} from "../src/loader.js"; describe("readConfig(undefined, root)", () => { before(() => setCurrentDate(new Date("2024-01-10T16:00:00"))); it("imports the config file at the specified root", async () => { - const {md, loaders, normalizePath, ...config} = await readConfig(undefined, "test/input/build/config"); + const {md, loaders, paths, normalizePath, ...config} = await readConfig(undefined, "test/input/build/config"); assert(md instanceof MarkdownIt); assert(loaders instanceof LoaderResolver); assert.strictEqual(typeof normalizePath, "function"); @@ -46,7 +46,7 @@ describe("readConfig(undefined, root)", () => { }); }); it("returns the default config if no config file is found", async () => { - const {md, loaders, normalizePath, ...config} = await readConfig(undefined, "test/input/build/simple"); + const {md, loaders, paths, normalizePath, ...config} = await readConfig(undefined, "test/input/build/simple"); assert(md instanceof MarkdownIt); assert(loaders instanceof LoaderResolver); assert.strictEqual(typeof normalizePath, "function"); diff --git a/test/fileWatchers-test.ts b/test/fileWatchers-test.ts index 542356e92..abfbb12b1 100644 --- a/test/fileWatchers-test.ts +++ b/test/fileWatchers-test.ts @@ -3,7 +3,7 @@ import {mkdirSync, renameSync, unlinkSync, utimesSync, writeFileSync} from "node import {basename, dirname, extname, join} from "node:path/posix"; import {InternSet, difference} from "d3-array"; import {temporaryDirectoryTask} from "tempy"; -import {LoaderResolver} from "../src/dataloader.js"; +import {LoaderResolver} from "../src/loader.js"; import {resolvePath} from "../src/path.js"; describe("FileWatchers.of(loaders, path, names, callback)", () => { diff --git a/test/input/build/params/[dir]/index.md b/test/input/build/params/[dir]/index.md new file mode 100644 index 000000000..aa404ca0e --- /dev/null +++ b/test/input/build/params/[dir]/index.md @@ -0,0 +1,5 @@ +# Hello, dir ${observable.params.dir} + +```js echo +observable.params.dir +``` diff --git a/test/input/build/params/[dir]/loaded.md.js b/test/input/build/params/[dir]/loaded.md.js new file mode 100644 index 000000000..12927d51e --- /dev/null +++ b/test/input/build/params/[dir]/loaded.md.js @@ -0,0 +1,14 @@ +import {parseArgs} from "node:util"; + +const { + values: {dir} +} = parseArgs({ + options: {dir: {type: "string"}} +}); + +process.stdout.write(`# Hello ${dir} + +~~~js +observable.params.dir +~~~ +`); diff --git a/test/input/build/params/foo/[param].md b/test/input/build/params/foo/[param].md new file mode 100644 index 000000000..c7c2fa264 --- /dev/null +++ b/test/input/build/params/foo/[param].md @@ -0,0 +1,5 @@ +# Hello, param ${observable.params.param} + +```js echo +observable.params.param +``` diff --git a/test/input/build/params/observablehq.config.js b/test/input/build/params/observablehq.config.js new file mode 100644 index 000000000..0a180653d --- /dev/null +++ b/test/input/build/params/observablehq.config.js @@ -0,0 +1,5 @@ +export default { + async *dynamicPaths() { + yield* ["/bar/index", "/bar/loaded", "/foo/bar", "/foo/index"]; + } +}; diff --git a/test/input/params/[dir]/[file].md b/test/input/params/[dir]/[file].md new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/[dir]/foo.md b/test/input/params/[dir]/foo.md new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/[file]-suffix.js b/test/input/params/[file]-suffix.js new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/[file].csv.js b/test/input/params/[file].csv.js new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/[file].md b/test/input/params/[file].md new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/[period]-[number].json.js b/test/input/params/[period]-[number].json.js new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/foo.md b/test/input/params/foo.md new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/foo/[file].md b/test/input/params/foo/[file].md new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/foo/foo.md b/test/input/params/foo/foo.md new file mode 100644 index 000000000..e69de29bb diff --git a/test/input/params/prefix-[file].js b/test/input/params/prefix-[file].js new file mode 100644 index 000000000..e69de29bb diff --git a/test/javascript/imports-test.ts b/test/javascript/imports-test.ts index 23f25f0e3..a1f641129 100644 --- a/test/javascript/imports-test.ts +++ b/test/javascript/imports-test.ts @@ -93,7 +93,7 @@ describe("findImports(body, path, input)", () => { ]); }); it("ignores import expressions with dynamic sources", () => { - const input = "import('./bar'+'.js');\nimport(`/${'baz'}.js`);\n"; + const input = "import(bar+'.js');\nimport(`/${baz}.js`);\n"; const program = parse(input); assert.deepStrictEqual(findImports(program, "foo/bar.js", input), []); }); diff --git a/test/javascript/source-test.ts b/test/javascript/source-test.ts index 16812789b..c5458cd1f 100644 --- a/test/javascript/source-test.ts +++ b/test/javascript/source-test.ts @@ -39,9 +39,17 @@ describe("isStringLiteral(node)", () => { assert.strictEqual(isStringLiteral(parseExpression("'foo'")), true); assert.strictEqual(isStringLiteral(parseExpression("`foo`")), true); }); + it("allows binary expressions of string literals", () => { + assert.strictEqual(isStringLiteral(parseExpression("'1' + '2'")), true); + assert.strictEqual(isStringLiteral(parseExpression("'1' + `${'2'}`")), true); + }); + it("allows template literals of string literals", () => { + assert.strictEqual(isStringLiteral(parseExpression("`${'1'}${'2'}`")), true); + assert.strictEqual(isStringLiteral(parseExpression("`${'1'+'2'}`")), true); + }); it("returns false for other nodes", () => { assert.strictEqual(isStringLiteral(parseExpression("42")), false); - assert.strictEqual(isStringLiteral(parseExpression("'1' + '2'")), false); + assert.strictEqual(isStringLiteral(parseExpression("1 + 2")), false); assert.strictEqual(isStringLiteral(parseExpression("`${42}`")), false); }); }); diff --git a/test/javascript/transpile-test.ts b/test/javascript/transpile-test.ts index 2481585e6..382a1fe5d 100644 --- a/test/javascript/transpile-test.ts +++ b/test/javascript/transpile-test.ts @@ -176,7 +176,7 @@ describe("transpileModule(input, root, path)", () => { assert.strictEqual(output, 'FileAttachment("../test.txt", import.meta.url)'); }); it("throws a syntax error with non-literal calls", async () => { - const input = "import {FileAttachment} from \"npm:@observablehq/stdlib\";\nFileAttachment(`./${'test'}.txt`)"; + const input = 'import {FileAttachment} from "npm:@observablehq/stdlib";\nFileAttachment(`./${test}.txt`)'; await assert.rejects(() => transpileModule(input, options), /FileAttachment requires a single literal string/); // prettier-ignore }); it("throws a syntax error with URL fetches", async () => { diff --git a/test/dataloaders-test.ts b/test/loader-test.ts similarity index 84% rename from test/dataloaders-test.ts rename to test/loader-test.ts index ec51fadcc..22d776194 100644 --- a/test/dataloaders-test.ts +++ b/test/loader-test.ts @@ -1,8 +1,10 @@ import assert from "node:assert"; import {mkdir, readFile, rm, stat, unlink, utimes, writeFile} from "node:fs/promises"; import os from "node:os"; -import type {LoadEffects} from "../src/dataloader.js"; -import {LoaderResolver} from "../src/dataloader.js"; +import {join} from "node:path/posix"; +import {clearFileInfo} from "../src/javascript/module.js"; +import type {LoadEffects} from "../src/loader.js"; +import {LoaderResolver} from "../src/loader.js"; const noopEffects: LoadEffects = { logger: {log() {}, warn() {}, error() {}}, @@ -12,34 +14,34 @@ const noopEffects: LoadEffects = { describe("LoaderResolver.find(path)", () => { const loaders = new LoaderResolver({root: "test"}); it("a .js data loader is called with node", async () => { - const loader = loaders.find("dataloaders/data1.txt")!; + const loader = loaders.find("/dataloaders/data1.txt")!; const out = await loader.load(noopEffects); assert.strictEqual(await readFile("test/" + out, "utf-8"), "node\n"); }); it("a .ts data loader is called with tsx", async () => { - const loader = loaders.find("dataloaders/data2.txt")!; + const loader = loaders.find("/dataloaders/data2.txt")!; const out = await loader.load(noopEffects); assert.strictEqual(await readFile("test/" + out, "utf-8"), "tsx\n"); }); it("a .sh data loader is called with sh", async function () { if (os.platform() === "win32") this.skip(); - const loader = loaders.find("dataloaders/data3.txt")!; + const loader = loaders.find("/dataloaders/data3.txt")!; const out = await loader.load(noopEffects); assert.strictEqual(await readFile("test/" + out, "utf-8"), "shell\n"); }); it("a .exe data loader is invoked directly", async () => { - const loader = loaders.find("dataloaders/data4.txt")!; + const loader = loaders.find("/dataloaders/data4.txt")!; const out = await loader.load(noopEffects); assert.strictEqual(await readFile("test/" + out, "utf-8"), `python3${os.EOL}`); }); it("a .py data loader is called with python3", async () => { - const loader = loaders.find("dataloaders/data5.txt")!; + const loader = loaders.find("/dataloaders/data5.txt")!; const out = await loader.load(noopEffects); assert.strictEqual(await readFile("test/" + out, "utf-8"), `python3${os.EOL}`); }); // Skipping because this requires R to be installed (which is slow in CI). it.skip("a .R data loader is called with Rscript", async () => { - const loader = loaders.find("dataloaders/data6.txt")!; + const loader = loaders.find("/dataloaders/data6.txt")!; const out = await loader.load(noopEffects); assert.strictEqual(await readFile("test/" + out, "utf-8"), "Rscript\n"); }); @@ -57,12 +59,13 @@ describe("LoaderResolver.find(path, {useStale: true})", () => { } } }; - const loader = loaders.find("dataloaders/data1.txt")!; + const loader = loaders.find("/dataloaders/data1.txt")!; + const loaderPath = join(loader.root, loader.path); // save the loader times. - const {atime, mtime} = await stat(loader.path); + const {atime, mtime} = await stat(loaderPath); // set the loader mtime to Dec. 1st, 2023. const time = new Date(2023, 11, 1); - await utimes(loader.path, atime, time); + await utimes(loaderPath, atime, time); // remove the cache set by another test (unless we it.only this test). try { await unlink("test/.observablehq/cache/dataloaders/data1.txt"); @@ -74,25 +77,25 @@ describe("LoaderResolver.find(path, {useStale: true})", () => { // run again (fresh) await loader.load(outputEffects); // touch the loader - await utimes(loader.path, atime, new Date(Date.now() + 100)); + await utimes(loaderPath, atime, new Date(Date.now() + 100)); // run it with useStale=true (using stale) - const loader2 = loaders.find("dataloaders/data1.txt", {useStale: true})!; + const loader2 = loaders.find("/dataloaders/data1.txt", {useStale: true})!; await loader2.load(outputEffects); // run it with useStale=false (stale) await loader.load(outputEffects); // revert the loader to its original mtime - await utimes(loader.path, atime, mtime); + await utimes(loaderPath, atime, mtime); assert.deepStrictEqual( // eslint-disable-next-line no-control-regex out.map((l) => l.replaceAll(/\x1b\[[0-9]+m/g, "")), [ - "load test/dataloaders/data1.txt.js → ", + "load /dataloaders/data1.txt → ", "[missing] ", - "load test/dataloaders/data1.txt.js → ", + "load /dataloaders/data1.txt → ", "[fresh] ", - "load test/dataloaders/data1.txt.js → ", + "load /dataloaders/data1.txt → ", "[using stale] ", - "load test/dataloaders/data1.txt.js → ", + "load /dataloaders/data1.txt → ", "[stale] " ] ); @@ -104,6 +107,8 @@ describe("LoaderResolver.getSourceFileHash(path)", () => { it("returns the content hash for the specified file’s data loader", async () => { await utimes("test/input/build/archives.posix/dynamic.zip.sh", time, time); await utimes("test/input/build/archives.posix/static.zip", time, time); + clearFileInfo("test/input/build/archives.posix", "dynamic.zip.sh"); + clearFileInfo("test/input/build/archives.posix", "static.zip"); const loaders = new LoaderResolver({root: "test/input/build/archives.posix"}); assert.strictEqual(loaders.getSourceFileHash("dynamic.zip.sh"), "516cec2431ce8f1181a7a2a161db8bdfcaea132d3b2c37f863ea6f05d64d1d10"); // prettier-ignore assert.strictEqual(loaders.getSourceFileHash("dynamic.zip"), "64acd011e27907a2594fda3272bfc951e75db4c80495ce41e84ced61383bbb60"); // prettier-ignore diff --git a/test/npm-test.ts b/test/npm-test.ts index 78d4d4f33..aa9067830 100644 --- a/test/npm-test.ts +++ b/test/npm-test.ts @@ -134,14 +134,14 @@ describe("rewriteNpmImports(input, resolve)", () => { assert.strictEqual(rewriteNpmImports("import('/npm/d3-array@3.2.4/+esm');\n", (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), 'import("../d3-array@3.2.4/_esm.js");\n'); }); it("ignores dynamic imports with dynamic module specifiers", () => { - assert.strictEqual(rewriteNpmImports('import(`/npm/d3-array@${"3.2.4"}/+esm`);\n', (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), 'import(`/npm/d3-array@${"3.2.4"}/+esm`);\n'); + assert.strictEqual(rewriteNpmImports("import(`/npm/d3-array@${version}/+esm`);\n", (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), "import(`/npm/d3-array@${version}/+esm`);\n"); }); it("ignores dynamic imports with dynamic module specifiers", () => { - assert.strictEqual(rewriteNpmImports('import(`/npm/d3-array@${"3.2.4"}/+esm`);\n', (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), 'import(`/npm/d3-array@${"3.2.4"}/+esm`);\n'); + assert.strictEqual(rewriteNpmImports("import(`/npm/d3-array@${version}/+esm`);\n", (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), "import(`/npm/d3-array@${version}/+esm`);\n"); }); it("strips the sourceMappingURL declaration", () => { - assert.strictEqual(rewriteNpmImports('import(`/npm/d3-array@${"3.2.4"}/+esm`);\n//# sourceMappingURL=index.js.map', (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), 'import(`/npm/d3-array@${"3.2.4"}/+esm`);\n'); - assert.strictEqual(rewriteNpmImports('import(`/npm/d3-array@${"3.2.4"}/+esm`);\n//# sourceMappingURL=index.js.map\n', (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), 'import(`/npm/d3-array@${"3.2.4"}/+esm`);\n'); + assert.strictEqual(rewriteNpmImports("import(`/npm/d3-array@3.2.4/+esm`);\n//# sourceMappingURL=index.js.map", (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), 'import("../d3-array@3.2.4/_esm.js");\n'); + assert.strictEqual(rewriteNpmImports("import(`/npm/d3-array@3.2.4/+esm`);\n//# sourceMappingURL=index.js.map\n", (v) => resolve("/_npm/d3@7.8.5/_esm.js", v)), 'import("../d3-array@3.2.4/_esm.js");\n'); }); }); diff --git a/test/output/build/params/_observablehq/client.00000001.js b/test/output/build/params/_observablehq/client.00000001.js new file mode 100644 index 000000000..e69de29bb diff --git a/test/output/build/params/_observablehq/runtime.00000002.js b/test/output/build/params/_observablehq/runtime.00000002.js new file mode 100644 index 000000000..e69de29bb diff --git a/test/output/build/params/_observablehq/stdlib.00000003.js b/test/output/build/params/_observablehq/stdlib.00000003.js new file mode 100644 index 000000000..e69de29bb diff --git a/test/output/build/params/_observablehq/theme-air,near-midnight.00000004.css b/test/output/build/params/_observablehq/theme-air,near-midnight.00000004.css new file mode 100644 index 000000000..e69de29bb diff --git a/test/output/build/params/bar/index.html b/test/output/build/params/bar/index.html new file mode 100644 index 000000000..800eff631 --- /dev/null +++ b/test/output/build/params/bar/index.html @@ -0,0 +1,45 @@ + + + + +Hello, dir + + + + + + + + + + +
+
+

Hello, dir

+
+
observable.params.dir
+
+
+ +
diff --git a/test/output/build/params/bar/loaded.html b/test/output/build/params/bar/loaded.html new file mode 100644 index 000000000..47b4f2ea6 --- /dev/null +++ b/test/output/build/params/bar/loaded.html @@ -0,0 +1,37 @@ + + + + +Hello bar + + + + + + + + + + +
+
+

Hello bar

+
+
+ +
diff --git a/test/output/build/params/foo/bar.html b/test/output/build/params/foo/bar.html new file mode 100644 index 000000000..6b85008f0 --- /dev/null +++ b/test/output/build/params/foo/bar.html @@ -0,0 +1,45 @@ + + + + +Hello, param + + + + + + + + + + +
+
+

Hello, param

+
+
observable.params.param
+
+
+ +
diff --git a/test/output/build/params/foo/index.html b/test/output/build/params/foo/index.html new file mode 100644 index 000000000..db27fa589 --- /dev/null +++ b/test/output/build/params/foo/index.html @@ -0,0 +1,45 @@ + + + + +Hello, param + + + + + + + + + + +
+
+

Hello, param

+
+
observable.params.param
+
+
+ +
diff --git a/test/preview/preview-test.ts b/test/preview/preview-test.ts index 136ea2c8d..dfa8a72cd 100644 --- a/test/preview/preview-test.ts +++ b/test/preview/preview-test.ts @@ -59,7 +59,7 @@ describe("preview server", () => { it("handles missing imports", async () => { const res = await chai.request(testServerUrl).get("/_import/idontexist.js"); expect(res).to.have.status(404); - expect(res.text).to.have.string("404 page"); + expect(res.text).to.have.string("File not found"); }); it("serves local files", async () => { @@ -71,6 +71,6 @@ describe("preview server", () => { it("handles missing files", async () => { const res = await chai.request(testServerUrl).get("/_file/idontexist.csv"); expect(res).to.have.status(404); - expect(res.text).to.have.string("404 page"); + expect(res.text).to.have.string("File not found"); }); }); diff --git a/test/route-test.ts b/test/route-test.ts new file mode 100644 index 000000000..8960fa372 --- /dev/null +++ b/test/route-test.ts @@ -0,0 +1,81 @@ +import assert from "node:assert"; +import {isParameterizedPath, route} from "../src/route.js"; + +describe("isParameterizedPath(path)", () => { + it("returns true for a parameterized file name", () => { + assert.strictEqual(isParameterizedPath("/[file].md"), true); + assert.strictEqual(isParameterizedPath("/prefix-[file].md"), true); + assert.strictEqual(isParameterizedPath("/[file]-suffix.md"), true); + assert.strictEqual(isParameterizedPath("/[file]-[number].md"), true); + assert.strictEqual(isParameterizedPath("/path/[file].md"), true); + assert.strictEqual(isParameterizedPath("/path/to/[file].md"), true); + assert.strictEqual(isParameterizedPath("/path/[dir]/[file].md"), true); + }); + it("returns true for a parameterized directory name", () => { + assert.strictEqual(isParameterizedPath("/[dir]/file.md"), true); + assert.strictEqual(isParameterizedPath("/prefix-[dir]/file.md"), true); + assert.strictEqual(isParameterizedPath("/[dir]-suffix/file.md"), true); + assert.strictEqual(isParameterizedPath("/[dir]-[number]/file.md"), true); + assert.strictEqual(isParameterizedPath("/path/[dir]/file.md"), true); + assert.strictEqual(isParameterizedPath("/[dir1]/[dir2]/file.md"), true); + }); + it("doesn’t consider an empty parameter to be valid", () => { + assert.strictEqual(isParameterizedPath("/[]/file.md"), false); + assert.strictEqual(isParameterizedPath("/path/to/[].md"), false); + }); + it("returns false for a non-parameterized path", () => { + assert.strictEqual(isParameterizedPath("/file.md"), false); + assert.strictEqual(isParameterizedPath("/path/to/file.md"), false); + }); +}); + +describe("route(root, path, exts)", () => { + it("finds an exact file", () => { + assert.deepStrictEqual(route("test/input/build/simple", "simple", [".md"]), {path: "simple.md", ext: ".md"}); // prettier-ignore + }); + it("finds an exact file with multiple extensions", () => { + assert.deepStrictEqual(route("test/input/build/simple", "data", [".txt", ".txt.js", ".txt.py"]), {path: "data.txt.js", ext: ".txt.js"}); // prettier-ignore + }); + it("finds a parameterized file", () => { + assert.deepStrictEqual(route("test/input/params", "bar", [".md"]), {path: "[file].md", ext: ".md", params: {file: "bar"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "baz", [".md"]), {path: "[file].md", ext: ".md", params: {file: "baz"}}); // prettier-ignore + }); + it("finds a parameterized file with multiple extensions", () => { + assert.deepStrictEqual(route("test/input/params", "data", [".csv", ".csv.js", ".csv.py"]), {path: "[file].csv.js", ext: ".csv.js", params: {file: "data"}}); // prettier-ignore + }); + it("finds a non-parameterized file ahead of a parameterized file", () => { + assert.deepStrictEqual(route("test/input/params", "foo", [".md"]), {path: "foo.md", ext: ".md"}); // prettier-ignore + }); + it("finds the most-specific parameterized match", () => { + assert.deepStrictEqual(route("test/input/params", "foo/foo", [".md"]), {path: "foo/foo.md", ext: ".md"}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "foo/bar", [".md"]), {path: "foo/[file].md", ext: ".md", params: {file: "bar"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "foo/baz", [".md"]), {path: "foo/[file].md", ext: ".md", params: {file: "baz"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "bar/foo", [".md"]), {path: "[dir]/foo.md", ext: ".md", params: {dir: "bar"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "bar/bar", [".md"]), {path: "[dir]/[file].md", ext: ".md", params: {dir: "bar", file: "bar"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "bar/baz", [".md"]), {path: "[dir]/[file].md", ext: ".md", params: {dir: "bar", file: "baz"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "baz/foo", [".md"]), {path: "[dir]/foo.md", ext: ".md", params: {dir: "baz"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "baz/bar", [".md"]), {path: "[dir]/[file].md", ext: ".md", params: {dir: "baz", file: "bar"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "baz/baz", [".md"]), {path: "[dir]/[file].md", ext: ".md", params: {dir: "baz", file: "baz"}}); // prettier-ignore + }); + it("finds a partially-parameterized match", () => { + assert.deepStrictEqual(route("test/input/params", "prefix-foo", [".js"]), {path: "prefix-[file].js", ext: ".js", params: {file: "foo"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "foo-suffix", [".js"]), {path: "[file]-suffix.js", ext: ".js", params: {file: "foo"}}); // prettier-ignore + }); + it("finds a multi-parameterized match", () => { + assert.deepStrictEqual(route("test/input/params", "day-14", [".json.js"]), {path: "[period]-[number].json.js", ext: ".json.js", params: {period: "day", number: "14"}}); // prettier-ignore + assert.deepStrictEqual(route("test/input/params", "week-4", [".json.js"]), {path: "[period]-[number].json.js", ext: ".json.js", params: {period: "week", number: "4"}}); // prettier-ignore + }); + it("returns undefined when there is no match", () => { + assert.strictEqual(route("test/input/build/simple", "not-found", [".md"]), undefined); + assert.strictEqual(route("test/input/build/simple", "simple", [".js"]), undefined); + assert.strictEqual(route("test/input/params", "foo/bar/baz", [".md"]), undefined); + }); + it("does not allow an empty match", () => { + assert.deepStrictEqual(route("test/input/params", "foo/", [".md"]), undefined); + assert.deepStrictEqual(route("test/input/params", "bar/", [".md"]), undefined); + }); + it("does not allow the empty extension", () => { + assert.throws(() => route("test/input/build/simple", "simple.md", [""]), /empty extension/); + assert.throws(() => route("test/input/build/simple", "data.txt", ["", ".js", ".py"]), /empty extension/); + }); +}); diff --git a/test/sourcemap-test.ts b/test/sourcemap-test.ts index 04d726402..b99a5530a 100644 --- a/test/sourcemap-test.ts +++ b/test/sourcemap-test.ts @@ -97,4 +97,12 @@ describe("new Sourcemap(source)", () => { sm.trim(); assert.strictEqual(sm.toString(), "hello;"); }); + it("replace a replacement", () => { + const input = "FileAttachment(`${observable.params.foo}.json`)"; + const sourcemap = new Sourcemap(input); + sourcemap.replaceLeft(18, 39, '"foo"'); + assert.strictEqual(sourcemap.toString(), 'FileAttachment(`${"foo"}.json`)'); + sourcemap.replaceLeft(15, 46, '"./foo.json"'); + assert.strictEqual(sourcemap.toString(), 'FileAttachment("./foo.json")'); + }); });