Plugin API

[claytercek]

Dumping some ideas I've been mulling over while using launchpad lately.

I'm wondering if it might be worth taking #65/#70 a step further and abandoning JSON configs altogether, rearchitecting the config (and launchpad core) to use a plugin API inspired by some common js build tools and frameworks like vite, rollup, webpack, etc.

Edit 05/10: The original proposal was vague on the role of content and monitor as plugins vs higher-level config options. After some thought, it seems like the latter makes more sense. I've updated the proposal to reflect that.

API Proposal

Basic Plugins

A simple plugin would look something like this:

{
  name: "simple-plugin",
  hookName: (event, context, next) => {
    // do something
  },
  postHookName: async (event, context, next) => {
    // do something
  },
}

// or maybe this?

{
  name: "simple-plugin",
  initialize: (emitter) => {
    emitter.once("hook-name", (event, context, next) => {
      // do something
    }
    emitter.on("post-hook-name", (event, context, next) => {
      // do something
    }
  },
}

Both options show basically the same thing:

• a name property, used for logging
• a method for hooking into some defined launchpad lifecycle events

Complex Plugins

In addition to this, a plugin could also be a function that returns or resolves an object with the same shape, allowing for more complex plugins that need to do some async work before returning the plugin object, or that need to be configured with some options:

function complexPlugin(options) {
  return {
    name: "complex-plugin",
    hookName: (event, context, next) => {
      if (options.someOption) {
        // do something
      }
    },
    postHookName: (event, context, next) => {
      // do something
    },
  };
}

Usage in Launchpad

I think it's worth thinking about launchpad as having two different lifecycles: one for content and one for monitor, each with their own set of events. Plugins would be focused on one or the other, not both.

The config for content and monitor would look largely the same, though each could have their own set of plugins.

// launchpad.config.mjs

import { strapiSource, sanitySource, markdownTransform } from "@bluecadet/launchpad-content";
import dashboard from "@bluecadet/launchpad-dashboard";
import simplePlugin from "./simple-plugin";

export default {
  content: {
    // content config ...,
    plugins: [
      // content sources / transforms could be plugins?
      strapiSource({
        // ...
      }),
      sanitySource({
        // ...
      }),
      markdownTransform({
        // ...
      }),
      simplePlugin,
    ],
  },
  monitor: {
    // monitor config...,
    plugins: [
      dashboard({
        // ...
      }),
    ],
  },
};

Plugin Specific Events (maybe?)

In addition to the lifecycle hooks from launchpad core, we could potentially also add a way for plugins to emit events that other plugins could listen for.

const pluginA = {
  name: "plugin-a",
  initialize: (emitter) => {
    // assuming 'hookName' is a launchpad core hook
    emitter.on("hookName", (event, context, next) => {
      // ...
      emitter.emit("pluginA:custom-event", event);
    });
  },
};

const pluginB = {
  name: "plugin-b",
  initialize: (emitter) => {
    emitter.on("pluginA:custom-event", (event, context, next) => {
      // ...
    });
  },
};

Why?

Project-specific needs

As launchpad is used in more and more projects, it seems like we're occasionally adding features, flags, and options to the config that are pretty project specific, and because of launchpad's API they need to be implemented at the library level. Making this change would open up the internals of launchpad, so that those project specific changes could instead be written as plugins in the project repo.

Isolated code

The event/hooks system also means more isolated, feature-focused code. Imagine a simple plugin that, after the content is done downloading, copies it to an additional directory – a feature that could be useful for mirroring the content directory in the project folder and build folder for unity projects (sounds super simple, but imagine we want to only perform this copy if that target directory exists, and also backup the existing data incase something goes wrong). With the current API, if we wanted to include this as a launchpad content feature, we would have to add at least one option to the content config, and add that logic somewhere in the content class. With the plugin API, this logic could live isolated in a copyContent() plugin.

In-place testing and modification

Additionally, this would allow for quick and easy testing of changes/additions to launchpad logic without having to setup a linked local clone of the launchpad repo. You could add (or copy) a plugin to your project repo, make updates and test locally before copying it to the launchpad repo.

Expose APIs for modifying json

For the content lifecycle, we can expose opportunities to alter json before it's written to disk. Currently the only way to modify json would be adding a custom post-content hook that would read in the data, modify it, and write it to disk again. Plus, a secondary script like that would either have to recreate the backup behavior, or sacrifice that security.

Env overrides, shareable configs, jsdoc hints

This approach also offers all of the benefits mentioned in #65.

To me this doesn't feel like a super big jump, considering launchpad already has defined lifecycle hooks, plus pre/post hooks. This would however likely involve a pretty big effort to refactor the existing launchpad tools and documentation to match this plugin API.

Potential Roadmap for Refactor

1. Define all events/hooks for monitor and content lifecycles, and the context for each
2. Determine signatures for each hook (what should the event object look like? what should be returned?)
3. Refactor launchpad core, content, and monitor to implement event emitter API
4. Determine what existing behaviors might be more appropriate as plugins, and pull them out into their into separate plugin files

Some Open Questions

• which behaviors and hooks are part of the core launchpad library, and which are plugins? The line can get a bit blurry.
• how do we handle the existing config? Do we just deprecate it and tell people to use plugins instead? Do we keep it around for backwards compatibility? Do we try to convert it to plugins?
• what does a hook API look like? Do all hooks have the same API? What would a hook return?

[benjaminbojko]

Thanks for the super thorough writeup and roadmap.

A few follow up thoughts and questions:

• I like the transition from pure config-based setup to script-based; somehow I always wanted to see launchpad as a drop-in dependency but in the end you have to install and run NPM regardless, so might as well embrace it as such.
• Next steps might be a proof of concept on the next project with backwards compatibility for JSON config, then migrate over.
• Your points about code isolation and testing out features/limiting project-specific code to plugins makes 💯 sense. E.g. the image filter support that LP has at the moment (resizing, cropping, ...) would probably make more sense as a plugin.
• I think the API for plugins will need some more thought. Do you have any references that use a similar structure to what you're proposing? Most of LP's core is built using polymorphism, and I wonder if that makes sense for plugins since they'd need to import LP anyways, or if that's just my old-timer thinking applied to modern JS.
• The ability to add plugins was always the ultimate goal for the library and even with the old version of launchpad, it was incredibly helpful to have that kind of support (albeit not very formalized). Here's what that looked like in the core package and a [client project]. The main point at that time was to leverage the HTTP server and provide a custom dashboard and GET/POST API for show control. The only reason content updates and monitoring weren't accessible was that those packages just weren't as streamlined then.

Let's discuss more next time we meet!

[benjaminbojko]

See https://github.com/rehypejs/rehype

[claytercek]

Some more examples of similar APIs:

• https://webpack.js.org/api/plugins/#plugin-types
• https://rollupjs.org/plugin-development/#a-simple-example
• https://vitejs.dev/guide/api-plugin.html#plugins-config (built on top of the rollup plugin API ^)
• https://svelte.dev/docs/svelte-compiler#preprocess
• https://vuejs.org/guide/reusability/plugins.html#introduction
• https://docs.astro.build/en/reference/integrations-reference/
• https://www.gatsbyjs.com/docs/how-to/plugins-and-themes/creating-a-transformer-plugin/

[claytercek]

Noticing two different patterns looking through these docs:

1. hooks always return void | Promise<void>. Often accompanied by context args that include readonly references to config/data, and some functions for updating like updateConfig() and updateData() if relevant
2. "transformer" hooks that expect an object as an argument, mutate it, then return the modified object. Args and return type differ depending on the hook.

I'm particularly drawn to the first, as it's consistent in that all hooks have the same return type (like astro), and it means you can use early returns in your logic without having to return something specific. Using update functions instead of mutating objects directly also feels safer?

Also, rollup has some nice documentation on types of hooks that could be worth thinking about if we need to distinguish between parallel and sequential hooks, plus some great diagrams that show the sequence of events, and when hooks are called.

[claytercek]

Another good use-case for plugins could be error tracking services, like rollbar or sentry. Log errors on unsuccessful fetches, broken assets, etc.

[benjaminbojko]

A few more questions and thoughts for our discussion this week:

Questions

• Start with image and text transform plugins? Heartbeat?
• Official repo of plugins? E.g. image/text transforms?
• Discontinue JSON config support?
• Are content sources just plugins now?
• Support different hook types like Rollup?
  • async, first, sequential, parallel
• Should plugins distinguish between hooks and events? Maybe hooks are just events to plugins. Then question arises around sequence, but plugins should probably go after config.

Requirements

• Access to the following (TBD if all plugins, hooks, events)
  • Config
  • Hooks/events
  • Content sources
  • Content source items (text and media) at various stages (before download, after item download, after content source finished)
  • Logging
  • Http server (when ready)
  • Launchpad instance?
  • App Instances (status, commands)
• Async operations that receive params, return values (via promises?), can throw errors/reject promises, skip/filter items.
• Support for async operations
• Support js-based config Support javascript config files in addition to json #70

[benjaminbojko]

Work Session Aug 10, 2023

• Session Mural
• Reviewed all events that plugins might need to leverage
• Looked at Content, Monitor, Core, Logging
• Discussed and defined high-level plugin API

Key Takeaways

• Events and hooks are the same
• pre/post could just be options for events and hooks, maybe default to post
• Plugins are async functions that are triggered by events
• Plugins receive a context param, followed by event-specific params
• Plugins always return Promise<void>
  • Allows them to just return; w/o addtl. requirements
  • Will require function results to passed via parameter API (e.g. skip() or abort() functions for individual items)
• Plugins only reject promises when there's a fatal error (e.g. network or corrupt data)
• All events, contexts and params should support IntelliSense auto complete (e.g. via jsdocs + strict typescript validation)

Key Open Questions

• Actual plugins API
  • Formalized object/class with mandatory/optional fields? E.g. name, callback, order, priority, ...)
• File structure

Next Steps

• Implement a simple transform (text or media) as a plugin
• Very rough refactor branch to note what refactoring would be needed eventually
• Create a PR with a summary of what changed, and what we need to do to formalize these changes with proper APIs and docs across the board
• Gradually implement other events + their APIs

[claytercek]

Plugin API for content and monitor is included in 2.0 release!