Add Steel as an optional plugin system

[mattwparas]

Notes:

• I still need to rebase up with the latest master changes, however doing so causes some headache with the lock file, so I'll do it after some initial feedback. Also, this depends on the event system in Add an event system #8021.
• The large diff size is a combination of lock file changes + the dependency on the event system PR. The diff has ended up quite large with all of the other stuff
• I'm currently pointing to the master branch of steel as a dependency. This will point to a stable release on crates once I cut a release.

---

Opening this just to track progress on the effort and gather some feedback. There is still work to be done but I would like to gather some opinions on the direction before I continue more.

You can see my currently functioning helix config here and there are instructions listed in the STEEL.md file. The main repo for steel lives here, however much documentation is in works and will be added soon.

The bulk of the implementation lies in the engine.rs and scheme.rs files.

Design

Given prior conversation about developing a custom language implementation, I attempted to make the integration with Steel as agnostic of the engine as possible to keep that door open.

The interface I ended up with (which is subject to change and would love feedback on) is the following:

pub trait PluginSystem {
    /// If any initialization needs to happen prior to the initialization script being run,
    /// this is done here. This is run before the context is available.
    fn initialize(&self) {}

    fn engine_name(&self) -> PluginSystemKind;

    /// Post initialization, once the context is available. This means you should be able to
    /// run anything here that could modify the context before the main editor is available.
    fn run_initialization_script(&self, _cx: &mut Context) {}

    /// Allow the engine to directly handle a keymap event. This is some of the tightest integration
    /// with the engine, directly intercepting any keymap events. By default, this just delegates to the
    /// editors default keybindings.
    #[inline(always)]
    fn handle_keymap_event(
        &self,
        _editor: &mut ui::EditorView,
        _mode: Mode,
        _cxt: &mut Context,
        _event: KeyEvent,
    ) -> Option<KeymapResult> {
        None
    }

    /// This attempts to call a function in the engine with the name `name` using the args `args`. The context
    /// is available here. Returns a bool indicating whether the function exists or not.
    #[inline(always)]
    fn call_function_if_global_exists(
        &self,
        _cx: &mut Context,
        _name: &str,
        _args: &[Cow<str>],
    ) -> bool {
        false
    }

    /// This is explicitly for calling a function via the typed command interface, e.g. `:vsplit`. The context here
    /// that is available is more limited than the context available in `call_function_if_global_exists`. This also
    /// gives the ability to handle in progress commands with `PromptEvent`.
    #[inline(always)]
    fn call_typed_command_if_global_exists<'a>(
        &self,
        _cx: &mut compositor::Context,
        _input: &'a str,
        _parts: &'a [&'a str],
        _event: PromptEvent,
    ) -> bool {
        false
    }

    /// Given an identifier, extract the documentation from the engine.
    #[inline(always)]
    fn get_doc_for_identifier(&self, _ident: &str) -> Option<String> {
        None
    }

    /// Fuzzy match the input against the fuzzy matcher, used for handling completions on typed commands
    #[inline(always)]
    fn available_commands<'a>(&self) -> Vec<Cow<'a, str>> {
        Vec::new()
    }

    /// Retrieve a theme for a given name
    #[inline(always)]
    fn load_theme(&self, _name: &str) -> Option<Theme> {
        None
    }

    /// Retrieve the list of themes that exist within the runtime
    #[inline(always)]
    fn themes(&self) -> Option<Vec<String>> {
        None
    }

    /// Fetch the language configuration as monitored by the plugin system.
    ///
    /// For now - this maintains backwards compatibility with the existing toml configuration,
    /// and as such the toml error is exposed here.
    #[inline(always)]
    fn load_language_configuration(&self) -> Option<Result<Configuration, toml::de::Error>> {
        None
    }
}

If you can implement this, the engine should be able to be embedded within Helix. On top of that, I believe what I have allows the coexistence of multiple scripting engines, with a built in priority for resolving commands / configurations / etc.

As a result, Steel here is entirely optional and also remains completely backwards compatible with the existing toml configuration. Steel is just another layer on the existing configuration chain, and as such will be applied last. This applies to both the config.toml and the languages.toml. Keybindings can be defined via Steel as well, and these can be buffer specific, language specific, or global. Themes can also be defined from Steel code and enabled, although this is not as rigorously tested and is a relatively recent addition. Otherwise, I have been using this as my daily driver to develop for the last few months.

I opted for a two tiered approach, centered around a handful of design ideas that I'd like feedback on:

The first, there is a init.scm and a helix.scm file - the helix.scm module is where you define any commands that you would like to use at all. Any function exposed via that module is eligible to be used as a typed command or via a keybinding. For example:

;; helix.scm

(provide shell)

;;@doc
;; Specialized shell - also be able to override the existing definition, if possible.
(define (shell cx . args)
  ;; Replace the % with the current file
  (define expanded (map (lambda (x) (if (equal? x "%") (current-path cx) x)) args))
  (helix.run-shell-command cx expanded helix.PromptEvent::Validate))

This would then make the command :shell available, and it will just replace the % with the current file. The documentation listed in the @doc doc comment will also pop up explaining what the command does:

Once the helix.scm module is require'd - then the init.scm file is run. One thing to note is that the helix.scm module does not have direct access to a running helix context. It must act entirely stateless of anything related to the helix context object. Running init.scm gives access to a helix object, currently defined as *helix.cx*. This is something I'm not sure I particularly love, as it makes async function calls a bit odd - I think it might make more sense to make the helix context just a global inside of a module. This would also save the hassle that every function exposed has to accept a cx parameter - this ends up with a great deal of boilerplate that I don't love. Consider the following:

;;@doc
;; Create a file under wherever we are
(define (create-file cx)
  (when (currently-in-labelled-buffer? cx FILE-TREE)
    (define currently-selected (list-ref *file-tree* (helix.static.get-current-line-number cx)))
    (define prompt
      (if (is-dir? currently-selected)
          (string-append "New file: " currently-selected "/")
          (string-append "New file: "
                         (trim-end-matches currently-selected (file-name currently-selected)))))

    (helix-prompt!
     cx
     prompt
     (lambda (cx result)
       (define file-name (string-append (trim-start-matches prompt "New file: ") result))
       (temporarily-switch-focus cx
                                 (lambda (cx)
                                   (helix.vsplit-new cx '() helix.PromptEvent::Validate)
                                   (helix.open cx (list file-name) helix.PromptEvent::Validate)
                                   (helix.write cx (list file-name) helix.PromptEvent::Validate)
                                   (helix.quit cx '() helix.PromptEvent::Validate)))

       (enqueue-thread-local-callback cx refresh-file-tree)))))

Every function call to helix built ins requires passing in the cx object - I think just having them be able to reference the global behind the scenes would make this a bit ergonomic. The integration with the helix runtime would make sure whether that variable actually points to a legal context, since we pass this in via reference, so it is only alive for the duration of the call to the engine.

Async functions

Steel has support for async functions, and has successfully been integrated with the tokio runtime used within helix, however it requires constructing manually the callback function yourself, rather than elegantly being able to use something like await. More to come on this, since the eventual design will depend on the decision to use a local context variable vs a global one.

Built in functions

The basic built in functions are first all of the function that are typed and static - i.e. everything here:

• Typed
• Static

However, these functions don't return values so aren't particularly useful for anything but their side effects to the editor state. As a result, I've taken the liberty of defining functions as I've needed/wanted them. Some care will need to be decided what those functions actually exposed are.

Examples

Here are some examples of plugins that I have developed using Steel:

File tree

Source can be found here

filetree.webm

Recent file picker

Source can be found here

recent-files.webm

This persists your recent files between sessions.

Scheme indent

Since steel is a scheme, there is a relatively okay scheme indent mode that only applied on .scm files, which can be found here. The implementation requires a little love, but worked enough for me to use helix to write scheme code 😄

Terminal emulator

I did manage to whip up a terminal emulator, however paused the development of it while focusing on other things. When I get it back into working shape, I will post a video of it here. I am not sure what the status is with respect to a built in terminal emulator, but the one I got working did not attempt to do complete emulation, but rather just maintained a shell to interact with non-interactively (e.g. don't try to launch helix in it, you'll have a bad time 😄 )

Steel as a choice for a language

I understand that there is skepticism around something like Steel, however I have been working diligently on improving it. My current projects include shoring up the documentation, and working on an LSP for it to make development easier - but I will do that in parallel with maintaining this PR. If Steel is not chosen and a different language is picked, in theory the API I've exposed should do the trick at least with matching the implementation behavior that I've outlined here.

Pure rust plugins

As part of this, I spent some time trying to expose a C ABI from helix to do rust to rust plugins directly in helix without a scripting engine, with little success. Steel supports loading dylibs over a stable abi (will link to documentation once I've written it). I used this to develop the proof of concept terminal emulator. So, you might not be a huge fan of scheme code, but in theory you can write mostly Rust and use Steel as glue if you'd like - you would just be limited to the abi compatible types.

System compatibility

I develop off of Linux and Mac - but have not tested on windows. I have access to a windows system, and will get around to testing on that when the time comes.

[RGBCube]

This thread is really long, so I haven't checked the history. Who's working on plugins?

Maybe we could make an awesome-helix repository that links Helix distributions & plugins. Would also be nice to have it on the website

[nik-rev]

This thread is really long, so I haven't checked the history. Who's working on plugins?

Maybe we could make an awesome-helix repository that links Helix distributions & plugins. Would also be nice to have it on the website

I think people are waiting until release to start working on plugins, since there may be breaking changes

[rockygolucky]

Maybe we could make an awesome-helix repository that links Helix distributions & plugins. Would also be nice to have it on the website

I would absolutely adore that.

[rockygolucky]

Not to put the horse before the cart too much, but I do wonder about the distribution mechanism for packages.

Other editors have entire package managers just for plugins. I can’t say I miss the days of deciding which Neovim plugin manager to use.

It would be cool if there was a sensible default.

[mattwparas]

Not to put the horse before the cart too much, but I do wonder about the distribution mechanism for packages.

Other editors have entire package managers just for plugins. I can’t say I miss the days of deciding which Neovim plugin manager to use.

It would be cool if there was a sensible default.

I have a functioning package manager working on a branch for steel, which will work for helix as well. Trying to dog food it a bit and iron out some details before releasing. At the moment the distribution mechanism is just git urls, which will probably work to start.

[RGBCube]

I think it shouldn't be bundled by default, as people might want to use other package managers. I'm going to use home-manager for example and a builtin :plugin-install would immediately break as the directory will be immutable.

But a recommended plugin would be great, for people who want the default experience

[rockygolucky]

I'm going to use home-manager for example and a builtin :plugin-install would immediately break as the directory will be immutable.

That’s a good point. I would use home-manager too.

[zetashift]

I think it shouldn't be bundled by default, as people might want to use other package managers. I'm going to use home-manager for example and a builtin :plugin-install would immediately break as the directory will be immutable.

I think it's fine if it's bundled by default, but not enabled by default(maybe you meant it that way, and I misread).
I'm also a Nix/home-manager user, and I don't see the situation happen where a you'd run :plugin-install manually? Wouldn't that be more like :sh home-manager switch.

Besides there are plenty of things in home-manager that do ship with a package manager, like Node or even Emacs comes with package.el(just not 'enabled' by default).

[eikenb]

As a happy (lurking) user I'd like to throw in vote on the package manager discussion... IMO it should be installed and enabled by default but easy to disable. I don't think standalone environment managers like home-manager are that pervasive and most users would benefit from having package management be a solved issue that they didn't have to consider. Thanks.

[mattwparas]

Including modules in steel is really quite simple; just tell steel where to look for them. Given that flexibility (for better or for worse), it means that no one package management solution will be the only way to do things. I'm hoping to provide a very sensible and functional default, but if you'd like to use another solution you should be more than welcome to.

I myself don't use nix, so rest assured we'll have something that works for non nix users

[Dietr1ch]

I guess that straight.el's comparison to use-package might come useful when discussing what's needed from the plugin management system, but starting from reading a single directory configured at startup seems flexible enough for plugin systems to proliferate and explore a common plugin definition, config, and versioning system.

Being able to configure that single directory is important, on Emacs trying out different configs or patches/fixes to existing ones and such was painful enough that chemacs and chemacs2 came out to help define what configuration and set of plugins should be used.

[noahfraiture]

Having a default package manager is one of the greatest feature in new language in my opinion, it standardize and regulate stuffs. It's good to have different possibility but it should be an option and you shouldn't have to make a choice in the first place to try stuff. Having a lot of possibility when it comes to plugin management in neovim is, in my opinion, a weakness concerning beginner as every plugin will have different instruction of installation and it's a real pain when you start your nvim journey

[noor-tg]

Will steel allow something like this ?

a-h/templ#387 (comment)

Quick and di

For any LazyVim users who could not get catgoose's plugin to work, here's a solution that works for me. Put the following in lua/plugins/lsp.lua:

return {
  "neovim/nvim-lspconfig",
  opts = function(args)
    local go_to_definition = function()
      if vim.bo.filetype == "go" then
        vim.lsp.buf.definition({
          on_list = function(options)
            if options == nil or options.items == nil or #options.items == 0 then
              return
            end

            local targetFile = options.items[1].filename
            local prefix = string.match(targetFile, "(.-)_templ%.go$")

            if prefix then
              local function_name = vim.fn.expand("<cword>")
              options.items[1].filename = prefix .. ".templ"

              vim.fn.setqflist({}, " ", options)
              vim.api.nvim_command("cfirst")

              vim.api.nvim_command("silent! /templ " .. function_name)
            else
              vim.lsp.buf.definition()
            end
          end,
        })
      else
        vim.lsp.buf.definition()
      end
    end
    local function go_goto_def()
      if vim.bo.filetype == "go" then
        return go_to_definition()
      else
        return vim.lsp.buf.definition()
      end
    end
    local keys = require("lazyvim.plugins.lsp.keymaps").get()
    keys[#keys + 1] = { "gd", go_goto_def }
  end,
}

[nik-rev]

I was thinking what kind of plugins would be really nice to have when Helix has a plugin system. So I browsed Dotfyle Neovim Plugins for quite a while. Most of the features I'd want Helix either already has, PRs already exist to add those features, or doesn't make sense to be in core. But those that don't make sense in core would still be nice features that could exist as plugins.

So I feel like plugins which target specific tooling or languages and don't make an overall impact on the editor are the perfect example of what Helix plugins could be used for. For example, it would be nice to have an SQL query builder within Helix. But it doesn't make sense for that to be anything but a plugin, for example

Once this PR drops, I may wish to develop a plugin so I'm putting this list I've compiled of nice-to-have plugins down here. Feel free to add to it or use it for inspiration.

---

• markview.hx: Fancy markdown preview like in markview.nvim:

• leetcode.hx: A plugin like leetcode.nvim, would be neat to complete challenges inside of Helix. Currently I'm using https://github.com/clearloop/leetcode-cli for that

• extra-textobjects.hx: Plugin which adds text objects specific to a language. "delete in HTML tag"

• Plugin which allows auto-closing and auto-renaming of HTML tags

• pkgman.hx: Plugin for package managers which shows if an upgrade is available, or adds code actions to upgrade to the latest release

• termshot.hx: Renders terminal cells as HTML. (fun fact: Neovim has this built-in as :tohtml oddly enough)

• pretty-ts-error.hx: TypeScript errors can be hard to read so it would be nice to have a plugin for it that makes it human readable.

• screenkey.hx: Display the keys that I am entering in a tiny buffer

• A plugin which automatically changes regular strings to template strings when we type ${ in JavaScript. In python, regular strings become fstrings. etc.

[VetleHvoslef]

The two plugins I would really like to see is something like hardtime.nvim and precognition.nvim. These would really help beginners learn commands

[jerabaul29]

More ideas of plugins that would be very nice:

• some good REPL support to be able to nicely run jupyter notebooks and similar
• some way to use "helix as the terminal", where everything can be copy / pasted, with history search, with both commands and commands outputs being "just text" that can be manipulated

[MatrixManAtYrService]

As far as I'm aware, helix has the most sophisticated text-selection management system. I would like to play around with leaning on it to support arbitrary highlighting. We're all familiar with rules-driven highlighting, where language keywords might appear in one color and variable names might appear in another. By "arbitrary" I mean that you'd press some key while text is selected, and the selected text would turn green and stay green even after you've gone on to select something else.

Presumably it would be a plugin which is remembering which text is painted in which colors, and then you could extract it as metadata later. Characters 0-47 were painted red, 500-505 were blue, etc. This would be a precursor towards tooling that thinks in terms of text selections rather than indicating a row and line numbers (Imagine receiving advice from an LLM where substrings like this function and this expression here are colored, and those colors correspond with the background/highlight color for some nearby code).

Does anyone have any tips about where I'd go looking to see if such functionality is accessible from within a plugin?

[merisbahti]

Two plugins I've made the last few days are a quick and dirty OpenAI hinting plugin, and a steel "evaluate all top level forms" plugin. Note that both require the add-inlay-hint and remove-inlay-hint steel functions from my pr here: mattwparas#14.

Source and demo vids here:

AI Hint suggestions

ai-hints.mov

source

Evaluating top level expressions and displaying them as ghost hints with steel

Screen.Recording.2024-12-26.at.20.54.03.mov

source

[ofzo]

What is the progress, Whether you can publish a preview version

[noor-tg]

@ofzo you can test it now by your self. just check how to integrate steel with helix from this pr post and comments.

And checkout steel repo

[satvikpendem]

Just curious why Scheme instead of writing packages in Rust itself? Part of the reason I use Helix is because everything is in Rust and can be very performant. I worry if people start writing many popular plugins in Steel that we might get a situation like neovim which is slowed down by package loading and general performance speed, as it seems like Steel can be as slow as Python, even Lua which is neovim's plug-in language is faster than that.

[UltraBlackLinux]

I don't see what needs to be particularly performant about helix, apart from opening large files in a timely manner, which happens regardless of the plugin system.

Compiled plugins are not cross compatible between all the architectures that helix can run on, and it's also much less approachable. rust is from what I've seen not an easy language to grasp, even if you have programming experience. Its syntax and concepts can be hard to understand.

A plugin system making use of scheme makes all of this much easier. The plugins are cross-compatible since the plugin-system is part of the editor itself, and the language is very simplistic (which doesn't have to be a bad thing, and emacs is proof for that. scheme is merely a lisp dialect)

[lcpichette]

@satvikpendem I'm sharing this explanation here for you, and for anyone else reviewing this PR, who may wonder why your comment was downvoted so quickly.

It’s generally expected—out of courtesy for everyone involved—that before asking a question, you take a moment to see if the answer is readily available. More than 50 people are subscribed to the updates on this PR, and many of them may be feeling fatigued by having to re-answer questions that have already been addressed multiple times.

In this case, the answer to your question can be found by searching for “plugin” in the Helix discussions, specifically looking at the top-rated thread (#3806). You can also Google it... there are Reddit threads and other sources that cover this topic thoroughly.

[satvikpendem]

@lcpichette I see, thank you, I didn't think to look in the discussions tab (truthfully I personally never use the discussions tab myself for my own projects and those I contribute to so that's probably why), I just searched the issues tab. I appreciate the response.

[nik-rev]

If at some point there's going to be an LSP for Steel as well as a formatter, would it make sense to ship those with the editor so that users don't even need to configure it?

[mattwparas]

If at some point there's going to be an LSP for Steel as well as a formatter, would it make sense to ship those with the editor so that users don't even need to configure it?

There is an LSP for steel already. For formatting, I don't have one specifically for steel, but have been using raco fmt, a formatter for racket.