[WIP] External Plugins #1115

minchinweb · 2020-12-14T04:42:16Z

@wren This one's for you! This has been rattling around in my brain for soooooo long, but sometimes a little push at the right time is all that is needed to make the magic happen. So here we are. :)

This is a first pass at allowing external, namespace plugins.

Existing plugins are now in jrnl.plugins.importer and jrnl.plugins.exporter. "Private" plugins are to be in the jrnl.contrib.importer and jrnl.contrib.exporter namespaces (so I might have my exporter at jrnl.contrib.exporter.minchinwebexporter).

Some other code was moved around to avoid circular imports and other weird import issues. The code hasn't been extensively tested, but jrnl appears to still be working locally, including listing all the export formats.

No documentation has been added.
I also haven't run the tests (but the CI pipeline will soon tell me if I broken anything).

In the future, backends could probably also be added as/converted to plugins, but this doesn't try to do that.

c.f. #1006

github-actions · 2020-12-14T04:46:55Z

Unit Test Results

  16 files   16 suites 11s ⏱️
289 tests 289 ✔️ 0 💤 0 ❌
290 runs 290 ✔️ 0 💤 0 ❌

results for commit f35cea7

wren · 2020-12-15T08:49:57Z

Awesome, thank you! I'll take a look at it by this weekend.

wren · 2020-12-15T19:58:28Z

jrnl/plugins/util.py

 def get_tags_count(journal):
    """Returns a set of tuples (count, tag) for all tags present in the journal."""
    # Astute reader: should the following line leave you as puzzled as me the first time
-    # I came across this construction, worry not and embrace the ensuing moment of enlightment.
+    # I came across this construction, worry not and embrace the ensuing moment of enlightenment.


wren · 2020-12-15T20:03:11Z

jrnl/plugins/meta.py

+__exporter_types = {
+    **__exporter_types_builtin,
+    **__exporter_types_contrib,
+    **__exporter_types_template,
+}


So, this would override builtin plugins with contrib plugins of the same type, right? Does that behavior get communicated to the user in any way? If somebody runs jrnl --format markdown, do they have any way to know if that's a builtin, or some random plugin they forgot they installed?

This would also be relevant for users reporting bugs. We need a way for us to know which version of which plugin they are using, I think.

My first thought was to add the list of plugins (with versions) to the information displayed by jrnl --version. But nothing has been implemented quite yet.

(Trying to do this from my phone wasn't working... so now I'm at the computer)

this would override builtin plugins with contrib plugins of the same type, right?

Yes, that is what is supposed to happen. I haven't tested it yet....

Does that behavior get communicated to the user in any way?

No, not currently. My thought was to extend jrnl --version to list the installed plugins, but to note the plugin used at the start of the export would be good too. In any case, the version of the plugin should be printed as well.

wren · 2020-12-15T20:04:35Z

@minchinweb From a user standpoint, what does this look like? If I want to install a new plugin, what are the steps to do that? It seems like putting a file in the new contrib directory would work. Is that right? Are there other steps?

wren · 2020-12-15T20:07:32Z

jrnl/plugins/meta.py

+__exporters_builtin = list(
+    pkgutil.iter_modules(
+        jrnl.plugins.exporter.__path__, jrnl.plugins.exporter.__name__ + "."
+    )
+)
+__exporters_contrib = list(
+    pkgutil.iter_modules(
+        jrnl.contrib.exporter.__path__, jrnl.contrib.exporter.__name__ + "."
+    )
+)
+
+__importers_builtin = list(
+    pkgutil.iter_modules(
+        jrnl.plugins.importer.__path__, jrnl.plugins.importer.__name__ + "."
+    )
+)
+__importers_contrib = list(
+    pkgutil.iter_modules(
+        jrnl.contrib.importer.__path__, jrnl.contrib.importer.__name__ + "."
+    )
+)


Instead of duplicating each of these (one for builtin, and one for contrib), can we have a function that will do all of this work given a path? Then we could run that function once for each path (builtin, and contrib).

A standalone function would also be easier to add tests for.

Can be done!

minchinweb · 2020-12-15T21:14:01Z

From a user standpoint, what does this look like? If I want to install a new plugin, what are the steps to do that? It seems like putting a file in the new contrib directory would work. Is that right? Are there other steps?

I've done this using namespace plugins. So to develop a new plugin, you would create a new Python project, with a layout something like:

jrnl
  `- contrib
       `- exporter
            `- myfancyexporter.py

(Note, no __init__.py files.)

myfancyexporter.py would contain an Exporter class, with the properties names (a list, and what you use as a "name" to call the exporter from jrnl), extension (a string, which is what file extension the exported files will have), and class methods export_entry (to export a single entry) and export_journal (to export the entire journal). (There should also be something to return the version of the plugin; interface TBD.)

You would then package it like any other Python library. The user would then install it like any other Python library (via pip or poetry, etc). Assuming everything works, a new exporter would be available to jrnl simply by virtual of the plugin package being installed.

In theory, putting a file into jrnl's contrib folder would work, but with namespace plugins (like I'm aiming for here), the plugins available to Python at jnrl.contrib do not have to be in the same ("physical") folder. So ideally, you should never have to directly touch your jrnl installation to install (or use) your "personal" plugins, nor will they be wiped on upgrades. Your personal plugin can also be versioned like any other project dependency.

wren · 2020-12-16T02:54:07Z

In theory, putting a file into jrnl's contrib folder would work, but with namespace plugins (like I'm aiming for here), the plugins available to Python at jnrl.contrib do not have to be in the same ("physical") folder. So ideally, you should never have to directly touch your jrnl installation to install (or use) your "personal" plugins, nor will they be wiped on upgrades. Your personal plugin can also be versioned like any other project dependency.

In that case, why are we including an empty contrib directory in jrnl? If we don't want users touching it, then I think we should not include it to discourage its use, right?

minchinweb · 2020-12-16T16:18:36Z

In that case, why are we including an empty contrib directory in jrnl?

To make sure the namespace exists, even if it's empty. If it doesn't exist, we can't import it, and we can't iterate over the contained packages.

wren · 2020-12-26T19:49:10Z

I think this is looking okay. What this still needs:

tests
handling for name conflicts among plugins (to make it clear to the user, and us, when a plugin is installed)
some sort of output for versions when a plugin is installed (maybe --diagnostic?)

minchinweb · 2020-12-30T20:20:36Z

Also:

documentation on making your own plugin

minchinweb · 2021-03-17T02:23:19Z

@wren: I updated this with an output that shows the active plugins and their versions, and added documentation. Testing is more complicated, as I need a way to create a new "testing environment" to install the sample plugins as python packages. Maybe that's simple, but I haven't figured it out yet and I wanted to share what I have done.

I created a new branch because I gave up on trying to figure out how to cleanly push here, so the code is now at #1216.

Thank you for all you do to keep the program moving forward!

FIrst pass at allow external plugins

f35cea7

wren reviewed Dec 15, 2020

View reviewed changes

wren force-pushed the develop branch 10 times, most recently from b6f03d3 to 3fb9fe5 Compare December 20, 2020 06:39

wren force-pushed the develop branch from 15d364b to 3bce055 Compare December 26, 2020 20:08

wren marked this pull request as draft March 13, 2021 19:10

wren force-pushed the develop branch 6 times, most recently from 355cf2f to c73d0ff Compare March 13, 2021 22:24

wren force-pushed the develop branch 6 times, most recently from 8bd89b7 to 7bacf4a Compare March 13, 2021 23:28

minchinweb mentioned this pull request Mar 17, 2021

External Plugins v2 #1216

Merged

4 tasks

wren closed this Mar 20, 2021

minchinweb mentioned this pull request May 1, 2021

External formatter plugins #1239

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] External Plugins #1115

[WIP] External Plugins #1115

minchinweb commented Dec 14, 2020

github-actions bot commented Dec 14, 2020

wren commented Dec 15, 2020

wren Dec 15, 2020

wren Dec 15, 2020

minchinweb Dec 15, 2020

minchinweb Dec 15, 2020

wren commented Dec 15, 2020

wren Dec 15, 2020

wren Dec 15, 2020

minchinweb Dec 15, 2020

minchinweb commented Dec 15, 2020

wren commented Dec 16, 2020

minchinweb commented Dec 16, 2020

wren commented Dec 26, 2020

minchinweb commented Dec 30, 2020

minchinweb commented Mar 17, 2021 •

edited

Loading

[WIP] External Plugins #1115

[WIP] External Plugins #1115

Conversation

minchinweb commented Dec 14, 2020

github-actions bot commented Dec 14, 2020

Unit Test Results

wren commented Dec 15, 2020

wren Dec 15, 2020

Choose a reason for hiding this comment

wren Dec 15, 2020

Choose a reason for hiding this comment

minchinweb Dec 15, 2020

Choose a reason for hiding this comment

minchinweb Dec 15, 2020

Choose a reason for hiding this comment

wren commented Dec 15, 2020

wren Dec 15, 2020

Choose a reason for hiding this comment

wren Dec 15, 2020

Choose a reason for hiding this comment

minchinweb Dec 15, 2020

Choose a reason for hiding this comment

minchinweb commented Dec 15, 2020

wren commented Dec 16, 2020

minchinweb commented Dec 16, 2020

wren commented Dec 26, 2020

minchinweb commented Dec 30, 2020

minchinweb commented Mar 17, 2021 • edited Loading

minchinweb commented Mar 17, 2021 •

edited

Loading