config: add support for formatted keys

[alexfikl]

This adds support for entries like these in the configuration file (see discussion in #662):

[settings]
add-file-name = {doc[year]}-{doc[author]}-{doc[title]}
ref-format.jinja2 = {{ doc.author_list|slice(3)|join("", attribute="family") }}{{ doc.year }}

where the first setting uses the default formatter (whatever formatter is set to) and the second setting overwrites the default and forces the jinja2 formatter. In general, key[.formatter] is how this looks like.

For this, it adds some functionality:

• papis.config.getformattedstring: this looks thorough the config file for any version of key.[formatter] and returns the last one it finds.
• papis.strings.FormattedString: a class that's just a (formatter, string) tuple and gets passed to papis.format.format to allow picking the formatter for each individual string. This is mostly inspired by prompt_toolkit.FormattedText.
• papis.cli.FormattedStringParamType: a click.ParamType that automatically converts command-line parameters to FormattedString. It also shows a nice text --file-name FORMATTED-TEXT in the command line to mark entries that can be format strings.
• Update the rest of the code to use the above where appropriate!

TODO:

• Need to check what other keys should use formatted strings.
• Not sure the folder name construction in papis.commands.add::run works correctly with formatted strings. Need to add some tests for that.
• Probably needs some more testing to make sure nothing is broken. (Currently using this daily, so hopefully will catch some things)

Fixes #662.

[hermlon]

I just tried your fork because I need this feature, but I somehow can't start the ref-format with a lowercase letter. I'm using this Jinja format string:

ref-format.jinja2 = {{ doc.author_list[0]['family']|lower }}{{ doc.year }}

But I get the ref: Sanderson12

I guess there is some code upper-casing the ref-format again, since the Jinja2 documentation clearly states that the lower filter lowers all the characters.

[alexfikl]

I just tried your fork because I need this feature, but I somehow can't start the ref-format with a lowercase letter. I'm using this Jinja format string:

ref-format.jinja2 = {{ doc.author_list[0]['family']|lower }}{{ doc.year }}

@hermlon Can confirm that this doesn't work as expected. I'll look into it!
Thanks a lot for testing ❤️ this touches a lot of parts, so needs quite a bit more testing besides the unit tests..

[alexfikl]

@hermlon Turns out my case was just a config error. I added a test for the case you mentioned (I think), i.e. setting a ref-format.jinja2, and it seems to work as expected.

Can you give more details about your setup? Do other things work, e.g. does setting ref-format.jinja2 = {{ doc.year }} give the expected change in the ref? Can you also set ref-format in the config to see if it's picking that one by mistake?

EDIT: Gah, nevermind, that was another issue with my setup. Yes, your format didn't work because we do apparently capitalize things by default here:

papis/papis/bibtex.py

Lines 411 to 416 in c1a9d91

	return string.capwords(str(slugify.slugify(
	ref,
	lowercase=False,
	word_boundary=False,
	separator=" ",
	regex_pattern=allowed_characters))).replace(" ", "")

That doesn't sound like a good idea, so we can make a PR to revert it. Thanks for pointing that out and sorry for the confusion!

[OopsYao]

I just tried your fork because I need this feature, but I somehow can't start the ref-format with a lowercase letter. I'm using this Jinja format string:

ref-format.jinja2 = {{ doc.author_list[0]['family']|lower }}{{ doc.year }}

But I get the ref: Sanderson12

I guess there is some code upper-casing the ref-format again, since the Jinja2 documentation clearly states that the lower filter lowers all the characters.

I tried the commit 126825c; it works well, no abnormalities found yet. 👏 One thing is that something like papis update --set ref.jinja2 [ref-format] is not supported currently. Don't know if this needs extra heavy work.

[alexfikl]

I tried the commit 126825c; it works well, no abnormalities found yet. 👏

Awesome! Thank for you testing!

One thing is that something like papis update --set ref.jinja2 [ref-format] is not supported currently. Don't know if this needs extra heavy work.

Hm, it shouldn't be too hard to implement. I'll look into it! There are a few more places that take values that should be formatted from the command line, so we need some nice solution for them.

EDIT: I've looked a bit at this and there's two issues here:

• For the --set key value option that a few of the commands take, we can just use key[.formatter] and select the formatter that way. This shouldn't be hard to implement.
• Other arguments, e.g. papis add --file-name [FORMAT] ..., don't really have a way to specify the formatter. I'm not sure what a good way to do that would be. We can always recommend using

papis --set add-file-name.jinja2 [FORMAT] add ...

instead, but not sure that's very nice..

[alexfikl]

@OopsYao The latest version should also support papis update --set ref.jinja2 [format]. Let me know if you try it and something doesn't work (I just very briefly tested it)!

[pmiam]

I have continued to use this branch for some time and it has served me well. Everything I broke in my earlier escapade was due to my own experiments and config.py. I've opened a new PR to unlock some more power in-line with this feature set, though it does nothing to mitigate the possibility of other adventurous users writing their own volatile configs -- I reckon that comes with the territory.

Most of the CI builds are failing for some reason. I'll look into that hopefully within the next month.

[pmiam]

Seems I found a bug. I wrote an alias to help me generate my pretty new refs after adding a library entry from e.g. a bibtex file

alias refgen='papis update -s ref.jinja2 "$(papis config -s lit --json | jq ".\"ref-format.jinja2\"")"'

For reasons I don't fully understand, this delivers my "lit" library's format string to papis.update.run_set as a FormattedString object with formatter attr None. Therefore the formatter value from the command line was silently dropped in papis.strings.process_formatted_string_pair. I patched it, but my solution might be a bit brutish:

From 55b883e2f1e805c9d8e972bfb01b559a58181291 Mon Sep 17 00:00:00 2001
From: Panayotis Manganaris <[email protected]>
Date: Thu, 31 Oct 2024 03:01:03 -0400
Subject: [PATCH] strings [BUGFIX] guarantee FormattedString value has
 formatter attr

---
 papis/strings.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/papis/strings.py b/papis/strings.py
index 87e70ed4..c516663f 100644
--- a/papis/strings.py
+++ b/papis/strings.py
@@ -65,15 +65,14 @@ def process_formatted_string_pair(
     :param value: a unformatted value.
 
     :returns: a ``(key, value)`` pair, where the formatter was removed from the
-        key and the value is guaranted to be a :class:`FormattedString`.
+        key and the value is guaranteed to be a :class:`FormattedString`.
     """
     if "." in key:
         key, formatter = key.rsplit(".", maxsplit=1)
     else:
         formatter = None
 
-    if not isinstance(value, FormattedString):
-        value = FormattedString(formatter, value)
+    value = FormattedString(formatter, str(value))
 
     return key, value
 
-- 
2.47.0

[alexfikl]

Seems I found a bug. I wrote an alias to help me generate my pretty new refs after adding a library entry from e.g. a bibtex file

Thanks for reporting it! I'll take a look probably over the weekend.

Also noticed another thing that's missing. papis config isn't updated to work well with this, as this doesn't do what it should

papis config -s settings ref-format.jinja2

It currently considers ref-format as a section and fails miserably :(

[pmiam]

Also noticed another thing that's missing. papis config isn't updated to work well with this

Indeed. The current form of papis.config.general_get also causes papis rename to throw because

papis/papis/config.py

Line 404 in f390fd7

if key_name in config[section_name]:

always returns false if the user specifies .formatter in a config key.

also, papis rename --folder-name doesn't understand jinja format strings.

[alexfikl]

Seems I found a bug. I wrote an alias to help me generate my pretty new refs after adding a library entry from e.g. a bibtex file

alias refgen='papis update -s ref.jinja2 "$(papis config -s lit --json | jq ".\"ref-format.jinja2\"")"'

This should be fixed in the latest version with https://github.com/papis/papis/compare/e3f7629df475631a93b3e009f314e1cff8e1717e..bfbc54d909051f2c835bee1ab57bea3f3c081f9f.

The issue was indeed with that process_formatted_string_pair and how it interacted with the command line arguments transformed through FormattedStringParamType. It now insists on the formatter from the key as expected.

[alexfikl]

papis config -s settings ref-format.jinja2

This should also work now with the changes from https://github.com/papis/papis/compare/bfbc54d909051f2c835bee1ab57bea3f3c081f9f..dafed8ffa034a813fac0fd5387627d7faf516bdf.

[alexfikl]

also, papis rename --folder-name doesn't understand jinja format strings.

@pmiam This is a problem for other commands as well and I'm not sure how to fix it in an intuitive manner. For example if you just go

papis rename --folder-name "SOME_FORMATTING_STRING" <QUERY>

there's no way to know what formatter that text is supposed to have. It works for papis update because it uses the key name to specify the formatter like --set key.jinja2 value. For something like rename, I would suggest

papis --set formatter jinja2 rename --folder-name "JINJA2_FORMAT" <QUERY>

Also, if you're using another formatter than then one you set in the config in that many places, it might be worth switching your defaults?

Indeed. The current form of papis.config.general_get also causes papis rename to throw because

Can you post a traceback for that? I'm not sure in what scenario that would happen.

[pmiam]

I would suggest

papis --set formatter jinja2 rename --folder-name "JINJA2_FORMAT" <QUERY>

Ah, that's actually quite an elegant solution. I hadn't actually realized the set flag could be used like that. We could call attention to it somewhere in the docs.

The only alternative I see so far is a campaign to overhaul the CLI. I'm not sure how stable the basic interface is. I have seen other PRs working to merge/extend other commands, but I doubt deprecating basic flags like --folder-name in favor of universally relying on --set is something the main team is willing to do. (Or, even that something so drastic is needed).

might be worth switching your defaults?

Indeed, I might switch my defaults at this point, I've become enamored with my jinja-given powers. It's important to handle this middle ground, though, as it's an explicit feature of this PR.

[pmiam]

Indeed. The current form of papis.config.general_get also causes papis rename to throw [...]

Can you post a traceback for that? I'm not sure in what scenario that would happen.

To be more precise, the if-condition I referenced is always false in the case when I set my add-folder-name.jinja2 key under a custom library subsection, and leave the default under the global settings section alone.

e.g. with config:

[settings]
default-library = lit

[lit]
dir = ~/doc/lit
add-folder-name.jinja2 = {{ doc.author_list|map(attribute="family")|join }}

I run papis rename on your latest commit dafed8f

Traceback (most recent call last):
  File "/home/panos/.local/bin/papis", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 1686, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 2396, in handle_parse_result
    value, source = self.consume_value(ctx, opts)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 2934, in consume_value
    value, source = super().consume_value(ctx, opts)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 2290, in consume_value
    value = self.get_default(ctx)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 2868, in get_default
    return super().get_default(ctx, call=call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/.direnv/python-3.12/lib/python3.12/site-packages/click/core.py", line 2268, in get_default
    value = value()
            ^^^^^^^
  File "/home/panos/src/papis/papis/commands/rename.py", line 112, in <lambda>
    default=lambda: papis.config.getstring("add-folder-name"))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/panos/src/papis/papis/config.py", line 479, in getstring
    raise ValueError("Key '{}' should be a string: '{}'".format(key, result))
ValueError: Key 'add-folder-name' should be a string: ''

[alexfikl]

File "/home/panos/src/papis/papis/commands/rename.py", line 112, in
default=lambda: papis.config.getstring("add-folder-name"))

Oops, this just looks like I forgot to update the rename command! This should be getformattedstring("add-folder-name") 😁

[alexfikl]

I'm not sure how stable the basic interface is. I have seen other PRs working to merge/extend other commands, but I doubt deprecating basic flags like --folder-name in favor of universally relying on --set is something the main team is willing to do. (Or, even that something so drastic is needed).

I don't think the interface is very stable.. or at least we don't do any particular testing to ensure it stays too compatible :( That being said, stuff like --folder-name and --file-name was mostly added as helpful shorthands for

papis --set add-folder-name "MY_SPECIAL_FORMAT" add <SOMETHING>`

I'm not sure if we necessarily want to deprecate them..

[pmiam]

I'm happy to report no more issues over the last week. Your rename fix has performed perfectly and it's helped me re-organize my largest library's file tree without pains.

[pmiam]

I have another little patch to offer:

From 2aee8183e23dcfa49cfc94db89cae91be80e292f Mon Sep 17 00:00:00 2001
From: Panayotis Manganaris <[email protected]>
Date: Mon, 18 Nov 2024 14:35:47 -0500
Subject: [PATCH] notes [ITER] get formatted notes-name string

---
 papis/notes.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/papis/notes.py b/papis/notes.py
index bcc28176..4e643366 100644
--- a/papis/notes.py
+++ b/papis/notes.py
@@ -31,7 +31,7 @@ def notes_path(doc: papis.document.Document) -> str:
     from papis.paths import normalize_path
 
     if not has_notes(doc):
-        notes_name = papis.format.format(papis.config.getstring("notes-name"), doc,
+        notes_name = papis.format.format(papis.config.getformattedstring("notes-name"), doc,
                                          default="notes.tex")
         doc["notes"] = normalize_path(notes_name)
         papis.api.save_doc(doc)
-- 
2.47.0

[alexfikl]

I have another little patch to offer:

Nice catch! Added it in the latest rebase.