Max Line Length for Pretty-Format-ToML / Don't reformat multiline strings?

[Skylion007]

I tried running it on my project and it made my toml file very difficult to read with it's long exclude pattern.: facebookresearch/habitat-sim@91f8ea5

Enforcing a max line length would do wonders for readability in this case or at least not collapsing multiline strings like that.

Alternative not destroying multiline strings woudl also be helpful.

[macisamuele]

@Skylion007 thanks a lot for the reporting.
I'll be looking in the next days on how we can, if possible, address this.

I'm mentioning the "if" because we depend on an external library for doing the toml parsing and formatting and I'm not sure that the library supports it (or if there are other libraries that do it).

Side note: While checking #26 I've also noticed that the current pretty-format-toml strips comments as well

[Skylion007]

@macisamuele Seems like there is a quick and dirty fix here: uiri/toml#236 (comment)

[Skylion007]

This one also seems like it could do the job: https://github.com/jumpscale7/python-pretty-toml/

[CAM-Gerlach]

This is also a blocker for adoption for me, since pretty much all of the TOML files I use are heavily commented. Like on #26 , both problems should be pretty neatly solved by switching to tomlkit which has preserving comments, order, indentation and line breaks as its primary design goal. Its actively maintained, supports the latest 1.0.0 draft spec and seems to be pretty well-regarded (pip considered switching to it, though I think they went with toml instead since its more minimal and the extra features weren't needed for pip's application).

I couldn't find an explicit feature for reflowing line length automatically at least after skimming the code, but you can use it for interactive editing so its possible that could be added or done with some work, but at least it will respect your existing multi-line strings and other formatting (important for me).

The API should be identical or nearly so, so it should be close to a drop-in with at most a few lines of changes, and has pretty minimal dependencies. I'd be happy to go ahead with it, though if @Skylion007 you want to take it that's fine too of course.

[CAM-Gerlach]

@Skylion007 That looks like a really interesting option that seems to do a lot more explict prettifying than a straight parser like toml or tomlkit.

However, it only has 8 stars, no releases and hasn't been touched in over two years, since shortly before 0.4.0 was first released (two major versions behind the current 1.0.0rc2 spec), and implements a number of somewhat idiosyncratic rules with no apparent customizability (e.g. tabs for separating comments, hardcoded 2 space indents, hardcoded 120 character line length, mandatory key ordering, etc) which will likely be problematic for many users since there is no accepted style standard to TOML that would justify a Black-like approach.

So, it certainly might be something to base future work on, but as-is I would recommend dropping in tomlkit for now and considering perhaps adding something like that for the future. What are your thoughts?

[Skylion007]

@CAM-Gerlach Yeah, probably right. Might be something to consider forking and fixing up though.

As an end user, all I care is that it works and is customizable (or at the very least has sensible defaults / hard coding).

[Skylion007]

@CAM-Gerlach Feel free to go ahead and tackle it and switcher over to tomlkit.

[macisamuele]

@Skylion007 @CAM-Gerlach about TOML support comments, the toml library should in theory supposed to deal with them but apparently seems that the implementation is buggy.
I'll be looking in ways that I can support uiri/toml to properly address it.
If we'll see that the maintainer is unresponsive and/or fixing it would require much more effort I can look into alternatives.

About the multiline issue, my 2cents would be:

• use multiline toml if the string contains line breaks (either was originally a multiline string or something like something\non the new line)

Again (as in #40) I'd like to limit as much as possible the amount of customisation on the provided tool

[macisamuele]

I checked tomlkit and seems promising. This night/tomorrow I'll try to publish a PR addressing the different concerns.
I'm not going to directly target line length but trying to prefer usage of toml multiline strings if the string contains line breaks.

I'll keep you posted

[CAM-Gerlach]

@macisamuele Thanks. AFAIK comments are still unsupported in uiri/toml , per uiri/toml#77 and empirical testing. They appear open to a PR, but it appears to be a pretty substantial change vs. switching to a drop-in replacement package. It also doesn't preserve indentation as does tomlkit, which is commonly used (as suggested in the spec) to make TOML easier to read, especially with nested tables.

Ideally it would conform indentation to a specific standard automatically instead of just preserving it, but you should be able to use tomlkit's API to enforce that on the "AST" equivalent rather than with manual string munging, either now or at some point in the future.

[macisamuele]

About comments, toml kinda support the but in a very very limited and bugged fashion. I've tried to fix it in the code but feels way far from easy.

A substantial change does not concern me too much as far as maintainable.
I checked tomlkit implementation and feel well organised and shouldn't be too hard or hacky do do what we want.

About table indentation is not clear to me what you mean. My opinion is that we should force indentation to not be present or always present (accordingly to what is easier except if we have stronger arguments)

[CAM-Gerlach]

About table indentation is not clear to me what you mean. My opinion is that we should force indentation to not be present or always present (accordingly to what is easier except if we have stronger arguments)

I'm talking about using indents to make the hierarchical table structure clear, just as its used in most/all other config formats and programming languages. E.g.

[general]
spam = "eggs"
[subtable]
    sub_item = "foo"
    sub_item_2 = "bar"

Automatically applying the correct indentation is the "prettyfying" task that in my experience takes the most effort by hand, and it would be a big win to include it, but if that's out of scope here, it should at least be preserved for now (as tomlkit does by default) to avoid making existing TOML files less readable, as removing all indentation would. For a small, flat config file, it doesn't matter so much, but it makes a huge difference on larger ones with more nesting; e.g. this one would be much harder to understand without it.

The only things that need be customizable, IMO, are the indentation character and a sort_keys option (which can be passed through directly to tomlkit), just like some of your other prettifiers offer.

[macisamuele]

#46 has been merged and released in v2.0.0

Closing this for now.
Feel free to reopen this if not all the concerns have been covered