Add mult-line and literal strings

[mojombo]

Extension of #228.

• Clarifies some of the language.
• Allows for """\ as opening delimiter for multi-line basic strings.
• Renames "raw string" to "literal string".

[BurntSushi]

LGTM.

[mojombo]

@wycats I'd love your input on this one, if you have a minute. It's a significant (but backwards compatible) addition to the spec.

[wycats]

@mojombo I took a look and this seems pretty reasonable. I wonder if the difference between single- and double-quoted strings is too subtle, but Toml is a pretty simple language and people can read the docs if they're confused.

Any Unicode character may **be used except those that must be escaped

I assume this doesn't mean to disallow UTF-8 encoded Unicode characters?

If the first character after the opening delimiter is a newline (0x0A), then it is trimmed

I'm inclined to suggest the same treatment for a trailing newline to enable:

key3 = """
One
Two
"""

Since there is no escaping, there is no way to write a single quote inside a literal string enclosed by single quotes

An alternative solution to this problem would be to allow two adjacent ' characters to represent an escaped single-quote. I can't think of any obvious downsides to that approach.

Luckily, TOML supports a mult-line version of literal strings that solves this problem

Typo in mult-line.

---

One specific use-case worth considering explicitly is shell-escaping in commands embedded in string literals. I think that single-quote raw strings solve this use-case.

[BurntSushi]

I'm not really a fan of trimming the trailing new line since it isn't usually what you want. If we did, the common case would be:

key3 = """
One
Two

"""

I'm pretty ambivalent about ''. I can definitely see its benefits. But it's also another rule. I'm fine either way I think. (We could opt to leave out the '' escaping for now and add it later. It would be backwards compatible because ' is otherwise outlawed in the current proposal.)

[wycats]

@BurntSushi Sounds fine re: trailing newline. I don't see a good reason not to allow ' escaping, to avoid requiring ''' just to use a single quote in raw strings.

[ChristianSi]

I don't think the "double ' to escape it" rule would be very useful. Writing '''... don't ...''' instead of '... don''t ...' shouldn't cause serious inconvenience.

On the other hand, allowing '' in '-strings to mean ', while in '''-strings it would (of course) still mean '' could be confusing, I think.

[ChristianSi]

One issue that might deserve an explicit clarification is whether, if a triple-quoted strings ends with 4 or 5 quotes, the first 1 or 2 of them are part of the content of the string. If yes, the string

"This", he said, "is the most awful thing I've heard in my life."

could be written like this:

""""This", he said, "is the most awful thing I've heard in my life.""""

If not, the final quote must be escaped. That's what Python seems to expect.

I'm not biased on way or the other but think a clarification would be in order to help parser writers.

[mojombo]

Any Unicode character may **be used except those that must be escaped

I assume this doesn't mean to disallow UTF-8 encoded Unicode characters?

Correct. The intention is that any UTF-8 encoded character is ok to use except those explicitly mentioned.

Since there is no escaping, there is no way to write a single quote inside a literal string enclosed by single quotes

An alternative solution to this problem would be to allow two adjacent ' characters to represent an escaped single-quote. I can't think of any obvious downsides to that approach.

I'd prefer to keep literal strings 100% literal. It makes it easier to decide which string type to use. It also prevents any ambiguities, for instance, what does '''foo''' represent? With '' escaping, it could mean either 'foo' as a literal string or foo as a multi-line literal string. The current proposal always gives you a good, clear alternative when you run up against a string constraint, and the number of rules is small. I like that.

One issue that might deserve an explicit clarification is whether, if a triple-quoted strings ends with 4 or 5 quotes, the first 1 or 2 of them are part of the content of the string. If yes, the string

"This", he said, "is the most awful thing I've heard in my life."

could be written like this:

""""This", he said, "is the most awful thing I've heard in my life.""""

If not, the final quote must be escaped. That's what Python seems to expect.

I think it's reasonable and straightforward to require that quotes be escaped in that scenario. The delimiter is """ and once the parser finds it, that should be it. I'll work on clarifying the text to communicate the fact.

Any other final concerns from anyone? I'd like to merge this soon.

[mojombo]

@ChristianSi I just pushed a commit (7f33170) that should address the quotation mark escaping you brought up.

[BurntSushi]

@mojombo 👍

[ChristianSi]

@mojombo 👍

[mojombo]

Merged!