Sourcemap: Mapping of end of blocks is unexpected

[paulpflug]

func =>
  inBlock = true

compiles to

func(() => {
  var inBlock;
  return inBlock = true;
});

Each column of the end of the expression statement in js }; (line:4, column: 1 to 3} is mapped to the start of the expression in cs (line:1, column:0), expected would be end (line:2, column:15).

This makes it impossible to find expressions in js and translate back to coffee.

Boiled down, what I'm trying to do:

cs = require "coffeescript"
acorn = require "acorn"
sourceMap = require "source-map"

src = """
func =>
  inBlock = true
"""

{js,v3SourceMap} = cs.compile src, sourceMap: true, header: false, bare:true

# js:
# func(() => {
#   var inBlock;
#   return inBlock = true;
# });

ast = acorn.parse js, locations: true
loc = ast.body[0].loc # start: { line: 1, column: 0 }, end: { line: 4, column: 3 } }

consumer = new sourceMap.SourceMapConsumer(v3SourceMap)
start = consumer.originalPositionFor loc.start # { line: 1, column: 0 }
end = consumer.originalPositionFor loc.end # { line: 1, column: 0 }

# should have the original coffee expression
# but doesn't
console.log src.split("\n").slice(start.line-1, end.line).join("\n")

https://sokra.github.io/source-map-visualization/#coffee also shows this strange mapping

[GeoffreyBooth]

I assume this is the visualization you mean?

I’m not sure the behavior you describe is incorrect. Just taking the last );, it gets mapped to func:

I can see the argument for why this is correct: that ); is part of the func function call, even though both of those characters are implicit in CoffeeScript; and since there’s no ) or ; to map to in the original CoffeeScript, we might as well map to what we do have, which is func. What else would you expect to happen?

It seems like the bigger picture is that you want to get a proper abstract syntax tree out of the CoffeeScript compiler, and you’re using source maps as an approximation for that. I proposed some improvements along those lines in coffeescript6/discuss#92. See also #4556.

[paulpflug]

In my understanding ); is not really implicit but is directly linked to newline followed by a lesser indentation. So maybe it could be linked to the newline character?

I simplified my case by leaving out, that my approach has to work for any js compiler.
I get the compiled js and source map and have to find specific calls in the original source.
This works for babel and typescript, but not for coffeescript..

[GeoffreyBooth]

Source maps map characters to nodes of the syntax tree. Whitespace isn't mapped. ); is part of func, so it needs to be mapped to that.

[paulpflug]

First, thanks for the time you are investing 👍

Then I must say I disagree:
As you wrote:
Source maps map characters to nodes of the syntax tree
A node consists of more than just the beginning of the node, the end is equally important.
The information, where the node ends is not contained in the current approach and that is exactly my problem. I want to find the node, but I only have the information where it starts.

Where did you get Whitespace isn't mapped from? I can't find any information on how to do the mapping.
But even if this would be the convention, when talking about whitespace-significant languages this doesn't make sense.
From information-density-view some newlines in coffeescript carry the same information as the ) or } in js and should be treated equally.

I hope I don't offend you by opposing.

[GeoffreyBooth]

When I say that whitespace isn’t mapped, I just mean that at present, in CoffeeScript, it isn’t. But as far as I can tell, other tools that generate source maps (see above) don’t map whitespace either. Something to understand is that source maps are tracking grammar—func in the original source gets turned into func in the generated JS, and the source map is tracking the former to the latter. An indentation or a newline in the original CoffeeScript is significant to CoffeeScript, but it’s not part of the grammar per se; it doesn’t correspond to anything in particular in the generated JavaScript. CoffeeScript of a =   3 just becomes JavaScript of a = 3; there’s no effort to map each space in the original to anything in the latter, and therefore there are no source maps for those whitespace characters. Or put another way, whitespace isn’t part of the CoffeeScript AST, just as comments aren’t.

Another thing to remember is that source maps are unidirectional: they go from the CoffeeScript to the JavaScript, not the other way around. The point of source maps is to be able to set breakpoints in a debugger. For that purpose, when I click on the CoffeeScript func, I want it to set a breakpoint at the start of the JavaScript func. The only length/end that matters is on the CoffeeScript side, so that the debugger knows whether I clicked on func or the next mapped item. I suppose you could probably figure out the length by measuring the start coordinates of one item to the start coordinates of the next item, but that doesn’t help you for cases like func = => { ... } where there are intermediate items.

Rather than working from the source maps, I think what you should look into is working from the CoffeeScript nodes. Go to http://asaayers.github.io/clfiddle/ and click on AST, and click on any of the nodes in the tree. You’ll see last_line and last_column values.

[paulpflug]

Another thing to remember is that source maps are unidirectional: they go from the CoffeeScript to the JavaScript, not the other way around.

All sort of error-generation need the js -> coffeescript path.
I'm trying to get some context to a error similar to https://github.com/avajs/ava#magic-assert

And I didn't propose to map all the whitespaces, but only the one significant for the end of a block.
In my eyes, only the last mapping from ) -> func must be changed to ) -> \n

[GeoffreyBooth]

@lydell?

[lydell]

Sorry, I have only skimmed through this thread.

As far as I know, source maps are used for two things, mainly in browsers:

• Translate stack traces to the original locations. JS -> CS (Similarly, for CSS, the inspector shows the original locations of rules. CSS -> SCSS)
• Set breakpoints in the original code. CS -> JS

It can be debated what is the most "correct" mapping for every token, but all that matters (I think) is how well the browsers' devtools manage to do the above.

I don't understand what your end goal is @paulpflug? What are you trying to do? (Sorry if I missed that among all the comments.) An example would be perfect. Input CSS code, and your desired output. What you need to do that. Why you encounter problems.

Also remember that it is possible to map generated code to nothing (JS -> null) if that helps.

[paulpflug]

I want to search generated js ast for function calls (normally multi-line) and output those in the original used language. But no information about the end of those calls is contained in the coffeescript source map. Use case is a test runner where the test content is printed along with the error.

[GeoffreyBooth]

I’m sorry, I’m just not understanding what you’re asking for. Looking at https://sokra.github.io/source-map-visualization/, even for Babel all I see are mappings from the start of an item in one language to the start of the corresponding item in the other language. I don’t see any lengths or end positions.

If you know where a function call ends on the JavaScript side, by finding }, then presumably the very next item on the JavaScript side is also the next item after the function call on the CoffeeScript side; and you could measure the start coordinates from the function call to the start coordinates of this following item, to determine the length of the function call.

If that doesn’t work, I think the next best thing is to use the CoffeeScript AST, as you can see in http://asaayers.github.io/clfiddle/ (click on the AST tab), as it does have start/end coordinates. You should be able to map the AST to the source maps, as the coordinates should match.

Even if we wanted to change something in the CoffeeScript compiler to make this easier for you, it’s not clear to me what exactly that would be. I would need detailed examples of source maps now versus the different source maps you wished were generated, and an explanation of how the latter is different (and how it could be generated, at least manually).