Thinking Elixir podcast and follow up questions

[josevalim]

@scohen, awesome job on the Thinking Elixir podcast interview (and to you as well @scottming).

If you don't mind, I have some follow up questions:

1. You mention UTF-16 encoding and that you work with diffs. I assume that this will be a problem once we implement Environments for code fragments/buffers (i.e. elixirsense) elixir-lang/elixir#12645? If you want to pass just a diff to Elixir, would you be able to compute the UTF-8 range of the diff and pass it forward? Or you can't do translate a UTF-16 range to UTF-8 without having the whole contents or at least the whole line? My gut says it is not possible but maybe it is hard to say upfront.

2. About namespacing, do you namespace the user code or Lexical code? Because I am not sure if I understand the need for namespacing. As you said, there are two Erlang VMs running: one for the language server and another that interacts with the user code (btw, I agree this is the right design). The language server is isolated and therefore it doesn't need namespacing. The runtime will interface with user code but because it communicates with the language server via Erlang distribution, there is less need for dependencies like Jason?

3. About dialyzer, honestly, I would not focus too much on it. If you have to, then focus on the new incremental mode in Erlang/OTP 26 which is much faster and will become the default in the future anyway.

4. What about compilation? Is my understanding correct that you currently do not compile the user project. Keep in mind that indeed there may be dragons here, especially because different compilations (LS compilation, Phoenix endpoint plug compilation, CLI compilation) may race each other and currently there is no locking (we are trying to get that into Erlang/OTP). The other thing is that some warnings will only happen once you compile as a project (Elixir compiler has two passes, a single file pass and a group pass that sees the whole project). Luckily, Mix compilers store all warnings, so if you decide to not compile yourself, we can add an API for you to retrieve the compilation warnings (without compiling).

Btw, Steve, do you mind shooting me a quick email? You can find it in my GH profile. :) And yes, I think all of you LS maintainers should have a convo on how to help each other. Thank you for the time!

[scohen]

@josevalim

1. In short, Don't worry about it. The UTF8 - UTF16 conversions happen at the boundary of the server, and I'd much rather work with utf8. Contributors to lexical won't have any idea that there's a conversion happening, and it's very fast. I wouldn't worry about it too much. For me, a bigger annoyance is the one-based line number and column counts (LSP is zero-based), yielded by compilation and mix, but we've fixed that too.
2. We namespace lexical's code, the code for lexical's dependencies, and lexical's plugins. Project (user) code is not namespaced. Most people won't need namespacing, since they're not working on lexical or one of its dependencies, but for those who are, it is very easy to break things. Namespacing was born out of this problem. I was working on code that runs inside the remote VM, and if I made a mistake, things would fall over and crash the project VM. The same is true for dependencies of elixir.
3. We have a task to work on dialyzer, but I haven't started on it yet. It does seem to be something that people want, but it's a whole can of worms. I don't know how I feel about it, but I'll implement it if there's enough interest.
4. We use mix to compile the whole project (into a separate build directory, <project_root>/.lexical/build), and we use the Code module to provide as-you-type compilation. Internally, we gate compilation with a GenServer and a build lock using :global. I'm worried about dragons, can you elaborate on what's concerning you? I was under the (maybe false) impression that since our build directory was separate from the project's build directory, we gate compilation with a lock and that we're using the mix tasks that we'd be fine. Is this not the case? That said, I'd love an API to get compile errors and warnings without compilation. Our current process involves parsing error strings, which isn't great, and is some of the uglier code in the codebase.

I really appreciate your thoughts about this space and making life easier for us, but I also would like to point out that adding new features to the language means that we're going to need to support two code paths for the foreseeable future. I'd like the project to have a good amount of backwards compatibility, and I wonder if there would be a way to maintain that, even with new features.

I'll shoot you an email. Thanks for the interest.

[josevalim]

Internally, we gate compilation with a GenServer and a build lock using :global. I'm worried about dragons, can you elaborate on what's concerning you?

I see. In this case, it can only happen on mix deps.get - but it is a bit more rare. Ideally though we wouldn't force the user to have several compilations (one for lexical, one for dev, one for test, etc) but that cannot be solved before the lock.

That said, I'd love an API to get compile errors and warnings without compilation. Our current process involves parsing error strings, which isn't great, and is some of the uglier code in the codebase.

Since you are already compiling, I believe you are already receiving the Mix.Tasks.Compiler.Diagnostics? In that case, you are already getting all information that we have. We are converting most of what we can at the compiler to diagnostics but random exceptions (since compiling is executing code) may still happen. But I believe we could additionally emit the compilation failures themselves as diagnostics, I will open up an issue. :) (done).

---

One additional question: you mention that you provide errors while formatting. Do you have to parse the output as well? Or can you hook up into the code and rescue the exceptions that arise from formatting? Can you link me to the code?

[scohen]

One additional question: you mention that you provide errors while formatting. Do you have to parse the output as well?

We're doing something kind of ugly, we kick off a complation attempt and catch exceptions during formatting. Kicking off a file compilation means that errors go through our normal processes.

[josevalim]

Thanks. I am taking notes and if I see anything I can improve I will let you know.