Indexing crashes with dependency Nebulex and hangs in a loop

[philipgiuliani]

I have a problem with indexing in a project where I'm using the Nebulex dependency. I was able to reproduce the problem in a clean project.

Steps to reproduce

1. mix new example
2. Add {:nebulex, "~> 2.5"} as a dependency

Expected
Indexing works.

Actual result
Indexing crashes and hangs in a loop. The provided log line is repeated every few seconds.

2023-09-19T11:02:33.971460+02:00 error: Task #PID<0.241.0> started from #PID<0.233.0> terminating, ** (FunctionClauseError) no function clause matching in :elixir_aliases.do_concat/2, (elixir 1.15.4) src/elixir_aliases.erl:150: :elixir_aliases.do_concat([{:__MODULE__, [line: 5, column: 15], nil}, :DynamicSupervisor], "Elixir.Nebulex.Adapters.Local.Backend.Shards"), (elixir 1.15.4) src/elixir_aliases.erl:138: :elixir_aliases.concat/1, (lx_common 0.3.0) lib/lexical/ast/aliases.ex:240: LXical.Ast.Aliases.Reducer.push_scope/4, (lx_common 0.3.0) lib/lexical/ast/aliases.ex:119: LXical.Ast.Aliases.Reducer.apply_ast/2, (lx_common 0.3.0) lib/lexical/ast.ex:251: anonymous fn/5 in LXical.Ast.prewalk_until/5, (lx_sourceror 0.12.3) lib/sourceror/zipper.ex:420: LXSourceror.Zipper.do_traverse_while/3, (lx_common 0.3.0) lib/lexical/ast.ex:232: LXical.Ast.prewalk_until/5, (lx_common 0.3.0) lib/lexical/ast/aliases.ex:308: LXical.Ast.Aliases.at/3, Function: #Function<9.44212591/0 in LXical.RemoteControl.Search.Indexer.async_chunks/3>, Args: []

Note: I've tested it in theza-async-indexing branch and in the main branch. I've also tested various Elixir versions.

Seems that this is the offending line: https://github.com/cabol/nebulex/blob/v2.5.2/lib/nebulex/adapters/local/backend/shards.ex#L5

Update:

After looking at the Nebulex code, I found the exact offending line. Using __MODULE__ inside a defmodule is producing the problem:

defmodule Example do
  defmodule __MODULE__.Hello do
    def world(), do: :boom
  end
end

[zachallaun]

Thank you for tracking this down! Looks like it should be an easy fix.

@scohen The problem is that __MODULE__.Foo parses as

{:__aliases__, [], [{:__MODULE__, [], nil}, :Foo]}

so we can't pass the args to Module.concat without unwrapping the first arg. We have to handle this in other places as well, so maybe we should add a Lexical.Ast.Module.concat(segments) that handles it?

There are other cases that we'll need to handle as well, the first I can think of is something like:

namespace = ...

defmodule unquote(namespace).Inner do
  ...
end

Basically, the first argument in an :__aliases__ node can be an arbitrary complex expression that we may not be able to resolve. In these cases we probably need to skip indexing the definition.

[scohen]

As an aside, this is insane

defmodule Example do
  defmodule __MODULE__.Hello do
    def world(), do: :boom
  end
end

that's exactly the same thing as

defmodule Example do
  defmodule Hello do
    def world(), do: :boom
  end
end

[philipgiuliani]

Yeah 😅 Was already thinking about opening a merge request in Nebulex to "fix" it. But since it's valid Elixir Syntax, who knows which other projects use this..

[scohen]

Agreed we should support it, I was just telling @zachallaun about this nutty syntax in another PR, but my goodness was it ever beaten with the ugly stick.

[scohen]

namespace = ...

defmodule unquote(namespace).Inner do
  ...
end

@zachallaun is this valid? I tried it, and elixir complains of "using unquote outside of a quote"

[renatoaguiar]

As an aside, this is insane

defmodule Example do
  defmodule __MODULE__.Hello do
    def world(), do: :boom
  end
end

that's exactly the same thing as

defmodule Example do
  defmodule Hello do
    def world(), do: :boom
  end
end

@scohen That's not right. It actually expands to:

defmodule Example do
  defmodule Elixir.Example.Hello do
    def world(), do: :boom
  end
end

I also thought the same, but I found that out after trying to "fix" Nebulex code:

diff --git a/lib/nebulex/adapters/local/backend/shards.ex b/lib/nebulex/adapters/local/backend/shards.ex
index 949863a..36af0bc 100644
--- a/lib/nebulex/adapters/local/backend/shards.ex
+++ b/lib/nebulex/adapters/local/backend/shards.ex
@@ -2,7 +2,7 @@ if Code.ensure_loaded?(:shards) do
   defmodule Nebulex.Adapters.Local.Backend.Shards do
     @moduledoc false
 
-    defmodule __MODULE__.DynamicSupervisor do
+    defmodule Elxir.Nebulex.Adapters.Local.Backend.Shards.DynamicSupervisor do
       @moduledoc false
       use DynamicSupervisor
 

I'm using that as a workaround for now by pointing other projects to a local copy of Nebulex with that change applied.

[scohen]

@renatoaguiar so the actual module name is :Elixir.Elixir.Example.Hello? I'm afk right now, but that feels off to me.

[renatoaguiar]

@renatoaguiar so the actual maoduke name is :Elixir.Elixir.Example.Hello? I'm afk right now, but that feels off to me.

No, it will be :Elixir.Example.Elixir.Example.Hello. It picks up the first Elixir.Example automatically from parent's name and the second one comes from explicitly added __MODULE__ (which is also translated to Elixir.Example).

[philipgiuliani]

Your explanation makes sense. That's what I initially expected to become. But according to this simple example in iex, it seems to be just Example.Hello.

That said, with the fix in PR #393 I still get an error, but its a bit different now. I actually think its a different case because it doesn't mention Nebulex in the error anymore:

2023-09-28T09:32:52.218874+02:00 error: ** Task <0.6545.0> terminating, ** Started from <0.7198.0>, ** When function == #Fun<Elixir.LXical.RemoteControl.Search.Indexer.9.43595129>, ** arguments == [], ** Reason for termination == , ** {function_clause,[{elixir_aliases,do_concat,[[{'__MODULE__',[{line,5},{column,15}],nil},'DynamicSupervisor'],<<"Elixir">>],[{file,"src/elixir_aliases.erl"},{line,154}]},{elixir_aliases,concat,1,[{file,"src/elixir_aliases.erl"},{line,142}]},{'Elixir.LXical.Ast',expand_aliases,4,[{file,"lib/lexical/ast.ex"},{line,476}]},{'Elixir.LXical.RemoteControl.Search.Indexer.Extractors.Module',resolve_alias,2,[{file,"lib/lexical/remote_control/search/indexer/extractors/module.ex"},{line,109}]},{'Elixir.LXical.RemoteControl.Search.Indexer.Extractors.Module',extract,2,[{file,"lib/lexical/remote_control/search/indexer/extractors/module.ex"},{line,23}]},{'Elixir.LXical.RemoteControl.Search.Indexer.Source.Reducer','-apply_extractors/2-fun-0-',2,[{file,"lib/lexical/remote_control/search/indexer/source/reducer.ex"},{line,70}]},{'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,[{file,"lib/enum.ex"},{line,2468}]},{'Elixir.LXical.RemoteControl.Search.Indexer.Quoted','-index/2-fun-0-',2,[{file,"lib/lexical/remote_control/search/indexer/quoted.ex"},{line,8}]}]}

[renatoaguiar]

Your explanation makes sense. That's what I initially expected to become. But according to this simple example in iex, it seems to be just Example.Hello.

Interesting. I guessed what it would happen in the example based on my Nebulex "fix", but it looks like I was wrong. However, I still don't understand why it seems to behave differently in the Nebulex case. When I tried to just remove the __MODULE__ prefix, I got this error:

== Compilation error in file lib/nebulex/adapters/local/backend/shards.ex ==
** (UndefinedFunctionError) function Nebulex.Adapters.Local.Backend.Shards.DynamicSupervisor.__using__/1 is undefined (function not available)
    Nebulex.Adapters.Local.Backend.Shards.DynamicSupervisor.__using__([])
    lib/nebulex/adapters/local/backend/shards.ex:7: (module)
    lib/nebulex/adapters/local/backend/shards.ex:5: (module)

Anyway, I'm now using the PR #393 instead and the crash loop is gone.

[scohen]

ok, i'm officially confused; @renatoaguiar did you use your patched nebulex, or the released version?

[renatoaguiar]

ok, i'm officially confused; @renatoaguiar did you use your patched nebulex, or the released version?

• Nebulex 2.5.2 with lexical main => crash loop
• Nebulex 2.5.2-patched with lexical main => works (NO crash loop)
• Nebulex 2.5.2 with lexical fix-module-alias (PR) => works (NO crash loop)

Sorry for the confusion.

[scohen]

now i'm wondering why you're seeing no problems, but @philipgiuliani is.

[lukaszsamson]

As an aside, this is insane

defmodule Example do
  defmodule __MODULE__.Hello do
    def world(), do: :boom
  end
end

that's exactly the same thing as

defmodule Example do
  defmodule Hello do
    def world(), do: :boom
  end
end

There's also a difference with aliases. The second version aliases Example.Hello as Hello. The first one creates an external submodule and no alias is set.

[scohen]

There's also a difference with aliases. The second version aliases Example.Hello as Hello.

That difference is only visible within the context of Example, correct? Outside of that context, there's nothing different about those modules than if you wrote them as defmodule Hello do right?

[lukaszsamson]

That difference is only visible within the context of Example, correct?

Yes, aliases are scoped