require fails when requiring a macro-utils file from other macro files (Fennel Issue #406)

[datwaft]

It seems that because Conjure disables the compiler sandbox it is also affected by bakpakin/Fennel#406.

Inside the issue is how to reproduce the issue.

Is there a way to activate the compiler sandbox so this issue doesn't happen?

I opened this issue mainly for tracking, but a workaround would be wonderful.

[Olical]

So this is compilerEnv: _G doing this? This is required for the Aniseed module macros to work as far as I'm aware 🤔 I'll have a tinker though.

[Olical]

Yeah, if I don't set that I can't access package.loaded in the module macros 🤔

[datwaft]

So this is compilerEnv: _G doing this?

Yes, you can see rktjmp/hotpot.nvim#48 for more specific information. Its a combination of another compiler option together with compilerEnv = _G.

[datwaft]

bakpakin/Fennel#406 is fixed now 🎉

[datwaft]

Is there a planned date for Conjure to use Fennel v1.1.0?

As you can see this issue is still happening on the latest version of Conjure:

But the bug was fixed on the release of Fennel v1.1.0 (here are the release notes).

• Fix a bug where disabling the compiler sandbox broke module require scope

[Olical]

Oh nice! I'll get that updated soon! Thank you for the heads up!

[Olical]

This is merged and released now, will close assuming it's all good now. If not, something else might be wrong!

[datwaft]

It seems that the bug is still reproduced.

On both Hotpot and raw Fennel the bug is no longer reproducible, so it is weird that it is reproduced on Conjure.

[Olical]

Hmm, okay, I'll have to try and reproduce it. You're sure you had the latest commit from this morning?

commit 842c81892648de759e639ad2d395757b98be06d5 (tag: v4.33.2, origin/master, origin/HEAD, master)
Author: Oliver Caldwell <[email protected]>
Date:   Fri Apr 29 09:32:27 2022 +0100

I've run into a few issues where people thought they'd updated but their plugin manager hadn't done it for some reason.

[Olical]

Also trying to understand the issue: If you try to require macros from another macro file it fails to load the helpers which breaks the parent file?

[Olical]

I can't reproduce this, I used (import-macros... to import a file then in there I pulled in another macro file and I can reference them no problem 🤔 which lends to my "something didn't update fully or properly" theory, but I'm very open to being proven wrong!

[Olical]

The other thing it could be is that your macro helpers file isn't on the path like the other macro modules are for some reason.

[datwaft]

Hmm, okay, I'll have to try and reproduce it. You're sure you had the latest commit from this morning?

As you can see in the screenshot, yes:

Also trying to understand the issue: If you try to require macros from another macro file it fails to load the helpers which breaks the parent file?

Not exactly, its if I try to require helper functions from a macro file.

I have a file called helpers.fnl which contains some helper functions like fn?, which checks if the input is a list and if the first element is the symbol hashfn or fn, when requiring this file from a macro file the bug is produced.

Here is an example of the code: https://github.com/datwaft/fennel-issue_macro_require

[datwaft]

Here is the important snippet:

(local {: fn?} (require :macro-utils))

(lambda if-fn-hello-else-bye [x]
  (if
    (fn? x) `(print "Hello")
    `(print "Bye")))

{: if-fn-hello-else-bye}

[Olical]

Got it, I understand now, will try to repro and work it out.

[datwaft]

Are there any updates?

I am still having this issue and it blocks me for using Conjure for testing my Neovim Fennel configuration.

As I said before the issue should have been solved in Fennel, so it's probably something related to Aniseed or Conjure.

Thank you for your help, btw!

[Olical]

Olical/aniseed#128 I think this is just this issue too. Haven't fixed it yet, sorry 😢 I keep working on other things and have been really busy recently. Will try to get this fixed very soon. If anyone wants to look into it, I think we just need to set the fennel path to the same as the lua path or something in the Aniseed compiler module.

[datwaft]

Hello!

It has been a few months so I thought I would ask if there has been any progress with this issue.

As you can see here I can still reproduce it:

[Olical]

I just spent another 3 hours trying to get to the bottom of this and still can't work it out, I'm sorry. I just don't know how to get the fennel compiler to do what I want. I can however get this working just fine if I use import-macros to load my macro util file, that works perfectly with static compilation and dynamic eval through Conjure.

I really need some help with this, if anyone can spot what args I can set or change to get the fennel compiler to treat requires inside macro files as further macro requires, please let me know.

Relevant lines are here https://github.com/Olical/aniseed/blob/a7445c340fb7a0529f3c413eb99d3f8d29f50ba2/fnl/aniseed/compile.fnl#L26-L42

Maybe it's because I use compileString instead of some file based functions? I've also made it so (on the dev branch) anything under a macro or macros directory is considered a macro and won't be statically compiled, hopefully this makes macro file organisation easier.

This test bed works for me, for context:

;; main.fnl
(import-macros {: if-fn-hello-else-bye} :macros.user)

(if-fn-hello-else-bye :hey)


;; macros/user.fnl
(import-macros {: is-fn} :macros.utils)

(lambda if-fn-hello-else-bye [x]
  (if
    (is-fn x) `(print "Hello")
    `(print "Bye")))

{: if-fn-hello-else-bye}


;; macros/utils.fnl
{:is-fn (fn [x]
                (= `fn (. x 1)))}

It's just the ability to use a regular require inside a macro file and for the compiler to know to load that as a macro as well that doesn't work. I've updated the Fennel compiler, maybe there's an option I need to set to get that working? Possibly to do with the options I provide to the compiler?

            (fnl.compileString
              (a.merge! {:allowedGlobals false
                         :compilerEnv _G} opts))

[datwaft]

Thanks for your effort!

Maybe @rktjmp can help as Hotpot doesn't have any problem with my use case after bakpakin/Fennel#406 was merged.

Sadly I don't have much experience tinkering with Fennel's internals so I don't have proper idea about how to fix this.

Maybe it's an upstream issue if fennel.compileString has the issue and bakpakin/Fennel#406 was only fixed for fennel.load or something like that.

Fun fact: when using stdio with Fennel it works without any problem:

As I don't need Aniseed maybe a workaround could be to use the stdio and somehow add to its path Neovim's API.

---

This test bed works for me, for context:

Btw, that test bed would always output Bye as the symbol x is what is passed to is-fn. That is why it's important to be able to use require instead of import-macros.