dune subst crashes on large files (32-bit systems)

[emillon]

dune subst crashes on 32-bit systems when a large file is present:

(true as of 3.12.1)

  Error: exception Invalid_argument("Bytes.create")
  Raised by primitive operation at Stdune__Io.Make.eagerly_input_string in file
    "otherlibs/stdune/src/io.ml", line 273, characters 14-30
  Called from Stdune__Io.Make.read_all.(fun) in file
    "otherlibs/stdune/src/io.ml", line 308, characters 16-40
  Called from Stdune__Exn.protectx in file "otherlibs/stdune/src/exn.ml", line
    10, characters 8-11
  Re-raised at Stdune__Exn.protectx in file "otherlibs/stdune/src/exn.ml", line
    16, characters 4-11
  Called from Dune__exe__Subst.subst_file in file "bin/subst.ml", line 113,
    characters 10-27
  Called from Stdlib__List.iter in file "list.ml", line 110, characters 12-15
  Called from Fiber__Core.O.(>>|).(fun) in file "vendor/fiber/src/core.ml",
    line 252, characters 36-41
  Called from Fiber__Scheduler.exec in file "vendor/fiber/src/scheduler.ml",
    line 76, characters 8-11
  Re-raised at Stdune__Exn.raise_with_backtrace in file
    "otherlibs/stdune/src/exn.ml", line 38, characters 27-56
  Called from Fiber__Scheduler.exec in file "vendor/fiber/src/scheduler.ml",
    line 76, characters 8-11

[moyodiallo]

dune subst crashes on 32-bit systems when a large file is present:

Can you give a size (example) ?

[emillon]

There's a test case in #9539.
This is caused by the read_file call here:

https://github.com/ocaml/dune/blob/3.13.0/bin/subst.ml#L113

(the root cause is that the size of strings needs to fit in the header of an ocaml block)

In term of fix, personally I would consider it enough to catch the exception and log a warning. It's extremely unlikely that some users rely on substitution working on large files. If that's the case, we'll see another bug report.

[moyodiallo]

In term of fix, personally I would consider it enough to catch the exception and log a warning. It's extremely unlikely that some users rely on substitution working on large files. If that's the case, we'll see another bug report.

You're right we'll see another bug report cause 16MB is not that huge though. I think if we can fix it without a performance loss, it could be fine.

[emillon]

Please, let's just emit a warning for this. 16MB files exist, but we don't need to make substitutions in them for now. There are more important fixes to do first.