[BUG] Io.copy_channels is called concurrently under Windows (at least), resulting in corrupted output

[nojb]

Recently at LexiFi we observed what looks like a bug in Dune. When building a medium-size workspace under Windows, sometimes the target of copy_files# will get corrupted. Concretely, the workspace includes Menhir, and the file src/stage2/lexmli.ml (which is an ocamllex-generated lexer and is copied using copy_files# from src/lexmli.ml) is sometimes corrupted:

For future reference, the symptom is that the compiler complained from an illegal backslash in this file:

Looking at the code (copy_files# ...) is implemented using Io.copy_channels from stdune which uses a global buffer:

dune/otherlibs/stdune/src/io.ml

Lines 68 to 79 in 25be1ea

	let copy_channels =
	let buf_len = 65536 in
	let buf = Bytes.create buf_len in
	let rec loop ic oc =
	match input ic buf 0 buf_len with
	| 0 -> ()
	| n ->
	output oc buf 0 n;
	loop ic oc
	in
	loop
	;;

Accordingly, the first suspicion is that this function is called concurrently. To test this conjecture I instrumented the code:

diff --git a/otherlibs/stdune/src/io.ml b/otherlibs/stdune/src/io.ml
index 576021ffb..0dbaae703 100644
--- a/otherlibs/stdune/src/io.ml
+++ b/otherlibs/stdune/src/io.ml
@@ -68,6 +68,7 @@ let input_zero_separated =
 let copy_channels =
   let buf_len = 65536 in
   let buf = Bytes.create buf_len in
+  let busy = ref false in
   let rec loop ic oc =
     match input ic buf 0 buf_len with
     | 0 -> ()
@@ -75,7 +76,14 @@ let copy_channels =
       output oc buf 0 n;
       loop ic oc
   in
-  loop
+  fun ic oc ->
+  if !busy then begin
+    Printf.printf "XXXXXXXX\n%!";
+    Printexc.print_raw_backtrace stdout (Printexc.get_callstack 20)
+  end;
+  busy := true;
+  loop ic oc;
+  busy := false
 ;;

and indeed, the function is called concurrently:

@rgrinberg: should we do the safe thing here and use a separate buffer for each call? (We could optimize that by using a global buffer but if the function is called again while the global buffer is being used, then use a fresh buffer.) What do you think?

[nojb]

@rgrinberg: should we do the safe thing here and use a separate buffer for each call? (We could optimize that by using a global buffer but if the function is called again while the global buffer is being used, then use a fresh buffer.) What do you think?

See #11194 for this approach.