Lwt 3.2.0

[aantron]

Lwt is a promise and concurrency library for OCaml.

Release 3.2.0 continues the ongoing Lwt cleanup, adds some features, and fixes several bugs. It also schedules some breaking changes for Lwt 4.0.0, which is to be released in March 2018. See ocsigen/lwt#453 for a summary of these breaking changes, and help with adapting ahead of time.

From the changelog:

Additions

• Lwt_mvar.take_available, Lwt_mvar.is_empty (Add Lwt_mvar.(take_available, is_empty) ocsigen/lwt#459, Hezekiah Carty).
• Lwt_io.open_temp_file, Lwt_io.with_temp_file (Issue #336: Add temporary file creation capabilities to Lwt_io ocsigen/lwt#467, Joe Thomas).
• New reference documentation for module Lwt (New reference docs for module Lwt ocsigen/lwt#469).
• Lwt_pool.clear and ?dispose argument for Lwt_pool.create (Support disposal of Lwt_pool.t elements ocsigen/lwt#483, Hezekiah Carty).
• Lwt_pool.wait_queue_length (EAGAIN when a new system thread can't be created ocsigen/lwt#493, Jerome Vouillon).

Bugs fixed

• Lwt.npick never worked (Npick fix ocsigen/lwt#447, Zack Coker).
• Lwt_pool.use now always calls ?validate on elements (Bugfix: Adjust validation behavior of Lwt_pool.use ocsigen/lwt#461, Joe Thomas).
• Better locations generated by the PPX (Fix locations in generated lwt ppx ocsigen/lwt#470, Fabian Hemmer).
• Keep worker thread count accurate in Lwt_unix when pthread_create fails (EAGAIN when a new system thread can't be created ocsigen/lwt#493, @koen-struyve).
• Leaked exceptions in Lwt_list (Improve exception safety of some Lwt_list APIs ocsigen/lwt#499).
• Memory leak in Lwt_unix.getnameinfo (lwt_unix_free_job if getnameinfo was successfull ocsigen/lwt#503, Hannes Mehnert).

Planned to break in 4.0.0

See ocsigen/lwt#453 for details and instructions about planned breakage in Lwt 4.0.0.

• The semantics of Lwt will be adjusted for better exception and stack safety (Lwt semantic improvements: exception safety and stack overflows ocsigen/lwt#500).
• The PPX will be factored out into its own opam package, lwt_ppx. This package is installable from opam now, as of Lwt 3.2.0 (Move the PPX to a separate package (?) ocsigen/lwt#338).
• Similarly, the deprecated Camlp4 syntax will be factored out into lwt_camlp4, which is installable from opam now (Move Camlp4 syntax to a separate package ocsigen/lwt#370).
• Modules Lwt_log, Lwt_log_core, Lwt_log_rules, and Lwt_daemon are being deprecated and factored out into opam package lwt_log, also installable from opam now. Use the logs library for logging, in particular module Logs_lwt. Direct daemonization is deprecated on most platforms (attempt to move lwt.log into separate lwt_log ocsigen/lwt#484, Hannes Mehnert).
• The >> construct from the PPX will be deleted (Deprecate >> in ppx ocsigen/lwt#471, Raphaël Proust).
• Package lwt.preemptive is being merged into lwt.unix. In 3.2.0, lwt.preemptive becomes an alias for lwt.unix, and the package name lwt.preemptive will be deleted in 4.0.0 (Make lwt.preemptive an alias for lwt.unix; remove it in 4.0.0 ocsigen/lwt#487).

Deprecations

• Lwt.waiter_of_wakener should not be used, as it can lead to soundness bugs in future (but not current) Lwt (Contravariance of Lwt resolvers is unsound ocsigen/lwt#458).
• Lwt_sequence was deprecated in Lwt 2.6.0, but it now has a warning attached, as do Lwt.add_task_r and Lwt.add_task_l, which use it (Factor out Lwt_sequence (doubly-linked list) ocsigen/lwt#361).
• Use of the following functions is discouraged, but they have not yet received deprecation warnings: Lwt.with_value, Lwt.cancel, Lwt.state, Lwt.ignore_result (Discussion: deprecate Lwt.ignore_result ocsigen/lwt#359, New reference docs for module Lwt ocsigen/lwt#469).

Miscellaneous

• Replace references to Camlp4 in the manual with the PPX (Remove camlp4 reference from the manual ocsigen/lwt#457, Bobby Priambodo).
• More tests for Lwt_pool (Issue #385: Increase test coverage for Lwt_pool. ocsigen/lwt#464, Joe Thomas).
• Expect tests for the PPX (Add expect tests for ppx ocsigen/lwt#474, Fabian Hemmer).

[camelus]

❗ opam-lint warnings 1f29c5e

• lwt.3.2.0 has some warnings:

  • warning 41: Some packages are mentionned in package scripts of features, but there is no dependency or depopt toward them: "lwt_camlp4", "lwt_log",
    "lwt_ppx"

• These packages passed lint tests: lwt_camlp4.1.0.0, lwt_log.1.0.0, lwt_ppx.1.0.0, ocamlfind.1.7.3-1

---

✅ Installability check (8007 → 8012)

• new installable packages (5): lwt.3.2.0 lwt_camlp4.1.0.0 lwt_log.1.0.0 lwt_ppx.1.0.0 ocamlfind.1.7.3-1

[aantron]

Disappointingly, the CI setup that rebuilds Lwt's online manual is down, and has been for months. As a result, the new reference docs for module Lwt can only be seen here: https://ocsigen.github.io/lwt/manual-draft/Lwt.html

[aantron]

Okay, don't merge this. It seems the change to lwt.preemptive packaging affected some ocamlfind-based builds. I'll look into it, update the predicates, and update this PR.

[aantron]

Alright, the problem is as follows:

1. lwt.unix previously did not depend on Findlib package threads, but now it does (because lwt.preemptive was merged into it).

   As a result, when compiling against lwt.unix, it has become necessary to pass -thread. Jbuilder effectively always does that, but ocamlfind does not, so non-Jbuilder builds are broken by this.

2. It is possible to work around that in Lwt, by making the threads dependency conditional on whether -thread is passed or not. This is done by messing with Lwt's META file:

   requires(-mt) = "bigarray bytes lwt lwt.log result unix"
   requires(mt) = "bigarray bytes lwt lwt.log result threads unix"
   

   This solves the problem for some of the ocamlfind builds that refer to lwt.unix.

3. However, ocamlfind builds depending on both lwt.unix and another library compiled by Jbuilder against lwt.unix still do not work. For the example, take cstruct-lwt, which depends on lwt.unix, and nocrypto, which depends on both cstruct-lwt and lwt.unix.

   When Jbuilder is packaging cstruct-lwt, it takes the transitive closure of cstruct-lwt's dependencies, unconditionally assuming that mt is set. It therefore takes threads from lwt.unix, and makes it a direct, unconditional dependency of cstruct-lwt, which makes the nocrypto build fail.

There are several ways to solve this problem. I'd like to gather opinions on it.

1. Make Jbuilder aware of mt, and perhaps predicates in general, for the purpose of being compatible with the ocamlfind world. Jbuilder would automatically emit requires(mt) and requires(-mt) for dependency lists that include threads, and it would have to do a little bit of algebra on predicate lists when doing transitive closure.

   This has the advantage that we wouldn't need to maintain a custom META in Lwt for the mt and -mt variants, the fix would propagate automatically to cstruct, and ocamlfind users like nocrypto don't have to be changed. We would just have to require recent Jbuilder for the Jbuilder projects (lwt, cstruct). The disadvantage is that this may be annoying to maintain in Jbuilder.

2. Get rid of transitive closure of dependencies in Jbuilder. Why does it do that anyway?

3. Constrain and/or patch all the projects like nocrypto. I'd probably then wait to break lwt.preemptive and lwt.unix in 4.0.0.

4. Generate a different META for the threads package in opam switches, that doesn't raise an error if -thread is not supplied. I'm not sure how that would interact with the compiler, though.

5. Other ideas?

cc @rgrinberg @diml

[aantron]

FYI removing these lines from lib/threads/META in my opam switch:

error(-mt) = "Missing -thread or -vmthread switch"
error(-mt_vm,-mt_posix) = "Missing -thread or -vmthread switch"

changes the error message, when compiling a project that does use threads (transitively), but fails to pass the -thread switch from

ocamlfind: Error from package `threads': Missing -thread or -vmthread switch

to

Error: No implementations provided for the following modules:
         Thread referenced from (...)/library.cmxa(Uses_threads)

But it also allows a project that does not use threads at all (does not pull in a module that uses them), but does depend on a library that depends on threads, to compile.

nocrypto also builds with Lwt 3.2.0 without these lines.

I'd say these lines are too restrictive, and we should remove them.

[aantron]

To summarize more clearly, I have a toy library that unconditionally depends on threads, but consists of two modules, one which doesn't actually use Thread, and one which does. It is built using an unmodified Jbuilder.

I have two toy ocamlfind projects, one which uses only the part of the library that does not use Thread, and another which does use the part of the library that uses Thread.

These two projects are tested in two cases each, with the threads META error message lines, and without them.

Here is the output in all four cases:

1. Stock threads META (with errors), project actually using Thread transitively:

   ocamlfind: Error from package `threads': Missing -thread or -vmthread switch
   

2. Stock threads META (with errors), project not using Thread at all:

   ocamlfind: Error from package `threads': Missing -thread or -vmthread switch
   

3. threads META without errors, project actually using Thread transitively:

   Error: No implementations provided for the following modules:
        Thread referenced from (...)/library.cmxa(Uses_threads)
   

4. threads META without errors, project not using Thread at all:

   (compiled successfully)
   

And I claim we want to go from the behavior in 1, 2 to the one in 3, 4.

[aantron]

I looked through all the logs. All the builds failing due to Lwt are failing due to the above problem. I tried installing some of the packages locally, and all would be fixed by deleting those error lines in threads/META.

[aantron]

Opened https://gitlab.camlcity.org/gerd/lib-findlib/merge_requests/9 against Findlib. Not sure if it's reasonable to block Lwt 3.2.0 on a release of Findlib, though.

[ghost]

We can probably stop putting the transitive closure in META files generated by jbuilder. It's likely we'll need to do it for ocaml/dune#136 anyway

[aantron]

Removing the transitive closure would allow us to work around the problem by having a custom META file in Lwt, so that's good. Removing the error lines seems most correct (see linked discussion at lib-findlib), and it would fix everything without requiring any workarounds.

I guess releasing either Jbuilder without transitive closure, or Findlib without error lines, can unblock Lwt 3.2.0.

We could also unblock in Lwt, by either including a custom threads Findlib package in Lwt's META until Findlib is released (argh), or just reverting the lwt.preemptive change. I'd rather not do the last one.

Not sure what's the least painful here :p

[hannesm]

@aantron FWIW my point of view: I'm a happy lwt and lwt.unix user, but usually use OCaml compilers without threading support. I'd be happy to keep the preemptive module separate (and not as part of lwt.unix). But if your changes / fixes require preemptive now being part of lwt.unix, then move along and we can fix up all the (broken) reverse dependencies. Since from I read here, you plan to do this change for 4.0.0 anyways, there's imho not much gained to delay the breaking change (we'll need to fix the libraries anyways).

[aantron]

@hannesm Actually this is useful to know, as it should stop us from merging lwt.preemptive into lwt.unix. Is lwt.unix being configured not to use system threads (directly) during your builds as well?

[ghost]

BTW, even if we change jbuilder and/or findlib, is it going to work for people who are currently linking against lwt.unix without -thread and with -linkall?

[hannesm]

@aantron it turns out while my system switch uses an OCaml without multithreading, installation of lwt there fails (tested 2.7.x and 3.x). please consider my comment as void, lwt already requires (for installation via opam) an OCaml with multithreading.

[aantron]

@diml If we change the threads META, indeed someone linking with lwt.unix and -linkall, but not -thread, will still have a problem. If we change Jbuilder, then it depends on what we do exactly. If we add the (mt) predicates and either remove transitive closure, or make Jbuilder respect predicates when computing it, then we shouldn't have a problem anymore.

However, I suspect -linkall is pretty rare. To check, I listed all the packages that had broken builds in the CI log: bencode_rpc.0.2.1 capnp.3.1.0 cohttp.0.22.0 cstruct.1.2.0 dht.0.1.3 hardcaml-waveterm.0.2.0 hiredis.0.4 i3ipc.0.1.2 iocaml-kernel.0.4.8 kinetic-client.0.0.6 links.0.7.1 lwt-binio.0.2.1 lwt-parallel.0.1.2 lwt-zmq.2.1.0 markup.0.7.5 mirage-block-unix.2.4.0 mirage-unix.2.6.0 mirage-unix.3.0.4 mqtt.0.0.2 named-pipe.0.4.0 nbd.3.0.0 nocrypto.0.5.4 nproc.0.5.1 ocurl.0.8.0 qfs.0.6 rawlink.0.7 redis.0.3.3 shared-block-ring.2.4.0 skkserv-lite.2.0.1 socket-daemon.0.3.0 statsd-client.1.0.1 zbar.0.9

I removed the error lines from the threads META. Almost all of these packages then installed locally on my system, so they don't use -linkall. qfs.0.6 and zbar.0.9 didn't install for other reasons, but I looked in their repos, and there is no mention of -linkall.

If we can use opam as a proxy for the whole OCaml world, I'd say we don't really have to worry about -linkall. Anyone maintaining a private package should be able to add -thread.

@hannesm Thanks for checking that. Can you say what the error was on installation? I'm not aware of a non-threads system I have easy access to. I guess I could build a custom compiler..

We still have three months before fully committing to this, so please let us know if you become aware of some system where Lwt is used, but which does not have compiler (or system) threads.

[hannesm]

@aantron:

lwt 3.1.0, stderr:

File "src/unix/jbuild", line 33, characters 0-81:
Warning: File lwt_config is both generated by a rule and present in the source tree.
As a result, the rule is currently ignored, however this will become an error in the future.
To keep the current behavior and get rid of this warning, add a field (fallback) to the rule.
Error: External library "threads" is unavailable.
-> required by "src/preemptive/jbuild (context default)"
External library "threads" is not available because it depends on the following libraries that are not available:
- threads.posix -> hidden (unsatisfied 'exist_if')
Hint: try: jbuilder external-lib-deps --missing -p lwt @install

lwt.3.0.0 (and below), stderr:

W: Cannot find source file matching module 'Lwt_unix' in library lwt-unix.
W: Use InterfacePatterns or ImplementationPatterns to define this file with feature "source_patterns".
W: Cannot find source file matching module 'Lwt_unix' in library lwt-unix.
W: Use InterfacePatterns or ImplementationPatterns to define this file with feature "source_patterns".
E: Failure("Command ''/usr/local/bin/ocamlbuild' src/core/lwt.cma src/core/lwt.cmxa src/core/lwt.a src/core/lwt.cmxs src/logger/lwt-log.cma src/logger/lwt-log.cmxa src/logger/lwt-log.a src/logger/lwt-log.cmxs src/unix/liblwt-unix_stubs.a src/unix/dlllwt-unix_stubs.so src/unix/lwt-unix.cma src/unix/lwt-unix.cmxa src/unix/lwt-unix.a src/unix/lwt-unix.cmxs src/simple_top/lwt-simple-top.cma src/simple_top/lwt-simple-top.cmxa src/simple_top/lwt-simple-top.a src/simple_top/lwt-simple-top.cmxs src/preemptive/lwt-preemptive.cma src/preemptive/lwt-preemptive.cmxa src/preemptive/lwt-preemptive.a src/preemptive/lwt-preemptive.cmxs src/ppx/ppx.cma src/ppx/ppx.cmxa src/ppx/ppx.a src/ppx/ppx.cmxs src/ppx/ppx_lwt_ex.native doc/examples/unix/logging.native doc/examples/unix/relay.native doc/examples/unix/parallelize.native -use-ocamlfind -plugin-tags 'package(cppo_ocamlbuild)' -tag debug' terminated with error code 10")
gmake: *** [Makefile:33: build] Error 1

stdout:

+ /home/hannes/.opam/system/bin/ocamlfind ocamlopt -c -g -annot -bin-annot -safe-string -I src/core -I src/logger -I src/unix -thread -package unix -package result -package bytes -package bigarray -package threads -w +A-29-58 -I src/preemptive -I src/core -I src/unix -o src/preemptive/lwt_preemptive.cmx src/preemptive/lwt_preemptive.ml
ocamlfind: ocamlopt does not support multi-threaded programs for your configuration
Command exited with code 2.

[talex5]

Use of the following functions is discouraged, but they have not yet received deprecation warnings: Lwt.with_value, Lwt.cancel, Lwt.state, Lwt.ignore_result (ocsigen/lwt#359, ocsigen/lwt#469).

Why is Lwt.state discouraged? I couldn't see it mentioned in the linked issues.

[aantron]

@hannesm yeah, Lwt 3.1.0 accidentally made building lwt.preemptive unconditional while porting to Jbuilder. Since nobody complained, this partially informed merging it into lwt.unix. I'm not sure what was broken about the non-threaded build before then, but it is again "co-"encouraging, since nobody complained the whole time either.

In any case, in the future, if we, for instance, switch to libuv, all of this will be obsolete, and I am not yet sure if we will be able to support non-threaded builds at all even in principle. Please let me know if non-threaded builds are important on some platform where I/others working on Lwt have blind spots. It will help to make wiser decisions :)

@talex5 Synchronous state querying is deliberately not supported by JavaScript promises. It's actually quite a pain not to have it. I'm not sure we would truly deprecate it in Lwt, but it seems reasonable to encourage people not to use it if they don't really need it.

[hannesm]

@aantron I guess it is fine to require a multi-threaded OCaml runtime, and only had the illusion that I used one without threads ;)

[aantron]

Ok, I added an opam patch to this PR to work around the error lines in the threads package, so that Lwt 3.2.0 doesn't have to wait for a Findlib release.

What the patch does is inline the threads package, without error lines, into Lwt's META. So, with this patch, Lwt 3.2.0 has a package lwt.threads-workaround, and lwt.unix internally depends on this instead of on threads.

I believe the content of the lwt.threads-workaround package is correct on any system. For reference, Findlib's own threads package is mostly unconditional text. The two values generated programmatically, type_of_threads and browse_interfaces, don't matter. type_of_threads is always posix in practice, and browse_interfaces seems to be a feature used internally by Findlib that was not fully developed.

Using the lwt.threads-workaround package allowed nocrypto to be installed locally. Let's see what happens with the revdeps builds in CI.

[aantron]

Adding lwt.threads-workaround to Lwt fixed some builds, but at least camltc, devkit, irmin, hvsock, iocaml-kernel started breaking with messages like

- Error: Files /home/opam/.opam/4.05.0/lib/ocaml/threads/threads.cmxa
-        and /home/opam/.opam/4.05.0/lib/ocaml/threads/threads.cmxa
-        both define a module named Thread

due, presumably, to threads.cm[x]a now being pulled in by two different Findlib packages, threads and lwt.threads-workaround. Some of those packages seem pretty important, so I removed lwt.threads-wokaround and replaced it by patching ocamlfind.1.7.3's threads package directly, using the same patch as in the Findlib merge request. For opinions, cc @dbuenzli (due to discussion at Findlib), @samoht (listed as ocamlfind maintainer).

Now waiting for the CI.

[aantron]

Ok, none of the remaining revdeps failures are due to Lwt 3.2.0. statsd-client must have been broken by an earlier version of Lwt (edit: actually, I think it was OCaml 4.05). The rest, that are broken, are broken for reasons other than Lwt.

The Ubuntu failure seems to be spurious, perhaps a network problem.

I guess merging this PR in its current form depends on whether the ocamlfind patch is reasonable. I think that it is. Opinions welcome.

[hannesm]

@aantron thanks for working on this! to avoid potential issues, I propose to bump the version number of ocamlfind (to 1.7.3-1!?) and include a conflicts: [ "ocamlfind" {< "1.7.3-1"} ] into the lwt.opam. does this sound reasonable?

[aantron]

@hannesm It sounds reasonable. I think we should change the ocamlfind constraint in lwt.opam though, since we already have a dependency on ocamlfind. I don't know if we need opinions from more people, otherwise I'm ready to edit this PR along the lines you propose.

[hannesm]

@aantron oh yes, adjusting the constraint in the dependency is perfectly fine. i somehow conjectured since lwt uses jbuilder, there won't be any ocamlfind dependency...

[aantron]

I just replaced the last commit, which added the patch to ocamlfind.1.7.3, by a commit that adds an ocamlfind.1.7.3-1 with the patch, and leaves ocamlfind.1.7.3 alone. Lwt now requires ocamlfind.1.7.3-1. If this is accepted, I'll also update the opam file in the Lwt repo along the same lines.

Thanks for the suggestion, it's definitely important to add a constraint on ocamlfind either way.

[aantron]

All seems good in CI.

[avsm]

Thanks! You may want to announce this on https://discuss.ocaml.org, where we have a Community category and an announce tag for this purpose.