feat: generalized imports supporting pattern matching and selective field import

Changelog.md

@@ -1,10 +1,22 @@
 # Motoko compiler changelog
 
+## 0.6.21 (2022-01-28)
+
+* motoko (`moc`)
+
+  * Implement a limited form of explicit symbol imports (#3076)
+  * Fix: insert critical overflow checks preventing rare heap corruptions
+    in out-of-memory allocation and stable variable serialization (#3077) 
+  * Implement support for 128-bit Cycles-API (#3042)
+
+* motoko-base
+
+  * 128-bit Cycles-API support (#???)
+
 ## 0.6.20 (2022-01-11)
 
 * motoko
 
-  * Fix: insert critical overflow checks preventing rare heap corruptions in out-of-memory allocation and stable variable serialization (#3077) 
   * Implement support for `heartbeat` system methods (thanks to ninegua) (#2971)
 
 * motoko-base

doc/modules/language-guide/examples/grammar.txt

@@ -298,7 +298,7 @@
     <obj_body>
 
 <imp> ::= 
-    'import' <id>? '='? <text>
+    'import' <pat_nullary> '='? <text>
 
 <prog> ::= 
     <list(<imp>, ';')> <list(<dec>, ';')> 

doc/modules/language-guide/pages/language-manual.adoc

@@ -397,7 +397,7 @@ The syntax of an _import_ `<imp>`  is as follows:
 
 ```bnf
 <imp> ::= imports
-  import <id>? =? <url>
+  import <pat> =? <url>
 
 <url> ::=
   "<filepath>"                      import module from relative <filepath>.mo
@@ -406,7 +406,12 @@ The syntax of an _import_ `<imp>`  is as follows:
   "canister:<name>"                 import external actor by <name>
 ```
 
-An import introduces a resource named '<id>?' referring to a local source module, module from a package of modules, or canister (imported as an actor).
+An import introduces a resource referring to a local source module, module from a package of modules, or canister (imported as an actor).
+The contents of the resource are bound to `<pat>`. 
+
+Though typically a simple identifier, `<id>`, `<pat>` can also be any composite pattern binding selective components of the resource.
+
+The pattern must be irrefutable.
 
 [[syntax-libraries]]
 === Libraries
@@ -1198,24 +1203,24 @@ Installation is (necessarily) asynchronous.
 [[imports]]
 === Imports and Urls
 
-An import `import <id>? =? <url>` declares an optional identifier `<id>?` bound to the contents of the text literal `<url>`. (`<id>?`, when absent, defaults to some fresh identifier `<id>`.)
+An import `import <pat> =? <url>` declares a pattern `<pat>` bound to the contents of the text literal `<url>`.
 
 `<url>` is a text literal that designates some resource: a local library specified with a relative path, a named module from a named package,
 or an external canister, referenced either by numeric canister id or by a named alias, and imported as a {proglang} actor.
 
 In detail, if `<url>` is of the form:
 
-* `"<filepath>"` then `<id>` is bound to the library module defined in file `<filepath>.mo`.
+* `"<filepath>"` then `<pat>` is bound to the library module defined in file `<filepath>.mo`.
   `<filepath>` is interpreted relative to the absolute location of the enclosing file.
   Note the `.mo` extension is implicit and should _not_ be included in `<url>`.
   For example, `import U "lib/Util"` defines `U` to reference the module in local file
   `./lib/Util`.
 
-* `"mo:<package-name>/<path>"` then `<id>` is bound to the library module defined in file `<package-path>/<path>.mo` in directory `<package-path>` referenced by package alias `<package-name>`.
+* `"mo:<package-name>/<path>"` then `<pat>` is bound to the library module defined in file `<package-path>/<path>.mo` in directory `<package-path>` referenced by package alias `<package-name>`.
   The mapping from `<package-name>` to `<package-path>` is determined by a compiler command-line argument `--package <package-name> <package-path>`. For example, `import L "mo:base/List"` defines `L` to reference the `List` library in
    package alias `base`.
 
-* `"ic:<canisterid>"` then `<id>` is bound to a {proglang} actor whose {proglang} type is determined by the canister's IDL interface.
+* `"ic:<canisterid>"` then `<pat>` is bound to a {proglang} actor whose {proglang} type is determined by the canister's IDL interface.
   The IDL interface of canister `<canisterid>` must be found in file `<actorpath>/<canisterid>.did`.
   The compiler assumes that  `<actorpath>` is specified by command line argument `--actor-idl <actorpath>` and that file `<actorpath>/<canisterid>.did` exists.
   For example, `import C "ic:lg264-qjkae"` defines `C` to reference the actor with canister id `lg264-qjkae` and IDL file `lg264-qjkae.did`.
@@ -1229,8 +1234,9 @@ In detail, if `<url>` is of the form:
 The case sensitivity of file references depends on the host operating system so it is recommended not to distinguish resources by filename casing alone.
 
 (Remark: when building multi-canister projects with the {sdk-short-name}, {proglang} programs can typically import canisters by alias (e.g. `import C "canister:counter"`), without specifying low-level canister ids (e.g. `import C "ic:lg264-qjkae"`).
-The SDK tooling takes care of supplying the appropriate command-line arguments to the {proglang} compiler).
+The SDK tooling takes care of supplying the appropriate command-line arguments to the {proglang} compiler.)
 
+(Remark: sensible choices for `<pat>` are identifiers, such as ``Array``, or object patterns like ``{ cons; nil = empty }``, which allow selective importing of individual fields, under original or other names.)
 
 [[decl-field]]
 === Declaration fields

doc/modules/language-guide/pages/modules-and-imports.adoc

@@ -21,6 +21,17 @@ import Result "mo:base/Result";
 
 Notice that the import declaration includes the `+mo:+` prefix to identify the module as a {proglang} module and that the declaration does not include the `+.mo+` file type extension.
 
+Above example uses an identifier pattern to import modules wholesale, but you can also selectively import
+a subset of symbols from a module by resorting to the object pattern syntax:
+
+[source,motoko]
+----
+import { map, find, foldLeft = fold } = "mo:base/Array";
+----
+
+In this example, the functions `+map+` and `+find+` are imported unaltered, while the `+foldLeft+`
+function is renamed to `+fold+`.
+
 == Importing local files
 
 Another common approach to writing programs in {proglang} involves splitting up the source code into different modules.

doc/modules/language-guide/pages/overview.adoc

@@ -69,6 +69,11 @@ Inspirations: Java, JavaScript, C#, Swift, Pony, ML, Haskell
 
 (`import MyLib "src/MyLib"` imports a library from the local file system.)
 
+Specific bindings can be imported from the module using object patterns
+....
+  import { cons; nil } = "mo:base/List";
+....
+
 === Libraries
 
 [source, motoko]

src/docs/extract.ml

@@ -51,16 +51,18 @@ and class_doc = {
 
 let un_prog prog =
   let comp_unit = Mo_def.CompUnit.comp_unit_of_prog true prog in
-  let Syntax.{ imports; body } = comp_unit.it in
+  let open Syntax in
+  let { imports; body } = comp_unit.it in
   let imports =
-    List.map
+    List.concat_map
       (fun i ->
-        let alias, path, _ = i.it in
-        (alias.it, path))
+        match i.it with
+        | { it = VarP alias; _ }, path, _ -> [ (alias.it, path) ]
+        | _ -> []) (* FIXME: explicit imports #3078 *)
       imports
   in
   match body.it with
-  | Syntax.ModuleU (_, decs) -> Ok (imports, decs)
+  | ModuleU (_, decs) -> Ok (imports, decs)
   | _ -> Error "Couldn't find a module expression"
 
 module PosTable = Trivia.PosHashtbl

src/ir_def/ir_utils.ml

@@ -6,4 +6,3 @@ let rec is_irrefutable p = match p.it with
   | AltP (pat1, _) -> is_irrefutable pat1
   | WildP | VarP _ -> true
   | TagP _ | LitP _ | OptP _ -> false
-

src/lib/lib.ml

@@ -195,6 +195,11 @@ struct
     else
       None
 
+  let starts_with prefix s = (* in OCaml 4.13 *)
+    match chop_prefix prefix s with
+    | Some _ -> true
+    | _ -> false
+
   let chop_suffix suffix s =
     let suffix_len = String.length suffix in
     let s_len = String.length s in

src/lib/lib.mli

@@ -154,6 +154,7 @@ sig
   val split : string -> char -> string list
   val breakup : string -> int -> string list
   val find_from_opt : (char -> bool) -> string -> int -> int option
+  val starts_with : string -> string -> bool
   val chop_prefix : string -> string -> string option
   val chop_suffix : string -> string -> string option
   val lightweight_escaped : string -> string

src/lowering/desugar.ml

@@ -858,7 +858,7 @@ let link_declarations imports (cu, flavor) =
 
 
 let transform_import (i : S.import) : import_declaration =
-  let (id, f, ir) = i.it in
+  let (p, f, ir) = i.it in
   let t = i.note in
   assert (t <> T.Pre);
   let rhs = match !ir with
@@ -869,7 +869,7 @@ let transform_import (i : S.import) : import_declaration =
       varE (var (id_of_full_path "@prim") t)
     | S.IDLPath (fp, canister_id) ->
       primE (I.ActorOfIdBlob t) [blobE canister_id]
-  in [ letD (var id.it t) rhs ]
+  in [ letP (pat p) rhs ]
 
 let transform_unit_body (u : S.comp_unit_body) : Ir.comp_unit =
   match u.it with

src/lowering/desugar.mli

@@ -1,7 +1,7 @@
 open Mo_def
 open Ir_def
 
-(* These declaratoins are all internal, either from prelude (@-prefixed)
+(* These declarations are all internal, either from prelude (@-prefixed)
    or to bring libraries into scope *)
 type import_declaration = Ir.dec list
 

src/mo_def/compUnit.ml

@@ -23,15 +23,16 @@ let comp_unit_of_prog as_lib (prog : prog) : comp_unit =
   let open Source in
   let f = prog.note in
 
-  let finish imports u = { it = { imports; body = u }; note = f; at = no_region } in
+  let finish imports u =
+    { it = { imports = List.rev imports; body = u }; note = f; at = no_region } in
   let prog_typ_note = { empty_typ_note with note_typ = Type.unit } in
 
   let rec go imports ds : comp_unit =
     match ds with
     (* imports *)
-    | {it = LetD ({it = VarP n; _}, ({it = ImportE (url, ri); _} as e)); _} :: ds' ->
-      let i : import = { it = (n, url, ri); note = e.note.note_typ; at = e.at } in
-      go (imports @ [i]) ds'
+    | {it = LetD (p, ({it = ImportE (url, ri); _} as e)); _} :: ds' ->
+      let i : import = { it = (p, url, ri); note = e.note.note_typ; at = e.at } in
+      go (i :: imports) ds'
 
     (* terminal expressions *)
     | [{it = ExpD ({it = ObjBlockE ({it = Type.Module; _}, fields); _} as e); _}] when as_lib ->
@@ -88,9 +89,9 @@ let obj_decs obj_sort at note id_opt fields =
 let decs_of_lib (cu : comp_unit) =
   let open Source in
   let { imports; body = cub; _ } = cu.it in
-  let import_decs = List.map (fun { it = (id, fp, ri); at; note}  ->
+  let import_decs = List.map (fun { it = (pat, fp, ri); at; note} ->
     { it = LetD (
-      { it = VarP id; at; note; },
+      pat,
       { it = ImportE (fp, ri);
         at;
         note = { note_typ = note; note_eff = Type.Triv} });

src/mo_def/syntax.ml

@@ -227,7 +227,7 @@ and stab_sig' = (dec list * typ_field list)      (* type declarations & stable a
 (* Compilation units *)
 
 type import = (import', Type.typ) Source.annotated_phrase
-and import' = id * string * resolved_import ref
+and import' = pat * string * resolved_import ref
 
 type comp_unit_body = (comp_unit_body', typ_note) Source.annotated_phrase
 and comp_unit_body' =

src/mo_frontend/parser.mly

@@ -871,9 +871,8 @@ class_body :
 (* Programs *)
 
 imp :
-  | IMPORT xf=id_opt EQ? f=TEXT
-    { let _, x = xf "import" $sloc in
-      let_or_exp true x (ImportE (f, ref Unresolved)) (at $sloc) }
+  | IMPORT p=pat_nullary EQ? f=TEXT
+    { LetD(p, ImportE(f, ref Unresolved) @? at $sloc) @? at $sloc }
 
 start : (* dummy non-terminal to satisfy ErrorReporting.ml, that requires a non-empty parse stack *)
   | (* empty *) { () }

src/mo_frontend/typing.ml

@@ -2274,7 +2274,6 @@ and gather_dec env scope dec : Scope.t =
       {it = VarP id; _},
       ({it = ObjBlockE (obj_sort, dec_fields); at; _} |
        {it = AwaitE { it = AsyncE (_, {it = ObjBlockE ({ it = Type.Actor; _} as obj_sort, dec_fields); at; _}) ; _  }; _ })
-       (* TODO include RecE? *)
     ) ->
     let decs = List.map (fun df -> df.it.dec) dec_fields in
     let open Scope in
@@ -2357,7 +2356,6 @@ and infer_dec_typdecs env dec : Scope.t =
       {it = VarP id; _},
       ( {it = ObjBlockE (obj_sort, dec_fields); at; _} |
         {it = AwaitE { it = AsyncE (_, {it = ObjBlockE ({ it = Type.Actor; _} as obj_sort, dec_fields); at; _}) ; _  }; _ })
-       (* TODO include RecE? *)
     ) ->
     let decs = List.map (fun {it = {vis; dec; _}; _} -> dec) dec_fields in
     let scope = T.Env.find id.it env.objs in
@@ -2446,6 +2444,11 @@ and infer_block_valdecs env decs scope : Scope.t =
     ) (env, scope) decs
   in scope'
 
+and is_import d =
+  match d.it with
+  | LetD (_, {it = ImportE _; _}) -> true
+  | _ -> false
+
 and infer_dec_valdecs env dec : Scope.t =
   match dec.it with
   | ExpD _ ->
@@ -2468,7 +2471,7 @@ and infer_dec_valdecs env dec : Scope.t =
     Scope.{empty with val_env = T.Env.singleton id.it obj_typ}
   | LetD (pat, exp) ->
     let t = infer_exp {env with pre = true} exp in
-    let ve' = check_pat_exhaustive warn env t pat in
+    let ve' = check_pat_exhaustive (if is_import dec then local_error else warn) env t pat in
     Scope.{empty with val_env = ve'}
   | VarD (id, exp) ->
     let t = infer_exp {env with pre = true} exp in
@@ -2533,11 +2536,6 @@ let is_actor_dec d =
     obj_sort.it = T.Actor
   | _ -> false
 
-let is_import d =
-  match d.it with
-  | LetD ({it = VarP n; _}, {it = ImportE _; _}) -> true
-  | _ -> false
-
 let check_actors scope progs : unit Diag.result =
   Diag.with_message_store
     (fun msgs ->

test/fail/empty-module.mo

@@ -0,0 +1 @@
+module {}

test/fail/error-imports.mo

@@ -0,0 +1 @@
+import "depri" = "text-module";

test/fail/nat-module.mo

@@ -0,0 +1 @@
+module { public let why = 42 }

test/fail/not-static.mo

@@ -1 +1 @@
-import "lib/not-static"
+import _ = "lib/not-static"

test/fail/ok/error-imports.tc.ok

@@ -0,0 +1,5 @@
+text-module.mo:0.1: warning [M0142], deprecated syntax: an imported library should be a module or named actor class
+error-imports.mo:1.8-1.15: type error [M0050], literal of type
+  Text
+does not have expected type
+  module {}

test/fail/ok/error-imports.tc.ret.ok

@@ -0,0 +1 @@
+Return code 1

test/fail/ok/refutable-import.tc.ok

@@ -0,0 +1,4 @@
+refutable-import.mo:1.8-1.20: type error [M0145], this pattern of type
+  module {why : Nat}
+does not cover value
+  {why = 0 or 1 or _}

test/fail/ok/refutable-import.tc.ret.ok

@@ -0,0 +1 @@
+Return code 1

test/fail/ok/self-import.tc.ok

@@ -1 +1 @@
-self-import.mo:1.1-1.21: import error [M0003], file self-import.mo must not depend on itself
+self-import.mo:1.1-1.25: import error [M0003], file self-import.mo must not depend on itself

test/fail/ok/type-error-imports.tc.ok

@@ -0,0 +1,2 @@
+type-error-imports.mo:4.8-4.14: type error [M0112], tuple pattern cannot consume expected type
+  module {}

test/fail/ok/type-error-imports.tc.ret.ok

@@ -0,0 +1 @@
+Return code 1

test/fail/refutable-import.mo

@@ -0,0 +1,6 @@
+import { why = 42 } = "nat-module";
+
+//SKIP run
+//SKIP run-ir
+//SKIP run-low
+//SKIP comp

test/fail/self-import.mo

@@ -1 +1 @@
-import "self-import";
+import _ = "self-import";

test/fail/text-module.mo

@@ -0,0 +1 @@
+"depri"

test/fail/type-error-imports.mo

@@ -0,0 +1,4 @@
+import _ "empty-module";
+import {} "empty-module";
+
+import (_, _) = "empty-module";

test/repl/double-import.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
 moc -v -i <<__END__
-import "lib/empty";
-import "lib/empty";
+import _ = "lib/empty";
+import _ = "lib/empty";
 __END__

test/repl/lib/actor-import.mo

@@ -1,2 +1,2 @@
-import "ic:em77e-bvlzu-aq";
-import "canister:alias";
+import _ = "ic:em77e-bvlzu-aq";
+import _ = "canister:alias";

test/repl/lib/nested.mo

@@ -1,4 +1,4 @@
 import A "nested/a";
-import "nested/b";
+import _ = "nested/b";
 
 let i = A.i;

test/repl/lib/triangle.mo

@@ -1 +1 @@
-import "a"
+import _ = "a"

test/repl/nested-import.sh

@@ -1,4 +1,4 @@
 #!/usr/bin/env bash
 moc -v -i <<__END__ | grep Parsing
-import "lib/nested";
+import _ = "lib/nested";
 __END__

test/repl/ok/double-import.stdout.ok

@@ -8,14 +8,12 @@ Motoko compiler (source XXX)
 -- Interpreting empty.mo:
 -- Interpreting stdin:
 -- Finished stdin:
-let anon-import-1.1 : module {...}
 
 -- Parsing stdin:
 > -- Checking stdin:
 -- Definedness stdin:
 -- Interpreting stdin:
 -- Finished stdin:
-let anon-import-2.1 : module {...}
 
 -- Parsing stdin:
 >