Improved handling of iteration in the compiler.

This commit improves the handling of iteration in a couple of ways.

- First, it enables pattern matching in `for` loops and `FlatMap` (vmware-archive#784).  One
  can now write:
    ```
    for ((k,v) in map) {}
    ```
    instead of
    ```
    for ((kv in map) { var k = kv.0}
    ```
    and
    ```
    (var x, var y) = FlatMap(expr)
    ```
    instead of:
    ```
    var xy = FlatMap(expr),
    var x = xy.0
    ```
  In general, the pattern can contain constructor names, tuples,
  placeholders, and variable declarations.  The pattern must be
  irrefutable, i.e., any constructor name used in the pattern must be
  the unique constructor for the given type.

- Second, we remove the hardwired knowledge about iterable types from
  the compiler.  Until now the compiler only knew how to iterate over
  `Vec`, `Set`, `Map`, `Group`, and `TinySet` types.  Instead we now
  allow the programmer to label any extern type as iterable, meaning
  that it implements `iter()` and `into_iter()` methods, that return
  Rust iterators using one of two attributes:
  ```
  #[iterate_by_ref=iter:<type>]
  ```
  or
  ```
  #[iterate_by_val=iter:<type>]
  ```
  where `type` is the type yielded by the `next()` method of the iterator
  The former indicates that the `iter()` method returns a by-reference
  iterator, the latter indicates that `iter()` returns a by-value
  iterator.

  Example:
  ```
  #[iterate_by_val=iter:('K,'V)]
  extern type Map<'K,'V>
  ```

  This feature will enable us to introduce new container types, e.g.,
  immutable sets and maps in the future.

  Unfortunately, it introduces one, hopefully minor, usability
  regression.  Since we can no longer distinguish a map from any other
  container whose iterator is a 2-tuple, we cannot reliably convert
  DDlog maps to/from Java maps in the FlatBuf-based API.  They must
  therefore be passed around as arrays of key-value tuples and converted
  to/from Java maps by the client.

Future todos:
- Support pattern matching in closure arguments and the LHS of `group_by` clauses.
- Should refutable patterns be supported for `FlatMap`?  This would have
  the effect of filtering the flattened collection.

Author: ryzhyk

CHANGELOG.md

@@ -3,6 +3,50 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [Unreleased]
+
+### Language improvements
+
+- Enable pattern matching in `for` loops and `FlatMap`. One can now write:
+    ```
+    for ((k,v) in map) {}
+    ```
+    instead of
+    ```
+    for ((kv in map) { var k = kv.0}
+    ```
+    and
+    ```
+    (var x, var y) = FlatMap(expr)
+    ```
+    instead of:
+    ```
+    var xy = FlatMap(expr),
+    var x = xy.0
+    ```
+
+- Remove the hardwired knowledge about iterable types from the compiler.
+  Until now the compiler only knew how to iterate over `Vec`, `Set`,
+  `Map`, `Group`, and `TinySet` types.  Instead we now allow the programmer
+  to label any extern type as iterable, meaning that it implements `iter()`
+  and `into_iter()` methods, that return Rust iterators using one of two
+  attributes:
+  ```
+  #[iterate_by_ref=iter:<type>]
+  ```
+  or
+  ```
+  #[iterate_by_val=iter:<type>]
+  ```
+  where `type` is the type yielded by the `next()` method of the iterator
+  The former indicates that the `iter()` method returns a by-reference
+  iterator, the latter indicates that `iter()` returns a by-value
+  iterator.
+
+  As a side effect of this change, the compiler no longer distinguishes
+  maps from other cotnainers that over 2-tuples.  Therefore maps are
+  represented as lists of tuples in the Flatbuf-based Java API.
+
 ## [0.36.0] - Feb 7, 2021
 
 ### API changes

doc/java_api.md

@@ -422,5 +422,5 @@ the same table.
 | Struct: `S<t1,..,tN>`  | `S__t1..__tnWriter`                  | `S__t1..__tnReader`      |
 | `Vec<T>`               | `List<T_w>`                          | `List<T_r>`              |
 | `Set<T>`               | `List<T_w>`                          | `List<T_r>`              |
-| `Map<K,V>`             | `Map<K_w,V_w>`                       | `Map<K_r,V_r>`           |
+| `Map<K,V>`             | `List<Tuple2__K_w__V_w>`             | `Map<Tuple2__K_r__V_r>`  |
 | `Ref<T>`               | `T_w`                                | `T_r`                    |

doc/language_reference/language_reference.md

@@ -497,7 +497,7 @@ cons_term    ::= (* positional arguments *)
                             ("," "." field_name "=" expr)* "}"]
 var_term     ::= var_name
 ite_term     ::= "if" term term [ "else" term ]
-for_term     ::= "for" "(" var_name "in" expr ")" term
+for_term     ::= "for" "(" for_pattern "in" expr ")" term
 return_term  ::= "return" [expr]
 vardecl_term ::= "var" var_name
 
@@ -526,6 +526,20 @@ pattern ::= (* tuple pattern *)
           | "_"             (* wildcard, matches any value *)
 ```
 
+```EBNF
+(* pattern that binds loop variables in a for-loop *)
+for_pattern ::= (* tuple pattern *)
+                "(" [pattern (,pattern)* ")"
+                (* constructor pattern with positional arguments. The constructor must be the unique constructor for the type.*)
+              | cons_name ["{" [pattern (,pattern)*] "}"]
+                (* constructor pattern with named arguments. The constructor must be the unique constructor for the type. *)
+              | cons_name "{" ["." field_name "=" pattern
+                               ("," "." field_name "=" pattern)*] "}"
+              | vardecl_term    (* binds variable to a field inside the matched value or the entire value. *)
+              | var_term        (* binds variable to a field inside the matched value (shorthand for vardecl_term) *)
+              | "_"             (* wildcard, matches any value *)
+```
+
 ### Constraints on expressions
 
 1. Number and types of arguments to a function must match function
@@ -647,9 +661,20 @@ rhs_clause ::= atom                                      (* 1.atom *)
              | "not" atom                                (* 2.negated atom *)
              | expr                                      (* 3.condition *)
              | expr "=" expr                             (* 4.assignment *)
-             | "var" var_name "=" "FlatMap" "(" expr ")" (* 5.flat map *)
+             | flatmap_pattern "=" "FlatMap" "(" expr ")" (* 5.flat map *)
              | "var" var_name = expr "." "group_by"      (* 6.grouping; in general a *)
                                 "(" expr ")"             (*   group_by clause can be any expression containing `expr.group_by(expr)` subexpression. *)
+
+(* pattern that binds variables in the left-hand side of a FlatMap clause *)
+flatmap_pattern ::= (* tuple pattern *)
+                    "(" [pattern (,pattern)* ")"
+                    (* constructor pattern with positional arguments.  The constructor must be the unique constructor for a type. *)
+                  | cons_name ["{" [pattern (,pattern)*] "}"]
+                    (* constructor pattern with named arguments. The constructor must be the unique constructor for the type. *)
+                  | cons_name "{" ["." field_name "=" pattern
+                                  ("," "." field_name "=" pattern)*] "}"
+                  | vardecl_term    (* binds variable to a field inside the matched value or the entire value. *)
+                  | "_"             (* wildcard, matches any value *)
 ```
 
 An atom is a predicate that holds when a given value belongs to a relation.

doc/tutorial/tutorial.md

@@ -1481,11 +1481,11 @@ containing item name, vendor, and price:
 function best_vendor_string(g: Group<string, (string, bit<64>)>): string
 {
     var min_vendor = "";
-    var min_price: bit<64> = 'hffffffffffffffff;
-    for (vendor_price in g) {
-        if (vendor_price.1 < min_price) {
-            min_vendor = vendor_price.0;
-            min_price = vendor_price.1
+    var min_price = 'hffffffffffffffff;
+    for ((vendor, price) in g) {
+        if (price < min_price) {
+            min_vendor = vendor;
+            min_price = price;
         }
     };
     "Best deal for ${group_key(g)}: ${min_vendor}, $${min_price}"
@@ -3060,6 +3060,38 @@ function returns a reference in Rust:
 extern function deref(x: Ref<'A>): 'A
 ```
 
+### `#[iterate_by_ref]` and `#[iterate_by_val]`
+
+These attributes apply to extern types only and tell the compiler that the given
+type is **iterable**.  Iterable types can be iterated over in a for-loop and
+flattened by `FlatMap`.  The syntax for these attributes is:
+
+```
+#[iterate_by_ref=iter:<type>]
+```
+
+```
+#[iterate_by_val=iter:<type>]
+```
+
+Both variants tell the DDlog compiler that the Rust type that the extern type
+declaration binds to implements the `iter()` method, which returns a type that
+implements the Rust `Iterator` trait, and that the `next()` method of the
+resulting iterator returns values of type `<type>`.  The first form indicates
+that the iterator returns the contents of the collection by reference, while
+the second form indicates that the iterator returns elements in the collection
+by value.
+
+Here are two examples from `ddlog_std.dl`:
+
+```
+#[iterate_by_ref=iter:'A]
+extern type Vec<'A>
+
+#[iterate_by_val=iter:('K,'V)]
+extern type Map<'K,'V>
+```
+
 ### `#[deserialize_from_array=func()]`
 
 This attribute is used in conjunction with `json.dl` library (and potentially

java/test_flatbuf1/Test.java

@@ -57,11 +57,11 @@ private String printNI(NIReader v) {
     }
 
     private String printQI(QIReader v) {
-        Map<Long,String> vs = v.m();
+        List<Tuple2__bit_32___stringReader> vs = v.m();
         ArrayList<String> vs_strs = new ArrayList<String>(vs.size());
-        for (Map.Entry<Long,String> entry : vs.entrySet()) {
-            vs_strs.add(entry.getKey() + "=>\"" + entry.getValue() + "\"");
-        }
+        vs.forEach((t) -> {
+            vs_strs.add(t.a0() + "=>\"" + t.a1() + "\"");
+        });
         return "QI{{" + String.join(", ", vs_strs) + "}}";
     }
 
@@ -117,19 +117,15 @@ private String printOptString(Object v) {
         }
     }
 
-    private String printZI5(Map<String, ManyReader> v) {
-        ArrayList<String> strings = new ArrayList<String>(v.size());
-        for (Map.Entry<String, ManyReader> e: v.entrySet()) {
-            strings.add("\"" + e.getKey() + "\"=>" + printMany(e.getValue()));
-        }
+    private String printZI5(List<Tuple2__string__ManyReader> vs) {
+        ArrayList<String> strings = new ArrayList<String>(vs.size());
+        vs.forEach((t) -> strings.add("\"" + t.a0() + "\"=>" + printMany(t.a1())));
         return "{" + String.join(", ", strings) + "}";
     }
 
-    private String printZI11(Map<ManyReader, String> v) {
-        ArrayList<String> strings = new ArrayList<String>(v.size());
-        for (Map.Entry<ManyReader, String> e: v.entrySet()) {
-            strings.add(printMany(e.getKey()) + "=>\"" + e.getValue() + "\"");
-        }
+    private String printZI11(List<Tuple2__Many__stringReader> vs) {
+        ArrayList<String> strings = new ArrayList<String>(vs.size());
+        vs.forEach((t) -> strings.add(printMany(t.a0()) + "=>\"" + t.a1() + "\""));
         return "{" + String.join(", ", strings) + "}";
     }
 
@@ -427,7 +423,7 @@ void onFBCommit(DDlogCommand<Object> command) throws IOException {
             }
             // output relation OZI5[Map<string, Many>]
             case typesTestRelation.OZI5: {
-                fb_file.println("From " + relid + " " + command.kind() + " " + printZI5((Map<String, ManyReader>)command.value()));
+                fb_file.println("From " + relid + " " + command.kind() + " " + printZI5((List<Tuple2__string__ManyReader>)command.value()));
                 break;
             }
             // output relation OZI6[Option<string>]
@@ -461,7 +457,7 @@ void onFBCommit(DDlogCommand<Object> command) throws IOException {
             }
             // output relation OZI11[Map<Many, string>]
             case typesTestRelation.OZI11: {
-                fb_file.println("From " + relid + " " + command.kind() + " " + printZI11((Map<ManyReader,String>)command.value()));
+                fb_file.println("From " + relid + " " + command.kind() + " " + printZI11((List<Tuple2__Many__stringReader>)command.value()));
                 break;
             }
             // output relation OZI12[(string, bigint, Vec<bigint>, (bit<16>, Many))]
@@ -612,9 +608,9 @@ void update() throws DDlogException {
             builder.insert_PI5(pi);
         }
         {
-            Map<Long, String> map = new HashMap<Long, String>();
-            map.put(Long.valueOf(2), "here");
-            map.put(Long.valueOf(3), "there");
+            ArrayList<Tuple2__bit_32___stringWriter> map = new ArrayList<Tuple2__bit_32___stringWriter>();
+            map.add(builder.create_Tuple2__bit_32___string(2, "here"));
+            map.add(builder.create_Tuple2__bit_32___string(3, "there"));
             builder.insert_QI(map);
         }
         builder.insert_RI(2);
@@ -677,9 +673,9 @@ void update() throws DDlogException {
             builder.insert_ZI4(strings);
         }
         {
-            Map<String, ManyWriter> map = new HashMap<String, ManyWriter>();
-            map.put("key1", builder.create_B(false));
-            map.put("key2", builder.create_A("val2"));
+            ArrayList<Tuple2__string__ManyWriter> map = new ArrayList<Tuple2__string__ManyWriter>();
+            map.add(builder.create_Tuple2__string__Many("key1", builder.create_B(false)));
+            map.add(builder.create_Tuple2__string__Many("key2", builder.create_A("val2")));
             builder.insert_ZI5(map);
         }
         builder.insert_ZI6_ddlog_std_Some("ZI6");
@@ -690,11 +686,11 @@ void update() throws DDlogException {
         builder.insert_ZI9(100);
         builder.insert_ZI10("Ref<IString>");
         {
-            Map<ManyWriter, String> map = new HashMap<ManyWriter, String>();
+            ArrayList<Tuple2__Many__stringWriter> map = new ArrayList<Tuple2__Many__stringWriter>();
             //map.put(builder.create_B(false), "v1");
             // Cannot add more than one record, since the map is printed in
             // non-deterministic order, so the diff fails.
-            map.put(builder.create_A("val2"), "v2");
+            map.add(builder.create_Tuple2__Many__string(builder.create_A("val2"), "v2"));
             builder.insert_ZI11(map);
         }
         {
@@ -1103,12 +1099,16 @@ void run() throws IOException, DDlogException {
 
         query_file.println("Query QI_by_m[(2=>\"here\", 3=>\"there\"]:");
         {
-            Map<Long, String> map = new HashMap<Long, String>();
-            map.put(Long.valueOf(2), "here");
-            map.put(Long.valueOf(3), "there");
-            typesTestQuery.queryQI_by_m(this.api, map, v -> {
-                query_file.println(printQI(v));
-            });
+            typesTestQuery.queryQI_by_m(this.api,
+                bldr -> {
+                    ArrayList<Tuple2__bit_32___stringWriter> map = new ArrayList<Tuple2__bit_32___stringWriter>();
+                    map.add(bldr.create_Tuple2__bit_32___string(Long.valueOf(2), "here"));
+                    map.add(bldr.create_Tuple2__bit_32___string(Long.valueOf(3), "there"));
+                    return map;
+                },
+                v -> {
+                    query_file.println(printQI(v));
+                });
         }
 
         query_file.println("Query RI_by_refm[2]:");
@@ -1274,9 +1274,9 @@ void run() throws IOException, DDlogException {
         query_file.println("Query ZI5_by_self[...]:");
         typesTestQuery.queryZI5_by_self(this.api,
                 bldr -> {
-                    Map<String, ManyWriter> map = new HashMap<String, ManyWriter>();
-                    map.put("key1", bldr.create_B(false));
-                    map.put("key2", bldr.create_A("val2"));
+                    ArrayList<Tuple2__string__ManyWriter> map = new ArrayList<Tuple2__string__ManyWriter>();
+                    map.add(bldr.create_Tuple2__string__Many("key1", bldr.create_B(false)));
+                    map.add(bldr.create_Tuple2__string__Many("key2", bldr.create_A("val2")));
                     return map;
                 },
                 v -> { query_file.println(printZI5(v)); });
@@ -1322,8 +1322,8 @@ void run() throws IOException, DDlogException {
         query_file.println("Query ZI11_by_self[\"val2\"=>\"v2\"]:");
         typesTestQuery.queryZI11_by_self(this.api,
                 bldr -> {
-                    Map<ManyWriter, String> map = new HashMap<ManyWriter, String>();
-                    map.put(bldr.create_A("val2"), "v2");
+                    ArrayList<Tuple2__Many__stringWriter> map = new ArrayList<Tuple2__Many__stringWriter>();
+                    map.add(bldr.create_Tuple2__Many__string(bldr.create_A("val2"), "v2"));
                     return map;
                 },
                 v -> { query_file.println(printZI11(v)); });

lib/ddlog_std.dl

@@ -435,6 +435,7 @@ extern function hash128(x: 'X): bit<128>
  * represents a non-empty list of objects.
  * `'K` is the type of group key, and `'V` is the type of value in the group.
  */
+#[iterate_by_val=iter:'V]
 extern type Group<'K,'V>
 
 /* Extracts group key.
@@ -559,6 +560,7 @@ function max(g: Group<'K, 'V>): 'V {
  * Vec
  */
 #[dyn_alloc]
+#[iterate_by_ref=iter:'A]
 extern type Vec<'A>
 
 extern function vec_empty(): Vec<'A>
@@ -669,6 +671,7 @@ function update_nth(v: mut Vec<'X>, idx: usize, value: 'X): bool {
  */
 
 #[dyn_alloc]
+#[iterate_by_val=iter:('K,'V)]
 extern type Map<'K,'V>
 
 extern function map_empty(): Map<'K, 'V>
@@ -716,6 +719,7 @@ function keys(m: Map<'K, 'V>): Vec<'K> {
  */
 
 #[dyn_alloc]
+#[iterate_by_ref=iter:'A]
 extern type Set<'A>
 
 extern function set_singleton(x: 'X): Set<'X>

lib/tinyset.dl

@@ -5,6 +5,7 @@
  * copying (e.g., 8, 16, 32, or 64-bit integers). */
 
 #[dyn_alloc]
+#[iterate_by_val=iter:'X]
 extern type Set64<'X>
 
 extern function size(s: Set64<'X>): bit<64>