Add support for explicit sparse lists and maps

The vast majority of all lists and maps in use today are dense, meaning
they cannot contain null values. However, we have historically had no
way to indicate that a list or map (value) supports nulls, so we had to
assume that all lists and maps are sparse. This change makes it so that
all lists and maps are considered dense by default, but services can
opt-in to sparse lists using the `sparse` trait. This matters because it
now allows languages that bake "null" into their type systems to provide
better generated types.

Given that nullability is now more abstract than just the box trait, I
think that deprecating BoxIndex in favor of NullableIndex makes the
concept more clear. BoxIndex still exists and can be used, but extends
from NullableIndex.

docs/source/1.0/guides/evolving-models.rst

@@ -87,7 +87,16 @@ Boxed shapes
 ============
 
 The :ref:`box-trait` is used to influence code generation in various
-programming languages. It is a backward-incompatible change for the boxed
+programming languages. It is a backward-incompatible change for the ``box``
+trait to be added or removed from a shape because it will affect types
+generated by tooling that uses Smithy models.
+
+
+Sparse lists and maps
+=====================
+
+The :ref:`sparse-trait` is used to influence code generation in various
+programming languages. It is a backward-incompatible change for the ``sparse``
 trait to be added or removed from a shape because it will affect types
 generated by tooling that uses Smithy models.
 

docs/source/1.0/spec/core/model.rst

@@ -692,7 +692,6 @@ List
 The :dfn:`list` type represents an ordered homogeneous collection of values.
 A list shape requires a single member named ``member``. Lists are defined
 in the IDL using a :ref:`list_statement <idl-list>`.
-
 The following example defines a list with a string member from the
 :ref:`prelude <prelude>`:
 
@@ -720,6 +719,45 @@ The following example defines a list with a string member from the
             }
         }
 
+.. rubric:: List member nullability
+
+Lists are considered *dense* by default, meaning they MAY NOT contain ``null``
+values. A list MAY be made *sparse* by applying the :ref:`sparse-trait`.
+The :ref:`box-trait` is not used to determine if a list is dense or sparse;
+a list with no ``@sparse`` trait is always considered dense. The following
+example defines a sparse list:
+
+.. tabs::
+
+    .. code-tab:: smithy
+
+        @sparse
+        list SparseList {
+            member: String
+        }
+
+    .. code-tab:: json
+
+        {
+            "smithy": "1.0",
+            "shapes": {
+                "smithy.example#SparseList": {
+                    "type": "list",
+                    "member": {
+                        "target": "smithy.api#String"
+                    },
+                    "traits": {
+                        "smithy.api#sparse": {}
+                    }
+                }
+            }
+        }
+
+If a client encounters a ``null`` value when deserializing a dense list
+returned from a service, the client MUST discard the ``null`` value. If a
+service receives a ``null`` value for a dense list from a client, it SHOULD
+reject the request.
+
 .. rubric:: List member shape ID
 
 The shape ID of the member of a list is the list shape ID followed by
@@ -735,7 +773,6 @@ Set
 The :dfn:`set` type represents an unordered collection of unique homogeneous
 values. A set shape requires a single member named ``member``.
 Sets are defined in the IDL using a :ref:`set_statement <idl-set>`.
-
 The following example defines a set of strings:
 
 .. tabs::
@@ -762,12 +799,16 @@ The following example defines a set of strings:
             }
         }
 
-.. rubric:: Providing values for a set
+.. rubric:: Sets MUST NOT contain ``null`` values
 
-The values provided for a set are not permitted to be ``null``. ``null`` set
+The values contained in a set are not permitted to be ``null``. ``null`` set
 values do not provide much, if any, utility, and set implementations across
-programming languages often do not support ``null`` values. If a ``null``
-value is encountered for a set, it MUST be discarded.
+programming languages often do not support ``null`` values.
+
+If a client encounters a ``null`` value when deserializing a set returned
+from a service, the client MUST discard the ``null`` value. If a service
+receives a ``null`` value for a set from a client, it SHOULD reject the
+request.
 
 .. rubric:: Set member shape ID
 
@@ -792,7 +833,6 @@ The :dfn:`map` type represents a map data structure that maps ``string``
 keys to homogeneous values. A map requires a member named ``key``
 that MUST target a ``string`` shape and a member named ``value``.
 Maps are defined in the IDL using a :ref:`map_statement <idl-map>`.
-
 The following example defines a map of strings to integers:
 
 .. tabs::
@@ -823,12 +863,55 @@ The following example defines a map of strings to integers:
             }
         }
 
-.. rubric:: Providing keys for a map
+.. rubric:: Map keys MUST NOT be ``null``
 
 Map keys are not permitted to be ``null``. Not all protocol serialization
 formats have a way to define ``null`` map keys, and map implementations
 across programming languages often do not allow ``null`` keys in maps.
 
+.. rubric:: Map value member nullability
+
+Maps values are considered *dense* by default, meaning they MAY NOT contain
+``null`` values. A map MAY be made *sparse* by applying the
+:ref:`sparse-trait`. The :ref:`box-trait` is not used to determine if a map
+is dense or sparse; a map with no ``@sparse`` trait is always considered
+dense. The following example defines a sparse map:
+
+.. tabs::
+
+    .. code-tab:: smithy
+
+        @sparse
+        map SparseMap {
+            key: String,
+            value: String
+        }
+
+    .. code-tab:: json
+
+        {
+            "smithy": "1.0",
+            "shapes": {
+                "smithy.example#SparseMap": {
+                    "type": "map",
+                    "key": {
+                        "target": "smithy.api#String"
+                    },
+                    "value": {
+                        "target": "smithy.api#String"
+                    },
+                    "traits": {
+                        "smithy.api#sparse": {}
+                    }
+                }
+            }
+        }
+
+If a client encounters a ``null`` map value when deserializing a dense map
+returned from a service, the client MUST discard the ``null`` entry. If a
+service receives a ``null`` map value for a dense map from a client, it
+SHOULD reject the request.
+
 .. rubric:: Map member shape IDs
 
 The shape ID of the ``key`` member of a map is the map shape ID followed by
@@ -905,6 +988,24 @@ by "#", followed by the member name, For example, the shape ID of the "foo"
 member in the above example is ``smithy.example#MyStructure$foo``.
 
 
+.. _default-values:
+
+Default structure member values
+-------------------------------
+
+The values provided for structure members are either always present and set to
+a default value when necessary or *boxed*, meaning a value is optionally present
+with no default value. Members are considered boxed if the member is marked with
+the :ref:`box-trait` or the shape targeted by the member is marked with the box
+trait. Members that target strings, timestamps, and aggregate shapes are always
+considered boxed and have no default values.
+
+- The default value of a ``byte``, ``short``, ``integer``, ``long``,
+  ``float``, and ``double`` shape that is not boxed is zero.
+- The default value of a ``boolean`` shape that is not boxed is ``false``.
+- All other shapes are always considered boxed and have no default value.
+
+
 .. _union:
 
 Union
@@ -958,7 +1059,7 @@ The following example defines a union shape with several members:
             }
         }
 
-.. rubric:: Providing a value to a union
+.. rubric:: Union member nullability
 
 Exactly one member of a union MUST be set to a non-null value. In protocol
 serialization formats that support ``null`` values (for example, JSON), if a
@@ -979,24 +1080,6 @@ by "#", followed by the member name. For example, the shape ID of the "i32"
 member in the above example is ``smithy.example#MyUnion$i32``.
 
 
-.. _default-values:
-
-Default values
-==============
-
-The values provided for :ref:`members <member>` are either always present
-and set to a default value when necessary or *boxed*, meaning a value is
-optionally present with no default value. Members are considered boxed if
-the member is marked with the :ref:`box-trait` or the shape targeted by the
-member is marked with the box trait. Members that target strings, timestamps,
-and aggregate shapes are always considered boxed and have no default values.
-
-- The default value of a ``byte``, ``short``, ``integer``, ``long``,
-  ``float``, and ``double`` shape that is not boxed is zero.
-- The default value of a ``boolean`` shape that is not boxed is ``false``.
-- All other shapes are always considered boxed and have no default value.
-
-
 Recursive shape definitions
 ===========================
 

docs/source/1.0/spec/core/type-refinement-traits.rst

@@ -18,10 +18,10 @@ the type of a shape.
 -------------
 
 Summary
-    Indicates that a shape is boxed. When a :ref:`member <member>` is marked
-    with this trait or the shape targeted by a member is marked with this
-    trait, the member may or may not contain a value, and the member has no
-    :ref:`default value <default-values>`.
+    Indicates that a shape is boxed. When a structure :ref:`member <member>` is
+    marked with this trait or the shape targeted by a structure member is marked
+    with the ``box`` trait, the member may or may not contain a value, and the
+    member has no :ref:`default value <default-values>`.
 
     Boolean, byte, short, integer, long, float, and double shapes are only
     considered boxed if they are marked with the ``box`` trait. All other
@@ -135,4 +135,81 @@ in Java).
             message: String,
         }
 
+
+.. _sparse-trait:
+
+----------------
+``sparse`` trait
+----------------
+
+Summary
+    Indicates that lists and maps MAY contain ``null`` values. The ``sparse``
+    trait has no effect on map keys; map keys are never allowed to be ``null``.
+Trait selector
+    ``:is(list, map)``
+Value type
+    Annotation trait.
+
+The following example defines a :ref:`list <list>` shape that MAY contain
+``null`` values:
+
+.. tabs::
+
+    .. code-tab:: smithy
+
+        @sparse
+        list SparseList {
+            member: String
+        }
+
+    .. code-tab:: json
+
+        {
+            "smithy": "1.0",
+            "shapes": {
+                "smithy.example#SparseList": {
+                    "type": "list",
+                    "member": {
+                        "target": "smithy.api#String",
+                    },
+                    "traits": {
+                        "smithy.api#sparse": {}
+                    }
+                }
+            }
+        }
+
+The following example defines a :ref:`map <map>` shape that MAY contain
+``null`` values:
+
+.. tabs::
+
+    .. code-tab:: smithy
+
+        @sparse
+        map SparseMap {
+            key: String,
+            value: String
+        }
+
+    .. code-tab:: json
+
+        {
+            "smithy": "1.0",
+            "shapes": {
+                "smithy.example#SparseMap": {
+                    "type": "map",
+                    "key": {
+                        "target": "smithy.api#String"
+                    },
+                    "value": {
+                        "target": "smithy.api#String"
+                    },
+                    "traits": {
+                        "smithy.api#sparse": {}
+                    }
+                }
+            }
+        }
+
 .. _Option type: https://doc.rust-lang.org/std/option/enum.Option.html