opn201-data-optimization-cheatsheet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "082098ff-c2ec-4aef-9c50-d8844f72450b",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Optimize data for analysis: Learnings from AWS Open Data\n",
    "\n",
    "[![Open In Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/pschmied/opn201/blob/main/opn201-data-optimization-cheatsheet.ipynb)\n",
    "\n",
    "(https://github.com/pschmied/opn201)\n",
    "\n",
    "## Contacts / links\n",
    " - Peter Schmiedeskamp - Statistical & Regulatory Lead (pws AT amazon), AWS Open Data\n",
    " - AWS Open Data (opendata AT amazon / https://opendata.aws)\n",
    " - Registry of Open Data (https://registry.opendata.aws)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19b9b218-3e15-4590-8d6b-664526de2f97",
   "metadata": {},
   "source": [
    "## Amazon Simple Storage Service (S3)\n",
    "\n",
    "Three main tuning knobs:\n",
    " 1. Object size\n",
    " 2. Object name\n",
    " 3. Object format (including compression)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47c3cc77-6538-4539-8485-431a32d84090",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Object size\n",
    " - Too small: many requests (i.e. >= 1 per object); implications for listing 🙀\n",
    " \n",
    " ```\n",
    " $ aws s3 ls s3://fictitious-bucket-of-a-billion-1KiB-objects\n",
    " 2022-05-17 18:10:49  1024 obj1.txt\n",
    " 2022-05-17 18:10:50  1024 obj2.txt\n",
    " 2022-05-17 18:10:51  1024 obj3.txt\n",
    " [...]\n",
    " 2022-05-18 19:14:04  1024 obj1000000000.txt\n",
    " ```\n",
    " \n",
    " - Too big: may force customers to download more data than they need 🤔\n",
    " \n",
    " ```\n",
    " $ aws s3 ls s3://fictitious-bucket-of-a-1TiB-object\n",
    " 2022-05-17 18:10:49  1.1e+12 obj1.txt\n",
    " ```\n",
    " \n",
    " - Just right?: It depends 🤷\n",
    " ```\n",
    " $ aws s3 ls s3://fictitious-bucket-of-a-thousand-1GiB-objects\n",
    " 2022-05-17 18:10:49  1.074e+9 obj1.txt\n",
    " 2022-05-17 18:10:50  1.074e+9 obj2.txt\n",
    " 2022-05-17 18:10:51  1.074e+9 obj3.txt\n",
    " [...]\n",
    " 2022-05-18 19:14:04  1.074e+9 obj1000.txt\n",
    " ```\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67e80ca1-aaaf-4689-9ba3-fdce57ad992a",
   "metadata": {},
   "source": [
    " ### Object name\n",
    " - Theory: S3 is a flat namespace; buckets have no concept of directories; '/' is [not special](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html)\n",
    " - Practice: '/' is, by convention, used as a delimiter by many services including Athena\n",
    " \n",
    " Recommend using \"Hive-style partition\" naming conventions [as outlined in Athena documentation](https://docs.aws.amazon.com/athena/latest/ug/partitions.html) (even if you won't use Athena)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3448d2e6-72c0-47bd-885a-e05e225d7984",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.10.3\n",
      "  latest version: 4.12.0\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base conda\n",
      "\n",
      "\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Install the 'tree' utility if not already installed\n",
    "!conda install -y tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b76c6c07-6595-48ef-a112-d0fd5995cf26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34m.\u001b[0m\n",
      "└── \u001b[01;34mshapefiles\u001b[0m\n",
      "    └── \u001b[01;34mperformance\u001b[0m\n",
      "        └── \u001b[01;34mtype=mobile\u001b[0m\n",
      "            └── \u001b[01;34myear=2021\u001b[0m\n",
      "                ├── \u001b[01;34mquarter=1\u001b[0m\n",
      "                │   └── \u001b[01;31m2021-01-01_performance_mobile_tiles.zip\u001b[0m\n",
      "                ├── \u001b[01;34mquarter=2\u001b[0m\n",
      "                │   └── \u001b[01;31m2021-04-01_performance_mobile_tiles.zip\u001b[0m\n",
      "                ├── \u001b[01;34mquarter=3\u001b[0m\n",
      "                │   └── \u001b[01;31m2021-07-01_performance_mobile_tiles.zip\u001b[0m\n",
      "                └── \u001b[01;34mquarter=4\u001b[0m\n",
      "                    └── \u001b[01;31m2021-10-01_performance_mobile_tiles.zip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!aws --no-sign-request s3 ls --recursive \\\n",
    "   s3://ookla-open-data/shapefiles/performance/type=mobile/year=2021/  \\\n",
    "   | awk ' { print $4 } ' \\\n",
    "   | tree --noreport --fromfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5d7f2029-5652-45a6-b82b-15aa0b816eaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34m.\u001b[0m\n",
      "└── \u001b[01;34mABI-L2-RRQPEF\u001b[0m\n",
      "    └── \u001b[01;34m2020\u001b[0m\n",
      "        └── \u001b[01;34m001\u001b[0m\n",
      "            ├── \u001b[01;34m00\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010000216_e20200010009524_c20200010010034.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010010216_e20200010019524_c20200010020035.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010020216_e20200010029524_c20200010030028.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010030216_e20200010039524_c20200010040050.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010040216_e20200010049524_c20200010050040.nc\u001b[0m\n",
      "            │   └── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010050216_e20200010059524_c20200010100027.nc\u001b[0m\n",
      "            ├── \u001b[01;34m01\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010100216_e20200010109524_c20200010110037.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010110216_e20200010119524_c20200010120042.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010120216_e20200010129524_c20200010130033.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010130216_e20200010139524_c20200010140036.nc\u001b[0m\n",
      "            │   ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010140216_e20200010149524_c20200010150030.nc\u001b[0m\n",
      "            │   └── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010150216_e20200010159524_c20200010200023.nc\u001b[0m\n",
      "            └── \u001b[01;34m02\u001b[0m\n",
      "                ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010200216_e20200010209524_c20200010210046.nc\u001b[0m\n",
      "                ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010210216_e20200010219524_c20200010220024.nc\u001b[0m\n",
      "                ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010220216_e20200010229524_c20200010230023.nc\u001b[0m\n",
      "                ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010230216_e20200010239524_c20200010240027.nc\u001b[0m\n",
      "                ├── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010240216_e20200010249524_c20200010250025.nc\u001b[0m\n",
      "                └── \u001b[00mOR_ABI-L2-RRQPEF-M6_G16_s20200010250216_e20200010259524_c20200010300029.nc\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Also works, but may cause users some additional effort\n",
    "!aws --no-sign-request s3 ls --recursive \\\n",
    "   s3://noaa-goes16/ABI-L2-RRQPEF/2020/ 2>/dev/null \\\n",
    "   | awk ' { print $4 } ' 2>/dev/null \\\n",
    "   | head -n 18 \\\n",
    "   | tree --noreport --fromfile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7028fd95-fb64-4775-aa00-d122aceb6b08",
   "metadata": {},
   "source": [
    "### Object format\n",
    "Theory:\n",
    " - Choose formats that support range requests (or [S3 Select... 🤔 or both!? 🤯](https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html)? )\n",
    " - Use compression; don't use archives\n",
    " \n",
    "Practice: gzip / NOT tar, ndjson.gz, csv.gz, COG, Parquet, ORC, Zarr, OH MY! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9635fd5-31bf-4647-87dc-c3488e6e801d",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Tabular data with Athena\n",
    "\n",
    "Athena needs three things: data, schema, (often) index of partitions.\n",
    "\n",
    "Data are stored as objects in S3, mapped to a relational schema via Data Definition Language (DDL). Partitions are represented as dimensions (columns) in the schema, and are either enumerated in AWS Glue catalog (or external Hive metastore), or inferred via Athena Partition Projection. Note Partition Projection is Athena-only, and doesn't apply to e.g. Glue.\n",
    "\n",
    "CSV\n",
    "\n",
    "```sql\n",
    "CREATE EXTERNAL TABLE `sentinel_s1_l1c_inventory_csv`(\n",
    "  `bucket` string COMMENT 'from deserializer', \n",
    "  `key` string COMMENT 'from deserializer', \n",
    "  `size` string COMMENT 'from deserializer', \n",
    "  `last_modified_date` string COMMENT 'from deserializer')\n",
    "PARTITIONED BY ( \n",
    "  `dt` string)\n",
    "ROW FORMAT SERDE \n",
    "  'org.apache.hadoop.hive.serde2.OpenCSVSerde' \n",
    "STORED AS INPUTFORMAT \n",
    "  'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' \n",
    "OUTPUTFORMAT \n",
    "  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'\n",
    "LOCATION\n",
    "  's3://sentinel-inventory/sentinel-s1-l1c/sentinel-s1-l1c-inventory/hive'\n",
    "```\n",
    "\n",
    "ORC\n",
    "\n",
    "```sql\n",
    "CREATE EXTERNAL TABLE `sentinel_s1_l1c_inventory_orc`(\n",
    "  `bucket` string, \n",
    "  `key` string, \n",
    "  `size` bigint, \n",
    "  `last_modified_date` timestamp)\n",
    "PARTITIONED BY ( \n",
    "  `dt` string)\n",
    "ROW FORMAT SERDE \n",
    "  'org.apache.hadoop.hive.ql.io.orc.OrcSerde' \n",
    "STORED AS INPUTFORMAT \n",
    "  'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' \n",
    "OUTPUTFORMAT \n",
    "  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'\n",
    "LOCATION\n",
    "  's3://sentinel-inventory/orc/sentinel-s1-l1c/sentinel-s1-l1c-inventory-orc/hive'\n",
    "```\n",
    "\n",
    "### Query performance ORC vs. CSV\n",
    "ORC\n",
    "```sql\n",
    "SELECT AVG(SIZE) FROM sentinel_s1_l1c_inventory_orc\n",
    "```\n",
    "(Run time: ~24 sec; Data scanned: ~120 MB)\n",
    "\n",
    "vs. CSV\n",
    "\n",
    "```sql\n",
    "SELECT AVG(SIZE) FROM sentinel_s1_l1c_inventory_orc\n",
    "```\n",
    "(Run time: ~10 min; Data scanned: ~165 GB)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62397bbc-25d5-4a63-8f65-9c0b4862358d",
   "metadata": {},
   "source": [
    "Something interesting is happening here... It's not necessarily easy, but Hive-convention partitions can be grafted on after the fact!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b429cf58-d442-44cb-bc8e-eb219d1bbcde",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34m.\u001b[0m\n",
      "└── \u001b[01;34morc\u001b[0m\n",
      "    └── \u001b[01;34msentinel-s1-l1c\u001b[0m\n",
      "        └── \u001b[01;34msentinel-s1-l1c-inventory-orc\u001b[0m\n",
      "            └── \u001b[01;34mhive\u001b[0m\n",
      "                ├── \u001b[01;34mdt=2021-01-26-00-00\u001b[0m\n",
      "                │   └── \u001b[00msymlink.txt\u001b[0m\n",
      "                ├── \u001b[01;34mdt=2021-01-27-00-00\u001b[0m\n",
      "                │   └── \u001b[00msymlink.txt\u001b[0m\n",
      "                ├── \u001b[01;34mdt=2021-01-28-00-00\u001b[0m\n",
      "                │   └── \u001b[00msymlink.txt\u001b[0m\n",
      "                ├── \u001b[01;34mdt=2021-01-29-00-00\u001b[0m\n",
      "                │   └── \u001b[00msymlink.txt\u001b[0m\n",
      "                └── \u001b[01;34mdt=2021-01-30-00-00\u001b[0m\n",
      "                    └── \u001b[00msymlink.txt\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!aws --no-sign-request s3 ls --recursive \\\n",
    "   s3://sentinel-inventory/orc/sentinel-s1-l1c/sentinel-s1-l1c-inventory-orc/hive 2>/dev/null \\\n",
    "   | awk ' { print $4 } ' 2>/dev/null \\\n",
    "   | head -n5 \\\n",
    "   | tree --noreport --fromfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "79fd01ad-dc42-4850-a83e-50e60b4c7fca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sentinel-inventory/orc/sentinel-s1-l1c/sentinel-s1-l1c-inventory-orc/data/661f4db0-398a-454c-b810-7856a9530067.orc\n",
      "s3://sentinel-inventory/orc/sentinel-s1-l1c/sentinel-s1-l1c-inventory-orc/data/5e536e18-735d-4181-ab84-4e4a1f43f973.orc\n",
      "s3://sentinel-inventory/orc/sentinel-s1-l1c/sentinel-s1-l1c-inventory-orc/data/399d7ac9-40d6-4de7-94fc-2eb233d8b6c9.orc\n",
      "s3://sentinel-inventory/orc/sentinel-s1-l1c/sentinel-s1-l1c-inventory-orc/data/7d21fdd0-9431-4212-91a3-b1ac3158550e.orc\n"
     ]
    }
   ],
   "source": [
    "!aws --no-sign-request s3 cp s3://sentinel-inventory/orc/sentinel-s1-l1c/sentinel-s1-l1c-inventory-orc/hive/dt=2021-01-26-00-00/symlink.txt - \\\n",
    "    | head -n4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d49220-cd23-4ce1-bc59-dbf764a441a7",
   "metadata": {},
   "source": [
    "Note: Examples here assume write few / read many. It's not unusual for tabular data to be added (`UPSERT`-ed) to regularly. Check out [Apache Hudi](https://docs.aws.amazon.com/athena/latest/ug/querying-hudi.html) if this sounds like you."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a1d69a7-e580-4651-b0ec-35db68941173",
   "metadata": {},
   "source": [
    "## Specialty formats / beyond tabular - Genomic Data\n",
    "\n",
    "Genomic variant call files (VCFs): https://registry.opendata.aws/ilmn-dragen-1kgp/\n",
    "\n",
    "...enhance! After initial header rows, data are tabular."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ea1f48a3-945f-409c-876c-108d0a3b4af7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chr10\t65272628\tMantaDEL:104013:0:0:0:0:0\tATTTAAGGTGCACAATAGAGAAGTCAGTTGTTAAAAAACAGAGAATGTGGACTTTGGAGATGGTATTGTGTAGTCAAGAGAGCCAACAATGAGTGAGATAATTTTCTGTATTACCTTTGTATCCTTGTACAAGTTTTAATTTCTCTAAGCCTAGCTTTCTCATCTGTAAATCTGGCATAATATTTATTATGATTAAGTGAAACAATGCTGATAAACTCTTTTTGTAGAAGCTGACCATAGAAAGTACTTAAAACAAAACATGTTTTAATATTATTTCCAAATGTCAAATAGCCAAATAAATTCTCCCAGATATTGTGTGACCAAATGAAACATAATGTTAATTGATACATAGATTATGACTATAAAACTTTTGTCAGAATAAGATGAGCTCTTACCCAGGTTAGGTTTACAAGGCA\tAGAACACAGAAAGACCTAAACT\t999\tPASS\tEND=65273043;SVTYPE=DEL;SVLEN=-415;CIGAR=1M21I415D\tGT:FT:GQ:PL:PR:SR\t1/1:PASS:100:999,103,0:0,14:0,28\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --no-sign-request s3://1000genomes-dragen/data/precisionFDA/hg38_altaware_nohla-cnv-anchored/HG004/sv/results/variants/diploidSV.vcf.gz - 2>/dev/null \\\n",
    "    | zcat 2>/dev/null \\\n",
    "    | head -n10000 \\\n",
    "    | tail -n1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ffc83bf-1319-475a-b148-61faab9cce01",
   "metadata": {},
   "source": [
    "This implies that at least some potential for conversion to columnar formats like Parquet and ORC.\n",
    "\n",
    "Side note, Athena is a passable distributed `grep` even if you take a naive approach to text-based data.\n",
    "\n",
    "```sql\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "Recommended reading: [Data Lake as Code](https://aws.amazon.com/blogs/startups/a-data-lake-as-code-featuring-chembl-and-opentargets/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e224ff-aaf9-4264-94ad-5f660a979ad4",
   "metadata": {},
   "source": [
    "## Specialty formats / beyond tabular - Geospatial\n",
    "Vector formats: is tabular; geometries are just a special column type; e.g. Parquet works well / `ST_*` operations are supported in Athena.\n",
    "\n",
    "Example, OSM Daylight in Parquet:\n",
    " - https://registry.opendata.aws/daylight-osm/\n",
    " - https://gist.github.com/jenningsanderson/3e42a99dcb8f760038ad8aa47ea38ce8\n",
    "\n",
    "\n",
    "Raster formats - imagery with geospatial metadata. We are fond of Cloud-Optimized GeoTIFF:\n",
    "\n",
    " - STAC/COG example(s) - https://registry.opendata.aws/deafrica-sentinel-2/\n",
    "     - [Blog Post on Using STAC with S-2](https://docs.digitalearthafrica.org/en/latest/sandbox/notebooks/Frequently_used_code/Downloading_data_with_STAC.html)\n",
    " - https://registry.opendata.aws/usgs-landsat/ and [blog post](https://pystac.readthedocs.io/en/stable/tutorials/creating-a-landsat-stac.html)\n",
    " - https://registry.opendata.aws/sentinel-2-l2a-cogs/ with [blog post](https://www.element84.com/blog/sentinel-2-cloud-optimized-geotiffs-now-available-on-aws-registry-of-open-data) and [another blog post](https://aws.amazon.com/blogs/apn/transforming-geospatial-data-to-cloud-native-frameworks-with-element-84-on-aws/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b68caed-c057-47e7-8e48-fd09f06b62d5",
   "metadata": {},
   "source": [
    "## Specialty formats / beyond tabular - Array / Matrix\n",
    "\n",
    "Current recommendation for customers is [Zarr, which came from the Python scientific community](https://zarr.readthedocs.io/en/stable/), but has now been replicated in e.g. [Julia](https://github.com/JuliaIO/Zarr.jl) and [Rust](https://github.com/sci-rs/zarr) to varying degrees of completeness.\n",
    "\n",
    "Noteworthy dataset examples:\n",
    "- https://registry.opendata.aws/mur/ MUR Level 4 SST dataset in Zarr format. The zarr/ directory contains a zarr store chunked along the dimensions (time, lat, lon).\n",
    "\n",
    "\n",
    "- https://registry.opendata.aws/nwm-archive/ with 3 Zarr Tutorials: \n",
    "    - [Explore Repository of Tutorials on National Water Model V2.1 Retrospective Dataset in Zarr](https://github.com/NCAR/rechunk_retro_nwm_v21/tree/main/notebooks)\n",
    "    - [Explore the National Water Model V2.0 Retrospective in Zarr](https://nbviewer.jupyter.org/gist/rsignell-usgs/d3dfaf3cd3d8b39894a69b22127dfe38)\n",
    "    - [Explore the National Water Model V2.1 Retrospective Dataset in Zarr](https://github.com/NCAR/rechunk_retro_nwm_v21)\n",
    "\n",
    "\n",
    "- https://registry.opendata.aws/noaa-hrrr-pds/ with:\n",
    "    - [The HRRR Zarr Archive Managed by MesoWest](https://mesowest.utah.edu/html/hrrr/)\n",
    "    - [HRRR-B Python package: download and read HRRR grib2 files](https://github.com/blaylockbk/HRRR_archive_download)\n",
    "\n",
    "\n",
    "See also [Kerchunk](https://fsspec.github.io/kerchunk/index.html) which is an emerging approach to working with chunked array data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "default:Python",
   "language": "python",
   "name": "conda-env-default-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}