extending_a_container.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extending an Amazon SageMaker container\n",
    "\n",
    "With Amazon SageMaker, you can package your own algorithms that can then be trained and deployed in the SageMaker environment. This notebook guides you through an example on how to extend one of our existing and predefined SageMaker containers: the scikit-learn container in order to support the LightFM algorithm.\n",
    "\n",
    "By packaging an algorithm in a container, you can bring almost any code to the Amazon SageMaker environment, regardless of programming language, environment, framework, or dependencies. \n",
    "\n",
    "1. [Extending an Amazon SageMaker container](#Extending-an-Amazon-SageMaker-container)\n",
    "  1. [When should I extend a SageMaker container?](#When-should-I-extend-a-SageMaker-container?)\n",
    "  1. [Permissions](#Permissions)\n",
    "  1. [The example](#The-example)\n",
    "  1. [The presentation](#The-presentation)\n",
    "\n",
    "## When should I extend a SageMaker container?\n",
    "\n",
    "You may not need to create a container to bring your own code to Amazon SageMaker. When you are using a framework such as [scikit-learn](https://github.com/scikit-learn/scikit-learn), [MXNet](https://github.com/aws/sagemaker-mxnet-container), [PyTorch](https://github.com/aws/sagemaker-pytorch-container) or [Chainer](https://github.com/aws/sagemaker-chainer-container) that has direct support in SageMaker, you can simply supply the Python code that implements your algorithm using the SDK entry points for that framework.\n",
    "\n",
    "Even if there is direct SDK support for your environment or framework, you may want to add additional functionality or configure your container environment differently while utilizing our container to use on SageMaker.\n",
    "\n",
    "**Some of the reasons to extend a SageMaker deep learning framework container are:**\n",
    "1. Install additional dependencies. (E.g. I want to install a specific Python library, that the current SageMaker containers don't install.)\n",
    "2. Configure your environment. (E.g. I want to add an environment variable to my container.)\n",
    "\n",
    "**Although it is possible to extend any of our framework containers as a parent image, the example this notebook covers is currently only intended to work with our sagemaker-scikit-learn container.**\n",
    "\n",
    "This walkthrough shows that it is quite straightforward to extend one of our containers to build your own custom container.\n",
    "\n",
    "## Permissions\n",
    "\n",
    "Running this notebook requires permissions in addition to the normal `SageMakerFullAccess` permissions. This is because it creates new repositories in Amazon ECR. The easiest way to add these permissions is simply to add the managed policy `AmazonEC2ContainerRegistryFullAccess` to the role that you used to start your notebook instance. There's no need to restart your notebook instance when you do this, the new permissions will be available immediately.\n",
    "\n",
    "## The example\n",
    "\n",
    "In this example we show how to package a LightFM container, extending the SageMaker scikit-learn container, with a Python example which works with the MovieLens dataset. By extending the SageMaker scikit-learn container we already have some of the necessary training and inference toolkits already preinstalled on the container to make it work with SageMaker. We will still have to write code to make sure that the LightFM library can nicely interact with the SageMaker entry points.\n",
    "\n",
    "We will use two separate images to support training and inference. This is for educational purposes, because splitting the process in two helps to understand it better. Sometimes you may want use a single image to support training and hosting. A single image simplifies the procedure because we only need to manage one image for both tasks. Choosing whether to use a single image or two images is a matter of what is most convenient for you to develop and manage, but also has implications on the size of the containers themselves.\n",
    "\n",
    "If you're only using Amazon SageMaker for training or hosting, but not both, only the functionality used needs to be built into your container.\n",
    "\n",
    "[MovieLens]: https://grouplens.org/datasets/movielens/latest/\n",
    "\n",
    "## The presentation\n",
    "\n",
    "This presentation is divided into several parts: \n",
    "- [An end to end example of a LightFM model](./lightfm_example.ipynb)\n",
    "- [Building and using the _training_ container](./adapt_train_container.ipynb)\n",
    "- [Building and using the _inference_ container]((./adapt_inference_container.ipynb)"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}