Reinvent keynote3 (#4490)

* feat: Add notebooks for step decorator (#272) * UCI heart example for step decorator with EMR step for preprocessing (#266) * UCI heart example for step decorator with EMR step for preprocessing * UCI heart example for step decorator with EMR step, after linter * UCI heart example for step decorator with EMR step, after linter * removed content_types * using prod S3 bucket * using XGBClassifier * fix code format --------- Co-authored-by: feliplp <[email protected]> Co-authored-by: Dewen Qi <[email protected]> * add notebook example on basic pipeline for batch inference using step decorator (#264) * add basic pipeline for batch inference using step decorator * change Booster to XGBClassifier; incorporating feedback from aws/amazon-sagemaker-examples-staging#264 * fix minor typos * incorporate comments from PR aws/amazon-sagemaker-examples-staging#264 * incorporate feedback from aws/amazon-sagemaker-examples-staging#264 --------- Co-authored-by: pprasety <[email protected]> * Add remote-function notebook fixes (#265) Co-authored-by: svia3 <[email protected]> * Add pipeline step decorator quick start notebook (#267) add pipeline scheduler examples Address comments and refine Add pipeline step decorator ablone notebook Address review meeting comments Update clean up sections Add rate-based schedules back Udpate notebooks for Public Beta Fix the colliding endpoint name across different executions Upgrading pandas to fix ImportError in Studio DataScience 3.0 image add scheduler-light additions to quick_start notebook add scheduler-light additions to quick_start notebook fix invalid notebook json Update notebooks for GA Add modular package for lightsaber Add a simple notebook to demonstrate mix use of training step and step deco Address comments fix pipeline delete resouce leak issue in using_step_decorator notebook dummy commit Co-authored-by: Dewen Qi <[email protected]> * remove local SDK tar and retrieve SDK from public --------- Co-authored-by: Felipe Lopez <[email protected]> Co-authored-by: feliplp <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Philips Kokoh <[email protected]> Co-authored-by: pprasety <[email protected]> Co-authored-by: Stephen Via <[email protected]> Co-authored-by: svia3 <[email protected]> * Notebook Job Step Example (#274) * Create README.md * Adding notebooks * Delete sagemaker-pipelines/notebook-job-step/README.md * Adding example for inference components and managed instance scaling for SageMaker real time hosting and inference (#275) * Cleaned up notebooks Cleaned up for initial push to staging * removed references to goldfinch * Updated readme * moved to proper directory * fixed session object reference * Updated session variable * Updated with logic to check store vars. Need to remove internal only code * linted notebooks and added test header and footers * Fixed prompt and parameters for codegen25 * Updated descriptions, added handling * Removed custom model shapes * Making Jumpstart notebooks Python 3.10 compatible (#269) * removed roleARN which is not needed * smart sifting notebooks (#281) * smart sifting notebooks * smart sifting notebooks updated description * Added new flow diagram --------- Co-authored-by: Arun Lokanatha <[email protected]> --------- Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Felipe Lopez <[email protected]> Co-authored-by: feliplp <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Philips Kokoh <[email protected]> Co-authored-by: pprasety <[email protected]> Co-authored-by: Stephen Via <[email protected]> Co-authored-by: svia3 <[email protected]> Co-authored-by: Ram Vegiraju <[email protected]> Co-authored-by: James Park <[email protected]> Co-authored-by: Pooja Karadgi <[email protected]> Co-authored-by: Arun Lokanatha <[email protected]> Co-authored-by: Arun Lokanatha <[email protected]>
aws · Nov 30, 2023 · abea136 · abea136
1 parent ef36606
commit abea136
Show file tree

Hide file tree

Showing 62 changed files with 17,917 additions and 66 deletions.
diff --git a/.gitignore b/.gitignore
@@ -7,3 +7,6 @@
 **/_build
 *.iml
 tox.ini
+
+**/sagemaker-pipelines/step-decorator/**/emr-examples/code
+**/sagemaker-pipelines/step-decorator/**/quick-start/**/dummy_train.py
diff --git a/README.md b/README.md
@@ -189,6 +189,13 @@ More examples for models such as BERT and YOLOv5 can be found in [distributed_tr
 - [Train EleutherAI GPT-J with Model Parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/model_parallel/gpt-j/11_train_gptj_smp_tensor_parallel_notebook.ipynb) shows how to train EleutherAI GPT-J with PyTorch and Tensor Parallelism technique in the SageMaker Model Parallelism Library.
 - [Train MaskRCNN with Data Parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/data_parallel/maskrcnn/pytorch_smdataparallel_maskrcnn_demo.ipynb) shows how to train MaskRCNN with PyTorch and SageMaker Data Parallelism Library.
 
+### Amazon SageMaker Smart Sifting
+
+These examples provide an Introduction to Smart Sifting library. Smart Sifting is a framework to speed up training of PyTorch models. The framework implements a set of algorithms that filter out inconsequential training examples during training, reducing the computational cost and accelerating the training process. It is configuration-driven and extensible, allowing users to add custom logic to transform their training examples into a filterable format. Smart sifting provides a generic utility for any DNN model, and can reduce the training cost by up to 35% in infrastructure cost.
+
+- [Train Image Classification using Vision Transformer with Smart Sifting](https://github.com/aws/amazon-sagemaker-examples/tree/main/training/smart_sifting/Image_Classification_VIT/Train_Image_classification.ipynb): This Example shows how to use Smart sifting to fine tune Vision Transformers for Image Classification.
+- [Train Text Classification using BERT with Smart Sifting](https://github.com/aws/amazon-sagemaker-examples/tree/main/training/smart_sifting/Text_Classification_BERT/Train_text_classification.ipynb): This Example shows how to use Smart Sifting to fine tune BERT for Text Classification.
+
 ### Amazon SageMaker Clarify
 
 These examples provide an introduction to SageMaker Clarify which provides machine learning developers with greater visibility into their training data and models so they can identify and limit bias and explain predictions.

diff --git a/...e/generativeai/llm-workshop/lab-inference-components-with-scaling/1_create_endpoint.ipynb b/...e/generativeai/llm-workshop/lab-inference-components-with-scaling/1_create_endpoint.ipynb