Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: handle oom #163

Merged
merged 5 commits into from
Sep 7, 2022
Merged

feat: handle oom #163

merged 5 commits into from
Sep 7, 2022

Conversation

Phelan164
Copy link
Contributor

@Phelan164 Phelan164 commented Sep 7, 2022 •
edited
Loading

Because

  • Triton inference server has a problem with memory when the model run with a huge batch size

This commit

  • add handling output an OOM error when the Triton server could not allocate the memory for model
  • add max_batch_size limitation for a model

@codecov
Copy link

codecov bot commented Sep 7, 2022 •
edited
Loading

Codecov Report

Merging #163 (992d71c) into main (9033c50) will decrease coverage by 0.53%.
The diff coverage is 0.00%.

@@           Coverage Diff            @@
##            main    #163      +/-   ##
========================================
- Coverage   5.02%   4.49%   -0.54%     
========================================
  Files          5       5              
  Lines       3024    3382     +358     
========================================
  Hits         152     152              
- Misses      2840    3198     +358     
  Partials      32      32
Flag Coverage Δ
unittests 4.49% <0.00%> (-0.54%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
internal/util/util.go 2.66% <0.00%> (-0.13%) ⬇️
pkg/handler/handler.go 0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@Phelan164 Phelan164 merged commit 4db1c45 into main Sep 7, 2022
@Phelan164 Phelan164 deleted the handle-oom branch September 7, 2022 14:59
This was referenced Sep 7, 2022
Merged
Merged
@xiaofei-du xiaofei-du mentioned this pull request Sep 20, 2022
Merged
xiaofei-du pushed a commit to instill-ai/instill-core that referenced this pull request Sep 20, 2022
@droplet-bot
chore(main): release 0.2.6-alpha (#147)
1522379 
🤖 I have created a release *beep* *boop*
---

## [0.2.6-alpha](v0.2.5-alpha...v0.2.6-alpha) (2022-09-20)

## Product Updates

BREAKING changes 🅱️
- updated VDP protocol

Features ✨
- add data mapping field `data_mapping_indices` in the data to be sent to the pipeline destination. The field includes unique IDs that can be mapped to CV task output for each processed image
- 
- support all CV tasks when building `ASYNC` pipelines
- add confidence score for OCR task

Fix bugs 🐛
- fix `ASYNC` trigger block issue, close https://github.com/instill-ai/pipeline-backend/issues/67
- allow user-defined description for GitHub model

## VDP

### Bug Fixes

* update examples to be compatible with the latest protocol ([36a847b](36a847b))


### Miscellaneous Chores

* release v0.2.6-alpha ([b291862](b291862))

## Pipeline-backend ([0.8.0-alpha](instill-ai/pipeline-backend@v0.7.2-alpha...v0.8.0-alpha))

### Features

* add data mapping ([0db4bfe](instill-ai/pipeline-backend@0db4bfe))


### Bug Fixes

* fix async trigger block issue ([59f0fb8](instill-ai/pipeline-backend@59f0fb8)), closes [#67](https://github.com/instill-ai/pipeline-backend/issues/67)
* fix multipart trigger data_mapping_indices empty ([d3160b4](instill-ai/pipeline-backend@d3160b4))

## Connector-backend ([0.6.0-alpha](instill-ai/connector-backend@v0.5.3-alpha...v0.6.0-alpha))

### Features

* add data mapping ([5596716](instill-ai/connector-backend@5596716))
* support all cv tasks defined in vdp protocol ([c38f9e8](instill-ai/connector-backend@c38f9e8)), closes [#33](https://github.com/instill-ai/connector-backend/issues/33)


### Bug Fixes

* add http/grpc release stage in connector definitions ([f2648c0](instill-ai/connector-backend@f2648c0)), closes [#30](https://github.com/instill-ai/connector-backend/issues/30)
* fix destination write for multiple models ([860f62e](instill-ai/connector-backend@860f62e))
* fix update not using name field ([f8e98c3](instill-ai/connector-backend@f8e98c3))
* fix zero-value field dropped ([31e297c](instill-ai/connector-backend@31e297c))

## Model-backend ([v0.8.1-alpha](https://github.com/instill-ai/model-backend/releases/tag/v0.8.1-alpha))

### Features

* handle oom ([#163](instill-ai/model-backend#163)) ([4db1c45](instill-ai/model-backend@4db1c45))
* add confidence score for ocr output ([#167](instill-ai/model-backend#167)) ([e915452](instill-ai/model-backend@e915452))

### Bug Fixes

* update description for GitHub model from user input ([#173](instill-ai/model-backend#173)) ([821dab3](instill-ai/model-backend@821dab3))

### Miscellaneous Chores

* release 0.7.3-alpha ([9033c50](instill-ai/model-backend@9033c50))

## Console ([v0.12.0-alpha](https://github.com/instill-ai/console/releases/tag/v0.12.0-alpha))

### Features

* add buffer into calculation of instance cell more button ([#268](instill-ai/console#268)) ([19d1470](instill-ai/console@19d1470))
* add destination configuration form ([#255](instill-ai/console#255)) ([0164946](instill-ai/console@0164946))
* Implement onboarding integration test ([#271](instill-ai/console#271)) ([f9c494e](instill-ai/console@f9c494e))
* Implement source integration test ([#272](instill-ai/console#272)) ([b85a285](instill-ai/console@b85a285))
* use env var in pipeline snippet ([#275](instill-ai/console#275)) ([3bc4d7a](instill-ai/console@3bc4d7a))

### Bug Fixes

* fix onboard cookie issue ([#252](instill-ai/console#252)) ([a4a93cc](instill-ai/console@a4a93cc))
* fix various typo ([#249](instill-ai/console#249)) ([d60f860](instill-ai/console@d60f860))
* update the sample github repos ([#253](instill-ai/console#253)) ([82592dd](instill-ai/console@82592dd))
* fix cookie_token wrongly set issue ([#269](instill-ai/console#269)) ([7e25d9d](instill-ai/console@7e25d9d))
* fix Select a existing to Select an existing ([#264](instill-ai/console#264)) ([87a8584](instill-ai/console@87a8584))
* fix the missing destination deletion ([#267](instill-ai/console#267)) ([f96e65c](instill-ai/console@f96e65c))
* fix various issues related to destination icons and displayed title ([#266](instill-ai/console#266)) ([d86c25c](instill-ai/console@d86c25c))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This was referenced Feb 23, 2023
Closed
Closed
This was referenced Apr 15, 2023
Closed
Merged
Phelan164 pushed a commit that referenced this pull request Apr 24, 2023
@droplet-bot
chore(main): release 0.11.0-alpha (#340)
8cd420b 
🤖 I have created a release *beep* *boop*
---


##
[0.11.0-alpha](v0.16.0-alpha...v0.11.0-alpha)
(2023-04-24)


### Features

* add codebase for model grpc service
([4defa3e](4defa3e))
* add confidence score for ocr output
([#167](#167))
([e915452](e915452))
* add credential definition
([#109](#109))
([92d3391](92d3391))
* add gRPC Gateway and GetModel API
([#7](#7))
([bff6fc9](bff6fc9))
* add model initialization module
([#332](#332))
([aa753a5](aa753a5))
* add private endpoint and gRPC test cases
([#306](#306))
([bb3c193](bb3c193))
* add release stage for model definition
([#153](#153))
([4e13ba5](4e13ba5))
* add support for text generation tasks
([#252](#252))
([767ec45](767ec45))
* add text to image task
([#239](#239))
([421eb1a](421eb1a))
* **controller:** add model state monitoring with controller
([#323](#323))
([4397826](4397826))
* create model from GitHub
([#61](#61))
([cf763cb](cf763cb))
* handle oom
([#163](#163))
([4db1c45](4db1c45))
* remove model instance
([#320](#320))
([15e1b62](15e1b62))
* support artivc
([#102](#102))
([b8e21a4](b8e21a4))
* support async deploy and undeploy model instance
([#192](#192))
([ed36dc7](ed36dc7))
* support creating a HuggingFace model
([#113](#113))
([1577d87](1577d87))
* support instance segmentation task
([#183](#183))
([d28cfdc](d28cfdc))
* support model caching
([#317](#317))
([d15ffba](d15ffba))
* support model name when creating model
([#25](#25))
([7d799b7](7d799b7))
* support ocr task
([#150](#150))
([7766c6f](7766c6f))
* support semantic segmentation
([#203](#203))
([f22262c](f22262c))
* support url/base64 content prediction
([#34](#34))
([a88ddfd](a88ddfd))


### Bug Fixes

* add link for guideline create Conda environment file
([7ee8e06](7ee8e06))
* add writeonly to description
([f59d98f](f59d98f))
* allow updating emtpy description for a model
([#177](#177))
([100ec84](100ec84))
* bug usage storage
([#103](#103))
([975fdc1](975fdc1))
* clone repository and make folder
([ac79386](ac79386))
* **config:** use private port for mgmt-backend
([#307](#307))
([3264e2b](3264e2b))
* correct version when making inference
([#31](#31))
([c918e77](c918e77))
* create a subfolder in model-repository if needed
([#290](#290))
([7f8d78b](7f8d78b))
* fix build and go version
([#9](#9))
([f8d4346](f8d4346))
* fix client stream server recv wrong file length interval
([#143](#143))
([0e06f7c](0e06f7c))
* fix config path
([a8cf2c0](a8cf2c0))
* fix creating subfolder
([105a11a](105a11a))
* fix duration configuration bug
([ee4a310](ee4a310))
* fix keypoint model payload parser
([#249](#249))
([461d54a](461d54a))
* fix list long-run operation error
([#220](#220))
([472696d](472696d))
* fix subfolder creation
([#292](#292))
([0b6ec3f](0b6ec3f))
* fix unload model issue causing Triton server OOM
([#42](#42))
([fb4d1d1](fb4d1d1))
* fix usage client nil issue when mgmt-backend not ready
([#241](#241))
([4290159](4290159))
* fix variable name
([#293](#293))
([a7995dd](a7995dd))
* HuggingFace batching bug in preprocess model
([b1582e8](b1582e8))
* init config before logger
([9d3fb4a](9d3fb4a))
* keep format for empty inference output
([#258](#258))
([e2a2e48](e2a2e48))
* list models and model instances pagination
([#304](#304))
([1f19ed4](1f19ed4))
* logic when essemble or not
([ab8e7c1](ab8e7c1))
* model configuration response in integration test
([0225c1e](0225c1e))
* model definition in list model and missing zero in output
([#121](#121))
([a90072d](a90072d))
* model instance state update to unspecified state
([#206](#206))
([14c87d5](14c87d5))
* panic error with nil object
([#208](#208))
([a342113](a342113))
* pass the context between package layers
([#345](#345))
([e6e7f2f](e6e7f2f))
* post process for unspecified task output
([ad88068](ad88068))
* post process ocr task
([e387154](e387154))
* postgres host
([a322165](a322165))
* refactor JSON schema
([f24db48](f24db48))
* refactor model definition and model JSON schema
([#73](#73))
([0cce154](0cce154))
* regexp zap logger with new protobuf package
([8b9c463](8b9c463))
* return list of models in list method
([b88ebd7](b88ebd7))
* status code when deploy model error
([#111](#111))
([31d3f11](31d3f11))
* trigger image with 4 channel
([#141](#141))
([7445f5f](7445f5f))
* update db schema, protobuf generated files and create model, version
in upload api
([7573e54](7573e54))
* update description for GitHub model from user input
([#173](#173))
([821dab3](821dab3))
* update docker compose file for building dev image
([#29](#29))
([83cba09](83cba09))
* update model definitions and tasks in usage collection
([#100](#100))
([c593087](c593087))
* update predict for essemble model
([016f11c](016f11c))
* update version order when get model version list
([#38](#38))
([83c054a](83c054a))
* wrong logic when checking user account and service account
([7058db6](7058db6))


### Miscellaneous Chores

* release 0.11.0-alpha
([d592acb](d592acb))
* release 0.3.2-alpha
([9f8cd91](9f8cd91))
* release 0.4.2-alpha
([fc5a14a](fc5a14a))
* release 0.7.2-alpha
([17529d6](17529d6))
* release 0.7.3-alpha
([9033c50](9033c50))
* release v0.5.1-alpha
([895056d](895056d))
* release v0.6.1-alpha
([f18dc30](f18dc30))
* release v0.6.2-alpha
([4365f32](4365f32))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
This was referenced Apr 24, 2023
Closed
Merged
@droplet-bot droplet-bot mentioned this pull request May 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pinglin pinglin Awaiting requested review from pinglin pinglin is a code owner

@xiaofei-du xiaofei-du Awaiting requested review from xiaofei-du xiaofei-du is a code owner

Assignees
No one assigned
Labels
None yet
Projects
No open projects
🔮 Instill Core
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Phelan164