Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Heroku - Maximum Storage Size #1

Closed
sunnyrjuneja opened this issue Mar 6, 2014 · 8 comments
Closed

Heroku - Maximum Storage Size #1

sunnyrjuneja opened this issue Mar 6, 2014 · 8 comments

Comments

@sunnyrjuneja
Copy link

When pushing to heroku I get the following:

Running `script/setup_ac_db` attached to terminal... up, run.1305
===> Setting up the DB.
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
===> Populating the DB with data/alamedaco.json
===> Hang tight, this will take a few seconds...
[deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.
===> Cleaning up the DB...
===> Creating the DB indexes...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
MONGOID: Created indexes on Category:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Created indexes on Location:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Index: {:coordinates=>"2d"}, Options: {:min=>-180, :max=>180}
MONGOID: Created indexes on Organization:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
===> Creating the Elasticsearch index...
===> Creating the ElasticSearch index...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
[IMPORT] Deleting index 'production-ohana_api'
[IMPORT] Creating index 'production-ohana_api' with mapping:
{"location":{"properties":{"coordinates":{"type":"geo_point"},"address":{"type":"object","properties":{"zip":{"index":"analyzed","type":"string"},"city":{"index":"not_analyzed","type":"string"}}},"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"description":{"analyzer":"snowball","type":"string"},"products":{"index":"not_analyzed","type":"string"},"kind":{"type":"string","analyzer":"keyword"},"emails":{"type":"multi_field","fields":{"exact":{"type":"string","index":"not_analyzed"},"domain":{"type":"string","index_analyzer":"email_analyzer","search_analyzer":"keyword"}}},"urls":{"type":"string","index_analyzer":"url_analyzer","search_analyzer":"keyword"},"organization":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"services":{"type":"object","properties":{"categories":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"keywords":{"type":"multi_field","fields":{"keywords":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"name":{"type":"string","analyzer":"snowball"},"description":{"type":"string","analyzer":"snowball"},"service_areas":{"index":"not_analyzed","type":"string"}}}}}}
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (1)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (2)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (3)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (4)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (5)...
[ERROR] Too many exceptions occured, giving up. The HTTP response was: 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}
Location:      100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:39
[IMPORT] Done.
@siruguri
Copy link

siruguri commented Mar 6, 2014

Is this with or without any data you've extracted for AC?

On Thu, Mar 6, 2014 at 3:28 PM, Sunny Juneja [email protected]:

When pushing to heroku I get the following:

Running script/setup_ac_db attached to terminal... up, run.1305
===> Setting up the DB.
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
===> Populating the DB with data/alamedaco.json
===> Hang tight, this will take a few seconds...
[deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.
===> Cleaning up the DB...
===> Creating the DB indexes...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
MONGOID: Created indexes on Category:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Created indexes on Location:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Index: {:coordinates=>"2d"}, Options: {:min=>-180, :max=>180}
MONGOID: Created indexes on Organization:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
===> Creating the Elasticsearch index...
===> Creating the ElasticSearch index...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
[IMPORT] Deleting index 'production-ohana_api'
[IMPORT] Creating index 'production-ohana_api' with mapping:
{"location":{"properties":{"coordinates":{"type":"geo_point"},"address":{"type":"object","properties":{"zip":{"index":"analyzed","type":"string"},"city":{"index":"not_analyzed","type":"string"}}},"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"description":{"analyzer":"snowball","type":"string"},"products":{"index":"not_analyzed","type":"string"},"kind":{"type":"string","analyzer":"keyword"},"emails":{"type":"multi_field","fields":{"exact":{"type":"string","index":"not_analyzed"},"domain":{"type":"string","index_analyzer":"email_analyzer","search_analyzer":"keyword"}}},"urls":{"type":"string","index_analyzer":"url_analyzer","search_analyzer":"keyword"},"organization":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"services":{"type":"obje
ct","properties":{"categories":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"keywords":{"type":"multi_field","fields":{"keywords":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"name":{"type":"string","analyzer":"snowball"},"description":{"type":"string","analyzer":"snowball"},"service_areas":{"index":"not_analyzed","type":"string"}}}}}}
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (1)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (2)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (3)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (4)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (5)...
[ERROR] Too many exceptions occured, giving up. The HTTP response was: 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}
Location: 100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:39
[IMPORT] Done.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1
.

@sunnyrjuneja
Copy link
Author

With data.
-- 
Sunny Juneja
Sent with Airmail

On March 6, 2014 at 3:31:00 PM, Sameer S ([email protected]) wrote:

Is this with or without any data you've extracted for AC?

On Thu, Mar 6, 2014 at 3:28 PM, Sunny Juneja [email protected]:

When pushing to heroku I get the following:

Running script/setup_ac_db attached to terminal... up, run.1305
===> Setting up the DB.
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
===> Populating the DB with data/alamedaco.json
===> Hang tight, this will take a few seconds...
[deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.
===> Cleaning up the DB...
===> Creating the DB indexes...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
MONGOID: Created indexes on Category:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Created indexes on Location:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Index: {:coordinates=>"2d"}, Options: {:min=>-180, :max=>180}
MONGOID: Created indexes on Organization:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
===> Creating the Elasticsearch index...
===> Creating the ElasticSearch index...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
[IMPORT] Deleting index 'production-ohana_api'
[IMPORT] Creating index 'production-ohana_api' with mapping:
{"location":{"properties":{"coordinates":{"type":"geo_point"},"address":{"type":"object","properties":{"zip":{"index":"analyzed","type":"string"},"city":{"index":"not_analyzed","type":"string"}}},"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"description":{"analyzer":"snowball","type":"string"},"products":{"index":"not_analyzed","type":"string"},"kind":{"type":"string","analyzer":&q uot;keyw ord"},"emails":{"type":"multi_field","fields":{"exact":{"type":"string","index":"not_analyzed"},"domain":{"type":"string","index_analyzer":"email_analyzer","search_analyzer":"keyword"}}},"urls":{"type":"string","index_analyzer":"url_analyzer","search_analyzer":"keyword"},"organization":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"services":{"type":"obje
ct","properties":{"categories":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"keywords":{"type":"multi_field","fields":{"keywords":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"name":{"type":"string","analyzer":"snowball"},"description":{"type":"string","analyzer":"snowball"},"service_areas" ;:{" ;index":"not_analyzed","type":"string"}}}}}}
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (1)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (2)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (3)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (4)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (5)...
[ERROR] Too many exceptions occured, giving up. The HTTP response was: 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}
Location: 100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:39
[IMPORT] Done.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1
.

—
Reply to this email directly or view it on GitHub.

@siruguri
Copy link

siruguri commented Mar 6, 2014

Whoa. How many rows of data are you pushing?

On Thu, Mar 6, 2014 at 3:28 PM, Sunny Juneja [email protected]:

When pushing to heroku I get the following:

Running script/setup_ac_db attached to terminal... up, run.1305
===> Setting up the DB.
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
===> Populating the DB with data/alamedaco.json
===> Hang tight, this will take a few seconds...
[deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.
===> Cleaning up the DB...
===> Creating the DB indexes...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
MONGOID: Created indexes on Category:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Created indexes on Location:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
MONGOID: Index: {:coordinates=>"2d"}, Options: {:min=>-180, :max=>180}
MONGOID: Created indexes on Organization:
MONGOID: Index: {:_slugs=>1}, Options: {:unique=>true}
===> Creating the Elasticsearch index...
===> Creating the ElasticSearch index...
[RailsAdmin] RailsAdmin initialization disabled by default. Pass SKIP_RAILS_ADMIN_INITIALIZER=false if you need it.
** [Raven] Raven 0.6.0 ready to catch errors
Creating scope :near. Overwriting existing method Location.near.
[IMPORT] Deleting index 'production-ohana_api'
[IMPORT] Creating index 'production-ohana_api' with mapping:
{"location":{"properties":{"coordinates":{"type":"geo_point"},"address":{"type":"object","properties":{"zip":{"index":"analyzed","type":"string"},"city":{"index":"not_analyzed","type":"string"}}},"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"description":{"analyzer":"snowball","type":"string"},"products":{"index":"not_analyzed","type":"string"},"kind":{"type":"string","analyzer":"keyword"},"emails":{"type":"multi_field","fields":{"exact":{"type":"string","index":"not_analyzed"},"domain":{"type":"string","index_analyzer":"email_analyzer","search_analyzer":"keyword"}}},"urls":{"type":"string","index_analyzer":"url_analyzer","search_analyzer":"keyword"},"organization":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"services":{"type":"obje
ct","properties":{"categories":{"type":"object","properties":{"name":{"type":"multi_field","fields":{"name":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}}}},"keywords":{"type":"multi_field","fields":{"keywords":{"type":"string","index":"analyzed","analyzer":"snowball"},"exact":{"type":"string","index":"not_analyzed"}}},"name":{"type":"string","analyzer":"snowball"},"description":{"type":"string","analyzer":"snowball"},"service_areas":{"index":"not_analyzed","type":"string"}}}}}}
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (1)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (2)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (3)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (4)...
[ERROR] 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}, retrying (5)...
[ERROR] Too many exceptions occured, giving up. The HTTP response was: 400 > {"status":"error", "message":"You have reached your maximum storage size, upgrade your plan for more storage"}
Location: 100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:39
[IMPORT] Done.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1
.

@sunnyrjuneja
Copy link
Author

It really isn't that much. Just 2.2k locations + 1k organizations. I'm sure that number is going to grow by a factor of at least 6 with relatively little work.

@daguar
Copy link

daguar commented Mar 7, 2014

This is really surprising to me -- by default, the setup script ( https://github.com/codeforamerica/ohana-api/blob/master/script/setup_heroku#L31-L32 ) uses the mongo-lab Heroku add-on, and the free tier seems to provide 496mb of storage ( https://addons.heroku.com/mongolab ).

@whatasunnyday -- can you run a mongo command to check the storage usage of the "database"? I feel like it shouldn't be anywhere near 496mb yet, but maybe it's from stuff like indexing?

Worse comes to worst, the 2gb tier is $15/mo, and OpenOak might have Heroku credits. ( @spjika @eddietejeda -- do we? or is it just AWS?)

@siruguri
Copy link

siruguri commented Mar 7, 2014

In case the Heroku instance adds too much overhead, let me know - we can
try deploying to one of my Digital Ocean droplets... I wouldn't mind
spending a hacknight wrestling Mongo from scratch.

On Fri, Mar 7, 2014 at 8:02 AM, Dave Guarino [email protected]:

This is really surprising to me -- by default, the setup script (
https://github.com/codeforamerica/ohana-api/blob/master/script/setup_heroku#L31-L32) uses the mongo-lab Heroku add-on, and the free tier seems to provide
496mb of storage ( https://addons.heroku.com/mongolab ).

@whatasunnyday https://github.com/whatasunnyday -- can you run a mongo
command to check the storage usage of the "database"? I feel like it
shouldn't be anywhere near 496mb yet, but maybe it's from stuff like
indexing?

Worse comes to worst, the 2gb tier is $15/mo, and OpenOak might have
Heroku credits. ( @spjika https://github.com/spjika @eddietejedahttps://github.com/eddietejeda-- do we? or is it just AWS?)

Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-37037587
.

@monfresh
Copy link

This is actually an Elasticsearch issue. You can see the error is generated after the Elasticsearch index has started:

===> Creating the Elasticsearch index...

But still, the error message itself should specify which add-on needs to be upgraded.

The Elasticsearch add-on will throw this error when you have the free plan. Upgrading to a paid plan will solve the issue.

I need to add this to the documentation, but keep in mind that we are planning on replacing Mongo and Elasticsearch with Postgres over the next month.

@monfresh monfresh mentioned this issue Mar 14, 2014
Closed
@sunnyrjuneja
Copy link
Author

documented upstream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@siruguri @monfresh @daguar @sunnyrjuneja