Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better folder layout for better scaling #5

Closed
tokee opened this issue Jan 6, 2017 · 1 comment
Closed

Better folder layout for better scaling #5

tokee opened this issue Jan 6, 2017 · 1 comment
Assignees

Comments

@tokee
Copy link
Owner

tokee commented Jan 6, 2017

With the default settings, creating static tiles for 1 million images will result in 12 million tile files in a single folder. With many file systems this is detrimental to performance. This can be solved by creating a custom tileSource for OpenSeadragon with more sub-folder, such as level/x/x_y.jpg instead of level/x_y.jpg.

This does not change the basic problem: A large amount of small files. Using an image server with pyramid TIFFs solves this, but this introduces a server-side requirement and puts us back to square 1: Having to produce 1 very large image.

A Frankenstein solution would be to create multiple pyramid-TIFFs, each one containing a subset of the tiles. This would make it possible to do near-infinite scale collages, at the cost of a significant increase in both implementation and infrastructure complexity.

@tokee tokee self-assigned this Jan 6, 2017
@tokee tokee closed this as completed in 87890d2 Feb 2, 2017
@tokee
Copy link
Owner Author

tokee commented Feb 2, 2017

The implemented solution is the fairly-large-scale one: If enabled, tiles are bucked together in sub-folders. The solution only uses a single sub-level, so the performance of some file systems will gradually deteriorate when creating (and possibly also when viewing) collages of more than 30 million images (using the default setup).

Guesstimating a bit, things will probably be okay up to a few hundred million images. Should anyone need to go beyond that, I'll be happy to write the code for using multi-level sub-foldering for the tiles.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant