Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle multiple data locations #4

Open
12 tasks done
Tracked by #2
donut opened this issue Feb 17, 2022 · 0 comments
Open
12 tasks done
Tracked by #2

Handle multiple data locations #4

donut opened this issue Feb 17, 2022 · 0 comments

Comments

@donut
Copy link
Owner

donut commented Feb 17, 2022

The main idea is to allow synced files to span multiple locations to address issues where you may have multiple drives with limited space. But I want to go beyond just monitoring free space at a location, I want to be able to set a maximum amount of space to be used at a location. That way I have the option to not use more than 1 TB on a 2 TB drive or something like that.

Another goal is that I want this to be flexible. If at one time I had the data files span multiple drives, I'd like the option to re-combine them a single drive. So saved file paths should be generic. This might create lookup problems. The basic (if slightly slow way), is to just check all locations for a specific file. This is fine as long as file names are unique within the whole system. Since the filenames include the ID of the video in the DB, we should be fine on this front. If lookups prove to be a choke point, we could add another field to the DB that includes a location prefix. In code we should just use the prefix as a hint of where to look first, but not assume that that's the only location the file could be at. This way we have some flexibility in merging files from different locations.

Tasks

  • update config example JSON with new locations structure (list of records? Prefix and max space usage?)
  • update dest config schema
  • comb through dest code for file system access
    • for reads, make sure we look in multiple locations before giving up
    • for writes, check current usage & remaining space before saving to a location
    • give clear error message and fail gracefully when out of space ( actual)
  • write out test objectives
  • switch to checking the 'Content-Length' header for size estimates
  • add missing .mli files (mainly File_store.mli)
  • go over touched code and add documentation comments
  • test filling up single volume
  • test spilling over into next volume

(prioritized by given order, then switch based on space)

@donut donut changed the title modify to handle multiple data locations (prioritized by given order, then switch based on space) Handle multiple data locations Feb 17, 2022
donut added a commit that referenced this issue Feb 23, 2022
Specifically, Config.Rdb_dest.file_stores. This code is untested but
compiles up to the point that something outside the config tries to
access the old files_path field.

issue #4
donut added a commit that referenced this issue Mar 9, 2022
Tracking usage became a significant complication, so we nixed it for
now since it's not a requirement. It's much simpler to just check for
free space.

The main complication is that checking current usage is a very
expensive operation (checking every file for size) that it we wouldn't
be able to do it for each file. That meant we would do the check once
on startup and then keep track of files added and removed from each
store and update the usage number based on that. It's definitely
something we can do, but doesn't make sense at this time since we're
using storage volumes dedicated for this app and don't need to keep
space available for other purposes.

issue #4
donut added a commit that referenced this issue Mar 9, 2022
Still needs some commenting and .mli files and other general cleanup,
but this is functional.

issue #4
donut added a commit that referenced this issue Mar 14, 2022
Before the priority looked at the JW provided `size` attribute of
media to determine the size of the video files. I had forgotten that
that size is the amount of all transcodes of the video combined which
usually ends up being double to triple the size of the actual video
file we're interested in. Now we checked the "content-length" header
of each file URI to estimate the space required. If that's unavailable
we fall back to the JW provided size value.

issue #4
donut added a commit that referenced this issue Mar 14, 2022
donut added a commit that referenced this issue May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant