-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle multiple data locations #4
Comments
22 tasks
donut
added a commit
that referenced
this issue
Feb 23, 2022
Specifically, Config.Rdb_dest.file_stores. This code is untested but compiles up to the point that something outside the config tries to access the old files_path field. issue #4
donut
added a commit
that referenced
this issue
Mar 9, 2022
Tracking usage became a significant complication, so we nixed it for now since it's not a requirement. It's much simpler to just check for free space. The main complication is that checking current usage is a very expensive operation (checking every file for size) that it we wouldn't be able to do it for each file. That meant we would do the check once on startup and then keep track of files added and removed from each store and update the usage number based on that. It's definitely something we can do, but doesn't make sense at this time since we're using storage volumes dedicated for this app and don't need to keep space available for other purposes. issue #4
donut
added a commit
that referenced
this issue
Mar 9, 2022
Still needs some commenting and .mli files and other general cleanup, but this is functional. issue #4
donut
added a commit
that referenced
this issue
Mar 14, 2022
Before the priority looked at the JW provided `size` attribute of media to determine the size of the video files. I had forgotten that that size is the amount of all transcodes of the video combined which usually ends up being double to triple the size of the actual video file we're interested in. Now we checked the "content-length" header of each file URI to estimate the space required. If that's unavailable we fall back to the JW provided size value. issue #4
donut
added a commit
that referenced
this issue
Mar 14, 2022
donut
added a commit
that referenced
this issue
May 4, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The main idea is to allow synced files to span multiple locations to address issues where you may have multiple drives with limited space. But I want to go beyond just monitoring free space at a location, I want to be able to set a maximum amount of space to be used at a location. That way I have the option to not use more than 1 TB on a 2 TB drive or something like that.
Another goal is that I want this to be flexible. If at one time I had the data files span multiple drives, I'd like the option to re-combine them a single drive. So saved file paths should be generic. This might create lookup problems. The basic (if slightly slow way), is to just check all locations for a specific file. This is fine as long as file names are unique within the whole system. Since the filenames include the ID of the video in the DB, we should be fine on this front. If lookups prove to be a choke point, we could add another field to the DB that includes a location prefix. In code we should just use the prefix as a hint of where to look first, but not assume that that's the only location the file could be at. This way we have some flexibility in merging files from different locations.
Tasks
(prioritized by given order, then switch based on space)
The text was updated successfully, but these errors were encountered: