Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decomposed based posix filesystem driver #4562

Merged
merged 103 commits into from
May 29, 2024
Merged
Changes from 1 commit
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
f4cbeb4
Switch posixfs to a local decomposedfs copy for now
aduffeck Feb 2, 2024
228be02
Fix creating/listing spaces
aduffeck Feb 6, 2024
8d3519c
Fix creating/listing dirs
aduffeck Feb 6, 2024
2592894
More fixes for CreateDir
aduffeck Feb 6, 2024
8612f5e
Don't do trashbin for now
aduffeck Feb 6, 2024
59a4506
Fix moving items
aduffeck Feb 7, 2024
a4838fb
Fix TouchFile
aduffeck Feb 7, 2024
db5b57f
Use inotifywatch to get notified about changes on disk
aduffeck Feb 7, 2024
6ed6c3e
Fix blobstore to retrieve the blob from the actual files
aduffeck Feb 7, 2024
9b45d3e
Use the proper lockfile extension for the meta lockfile
aduffeck Feb 7, 2024
372a50c
Ignore file with missing metadata
aduffeck Feb 7, 2024
643d5e1
Do not abort the watcher in case of errors
aduffeck Feb 7, 2024
c6b1b1f
Fix assimilating directories
aduffeck Feb 7, 2024
bbffead
Fix file uploads. Disable versioning for now.
aduffeck Feb 8, 2024
26efd39
Guard map access using a mutex
aduffeck Feb 8, 2024
03e0f3f
Scan files asynchronously
aduffeck Feb 8, 2024
43c6ebe
Lock the metadata when creating a dir
aduffeck Feb 8, 2024
07b2d79
Fix handling node ids when creating nodes
aduffeck Feb 8, 2024
be1878f
Remove decomposedfs layout specifics
aduffeck Feb 8, 2024
69625f3
Fix scanning new files
aduffeck Feb 8, 2024
07d3c17
Calculate checksums for assimilated files
aduffeck Feb 9, 2024
b3e7c7e
Make the upload folder configurable and make sure it exists
aduffeck Feb 9, 2024
718ef96
Create a new id if it's not set
aduffeck Feb 9, 2024
a9d7d47
Switch to github.com/aduffeck/inotifywaitgo to fix files with spaces
aduffeck Feb 9, 2024
9438606
Rescan the whole directory recursively if a directory was renamed
aduffeck Feb 12, 2024
7a9ef70
Fix a timing issue when touching files
aduffeck Feb 12, 2024
e06da68
Be more thorough cleaning up the meta lockfiles
aduffeck Feb 12, 2024
d97491a
Implement project spaces
aduffeck Feb 14, 2024
9d9a822
Switch back to upstream inotifywaitgo
aduffeck Feb 23, 2024
99d3e42
Make fs watching modular, introduce gpfsfileauditlogging watcher
aduffeck Feb 23, 2024
5a95bdb
Try to assimilate the parent before assimilating the item
aduffeck Feb 26, 2024
cf0f852
Fix deleting non-empty dirs
aduffeck Feb 26, 2024
b540d32
Switch to a store based id cache
aduffeck Feb 28, 2024
c0c7716
Add a "watchfolder" based gpfs watcher
aduffeck Mar 1, 2024
f4abdc4
Fix event names to listen for
aduffeck Mar 5, 2024
bf0b77e
Add a queue for items to be assimilated/scanned
aduffeck Mar 5, 2024
573f971
Fix recursive reindexing of moved directories
aduffeck Mar 5, 2024
9a78232
Fix reindexing
aduffeck Mar 5, 2024
4ff0d4f
Clean up mlock file after propagation
aduffeck Mar 5, 2024
232ce05
Take the list of kafka brokers from the config
aduffeck Mar 5, 2024
b27a7f8
Improve error logging
aduffeck Mar 11, 2024
c174301
Sync with decomposedfs from edge
aduffeck Mar 12, 2024
498764b
Fix space indexes
aduffeck Mar 12, 2024
f8b3c36
Get rid of the posix specific decomposedfs lib
aduffeck Mar 12, 2024
7088721
Refactor decomposedfs to be more extensible, e.g. for the posix driver
aduffeck Mar 18, 2024
3808c44
Improve logging
aduffeck Mar 18, 2024
8169ddc
Do not use a UserLayout by default
aduffeck Mar 18, 2024
9994da8
Fix creating new nodes
aduffeck Mar 18, 2024
b8098a7
Fix uploads
aduffeck Mar 18, 2024
eb5c8d9
Solve user mapping using a DecomposedFS middleware layer
aduffeck Apr 9, 2024
97f853f
Do not switch the user for project spaces. They belong to the ocis user
aduffeck Apr 22, 2024
5cb7687
Do not lock metadata by hand, use the metadata backend instead
aduffeck Apr 22, 2024
aba72a3
Do not switch the group id anymore
aduffeck Apr 22, 2024
fc1b603
Turn the user mapper into an interface for testability
aduffeck Apr 22, 2024
a4515c4
Pass the space type to the middlewares, so they can act accordingly
aduffeck Apr 22, 2024
86913ad
WIP
aduffeck Apr 23, 2024
7aef5d1
Merge branch 'more-extensible-decomposedfs' into decomposed-based-pos…
aduffeck Apr 23, 2024
3f13aee
Do not switch the uid anymore
aduffeck Apr 29, 2024
5cf4b81
Fix switching gid per space
aduffeck Apr 29, 2024
d9480bd
Set a null usermapper for non-switching fses
aduffeck Apr 29, 2024
1254b32
Create spaces with g+rx
aduffeck Apr 29, 2024
4c58789
Check if the space already exists before creating it
aduffeck Apr 29, 2024
58ad611
Fix updating storage spaces
aduffeck Apr 30, 2024
3fd1e9b
Give everything some time to settle down before assimilating
aduffeck Apr 30, 2024
4dfc398
Switch user when listing folder concurrently
aduffeck Apr 30, 2024
22aafd1
Set proper permission on spaces and the space base dir
aduffeck May 2, 2024
3025612
Panic if the inotifywait dependency is not fulfilled
aduffeck May 2, 2024
962aca7
Remove dead code
aduffeck May 2, 2024
c8264f7
Merge remote-tracking branch 'upstream/edge' into decomposed-based-po…
aduffeck May 6, 2024
d3520a7
Fix several issues and races in the posixfs driver
aduffeck May 15, 2024
d9d1a7d
Add pipeline for the posixfs driver
aduffeck May 15, 2024
3e37dd8
Merge remote-tracking branch 'upstream/edge' into decomposed-based-po…
aduffeck May 15, 2024
e5993ab
go mod tidy
aduffeck May 15, 2024
7ebebfd
Bump golang ci image
aduffeck May 16, 2024
e7bd1ec
Do not gather space gids when they are not needed
aduffeck May 16, 2024
74d268f
Bump ocis commit for the api tests
aduffeck May 16, 2024
0ebeeff
Kill previous watchers before starting a new one
aduffeck May 21, 2024
98dec69
Also listen to CLOSE_WRITE events to pick up changed files
aduffeck May 21, 2024
3b65e53
Only change the gid when the according option is enabled
aduffeck May 21, 2024
9e6bd81
Be more efficient and robus when assimilating. Fix updating on changes.
aduffeck May 21, 2024
fc610bc
Add unit tests
aduffeck May 21, 2024
38fa5d7
Fix treesize when assimilating directories. Increase test coverage.
aduffeck May 21, 2024
ccb49a6
Fix propagation
aduffeck May 22, 2024
190f4b1
Make the inotifywait watcher less verbose
aduffeck May 22, 2024
b0f46f2
Handle propagation errors
aduffeck May 22, 2024
ef70290
Try to make the tests more robust
aduffeck May 22, 2024
7171a68
Preserve treesize when assimilating moved directories
aduffeck May 22, 2024
1758f9b
Try to make tests more robust
aduffeck May 22, 2024
6878d90
Fix moving directories
aduffeck May 22, 2024
d63479c
Pin the image digest to pull in the inotifywait dependency
aduffeck May 23, 2024
e0bff14
Fix updating the id cache after a move
aduffeck May 23, 2024
c0ed917
Debounce scans for the same path
aduffeck May 23, 2024
ba18257
Update id cache for moved subtrees
aduffeck May 23, 2024
1e6267f
Merge remote-tracking branch 'upstream/edge' into decomposed-based-po…
aduffeck May 23, 2024
7158582
Adapt expected failures
aduffeck May 23, 2024
b635c61
Do not re-assimilate the whole tree after a move, just update the caches
aduffeck May 23, 2024
e3aa42e
Merge remote-tracking branch 'upstream/edge' into decomposed-based-po…
aduffeck May 28, 2024
c128a93
Adjust adapted failures
aduffeck May 28, 2024
9e67598
Try to fix the flakiness in CI
aduffeck May 28, 2024
712e922
Adjust adapted failures
aduffeck May 28, 2024
2b6c55c
Try to fix the flakiness
aduffeck May 28, 2024
06f6a5f
Adjust expected failures
aduffeck May 28, 2024
5ff2e82
Add changelog
aduffeck May 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix scanning new files
aduffeck committed Apr 9, 2024
commit 69625f3b21564b4f6b6a4eaad216f55b605090dd
45 changes: 25 additions & 20 deletions pkg/storage/fs/posix/tree/tree.go
Original file line number Diff line number Diff line change
@@ -139,24 +139,7 @@ func (t *Tree) Setup() error {

// Scan scans the given path and updates the id chache
func (t *Tree) Scan(path string) error {
_, ok := t.lookup.(*lookup.Lookup).IDCache.Get(context.Background(), "", path)
if ok {
return nil
}

id, err := t.lookup.MetadataBackend().Get(context.Background(), path, prefixes.IDAttr)
if err == nil {
return t.lookup.(*lookup.Lookup).IDCache.Set(context.Background(), "", string(id), path)
}

// lock the file for assimilation
lock, err := lockedfile.OpenFile(path+".flock", os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
return err
}
defer func() {
lock.Close()
}()
var err error

// find the space id
spaceID := []byte("")
@@ -172,7 +155,29 @@ func (t *Tree) Scan(path string) error {
return errors.New("could not find space id")
}

// check for the id attribute again after grabbing the lock, maybe the file was assimilated in the meantime
_, ok := t.lookup.(*lookup.Lookup).IDCache.Get(context.Background(), string(spaceID), path)
if ok {
return nil
}

id, err := t.lookup.MetadataBackend().Get(context.Background(), path, prefixes.IDAttr)
if err == nil {
return t.lookup.(*lookup.Lookup).IDCache.Set(context.Background(), string(spaceID), string(id), path)
}

// lock the file for assimilation
metaLockPath := t.lookup.MetadataBackend().LockfilePath(path)
mlock, err := lockedfile.OpenFile(metaLockPath, os.O_RDWR|os.O_CREATE, 0600)

if err != nil {
return err
}
defer func() {
_ = mlock.Close()
_ = os.Remove(metaLockPath)
}()

// check for the id attribute again after grabbing the lock, maybe the file was assimilated/created by us in the meantime
id, err = t.lookup.MetadataBackend().Get(context.Background(), path, prefixes.IDAttr)
switch err {
case nil:
@@ -205,7 +210,7 @@ func (t *Tree) Scan(path string) error {
attributes.SetString(prefixes.BlobIDAttr, id)
attributes.SetInt64(prefixes.BlobsizeAttr, stat.Size())
}
err = t.lookup.MetadataBackend().SetMultiple(context.Background(), path, attributes, true)
err = t.lookup.MetadataBackend().SetMultiple(context.Background(), path, attributes, false)
if err != nil {
return err
}