Hooks/Signals on fs events [suggestion]

[ghost]

There is a basic function missing that i'm sure a lot would appreciate.
To do something on certain events like before and after a file is added in the file system. The best example would be Django signals, also often called hooks depending on the context.
In my case i need to maintain custom metadata files for each actual files in the PyFilesystem.
Or have i fail to find a way in the already defined API?

P.S.
This is a very nice software design, thanks for making it available!

[willmcgugan]

Hi Carl,

The problem with such a system is that not all filesystems are capable of detecting changes outside of what your code writes. As far as I know, only the OS has this capability.

So it would be possible to implement in OSFS, it wouldn't work with MemoryFS, FTPFS etc.

You might want to look in to a library called 'watchdog' to watch the OS filesystem for changes. It works really well.

[ghost]

Not even a chance to talk! I see the kind, i think my project won't need yours.

[willmcgugan]

@cldtech I wasn't hostile to your suggestion. If there is more to discuss, feel free.

[ghost]

Perhaps i don't understand the meaning of closing the issue. I would have said that i am looking for tools to facilitate the development of a distributed operating system based on BSD.
Meaning "not all filesystems being capable of detecting changes" is not really a consideration.

[althonos]

I think there may be a misunderstanding about what @willmcgugan mean by "filesystems".

Pyfilesystem2 here provides an abstraction of filesystems using a unified Python interface, including zip files, tar archives, FTP server, etc. Out of the builtin filesystems that ship with Pyfilesystem2, only OSFS could possibly support this, and MemoryFS with some tweaking. So by filesystem, we mean a concrete class implementing the fs.base.FS interface.

In that context, it is very true that not all filesystems are capable of detecting changes: FTP servers are a good example. Users could not use the hook API in a generic manner; because of this, it's probably not a very useful addition to Pyfilesystem2, which strives at being as generic as possible.

However, this does not prevent a user from using the aforementioned watchdog library with Pyfilesystem2: if you are sure that you are using an OSFS, or any filesystem mapping to a real path on your machine, then you could use FS.getsyspath to obtain a system path, and feed it to watchdog.

[ghost]

I think you didn't read me at all, "facilitate the development of a distributed operating system" mean i decide what i expose to the next top layer, this is gonna be inside a kernel. I do have total control over everything at this level. NOTHING, from the BSD system is gonna be accessible through the userland. Again, i look for parts, not a complete solution there is none right now. But everything has to start somewhere! PyFilesystem would have been a good start for at the very least mapping the paths, exposing trees to userland and some protocols integrations like ftp into what will be the native file manager of the OS.

[jayvdb]

Just noting that tar/zip/etc FS can support this when backed by a file, because those drivers can have a watch on the file, and when backed by memory with the same approach MemoryFS would need.

I believe this would be a very powerful addition to pyfilesystem2 even if it is a part of the interface which some underlying FS do not / can not support. Lots of tools are becoming more aware of file modifications, and it makes no sense to adopt pyfilesystem2 and then also manage watchdog on raw files. If an app needs to manage watchdog, the codebase looses most the benefits of having an FS abstraction layer, and it becomes a duplication and maybe even source of extra complexity.

[lurch]

Perhaps a WrapFS could be a solution here? When constructed your WrapFS could see if the wrapped FS has a valid syspath, and then set up an appropriate OS-level watchdog on that syspath, returning the results/events in whatever format you'd like? 🤷

Although if you're writing a "distributed operating system" I wonder if maybe PyFilesystem is the wrong tool for the job? Especially given that there's no "expose" support in pyfs2 yet? #274