Change data capture of web sites

[gedw99]

My aim is to have a cdc web scraper .

So it collects a registry of what is in a web site , and then when you bind to one of them it tells you if it changes .

the registry is just a list of all images and content in a web site.

so then you can be alerted when any part of he web site changes and quickly see it .

this makes it a more useful thing in practical terms.

you can even record the registry over time , and then compare one against the other to work out what got refactored . Like a git diff.

I would be open to adding this .

the registry could be a simple SQLite db.
The store of content and images could be an s3 like minio .

The differencing system can be a simple compare of the 2 registry runs and some sort of json output

Is anyone else interested in this ?

it will make it easy to do production work I think

[GeorginaReeder]

Hi there @gedw99 , thanks for this!

Just tagging @dwisiswant0 to check whether this is something we're in need of.

[gedw99]

If you look at conduit it has a slew of cdc plugins

it would gain all the cdc base framework aspects if we used this.

for security ops it’s fantastic to be able to wiretap , if you know what I mean, any source.

https://github.com/ConduitIO/

https://github.com/conduitio-labs has extra cdc plugins

So I would add a web scraper to it that’s designed for CDC