You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It is yet not finally set how we figure out how to handle whether to get data from the live source or from cache.
This leads to the situation described in #60 were the sole presence of the node in the nodes table already inhibitated fetching the data from the live source.
Solutions/items collection:
Delegate the decision which nodes to fetch completely to the sampler (similarly to feat: allow strategies more autonomy in sampling decisions #47). However, the main problem is when to retrieve data for the neighbours. Each connector is different in what data it returns (the requested node and outgoing edges for the Telegram scraper for instance or all outgoing edges and neighbour node information but not the information on the requested node for the Twitter connectors).
Add a seperate prepare-step before sampling: this step the data set passed into the sampler will be curated, this step may be repeated several times until the data set for this iteation is complete (e.g. we could retry things here if necesarry).
Similarly, we could implemented the above described behaviour in the sampling step as well.
Facts in favor of the second solution is that the sampler is keeping track of which nodes it has seen and may want to see again. However, the sampler does not know which information is in the cache and whether to retrieve from there or to refresh the information.
The text was updated successfully, but these errors were encountered:
It is yet not finally set how we figure out how to handle whether to get data from the live source or from cache.
This leads to the situation described in #60 were the sole presence of the node in the nodes table already inhibitated fetching the data from the live source.
Solutions/items collection:
prepare
-step before sampling: this step the data set passed into the sampler will be curated, this step may be repeated several times until the data set for this iteation is complete (e.g. we could retry things here if necesarry).The text was updated successfully, but these errors were encountered: