Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(spikyball): filtering zero/NaN-probabilities inhibits unseen node discovery #61

Open
pekasen opened this issue May 31, 2023 · 0 comments

Comments

@pekasen
Copy link
Member

pekasen commented May 31, 2023

It is yet not finally set how we figure out how to handle whether to get data from the live source or from cache.
This leads to the situation described in #60 were the sole presence of the node in the nodes table already inhibitated fetching the data from the live source.


Solutions/items collection:

  • Delegate the decision which nodes to fetch completely to the sampler (similarly to feat: allow strategies more autonomy in sampling decisions #47). However, the main problem is when to retrieve data for the neighbours. Each connector is different in what data it returns (the requested node and outgoing edges for the Telegram scraper for instance or all outgoing edges and neighbour node information but not the information on the requested node for the Twitter connectors).
  • Add a seperate prepare-step before sampling: this step the data set passed into the sampler will be curated, this step may be repeated several times until the data set for this iteation is complete (e.g. we could retry things here if necesarry).
  • Similarly, we could implemented the above described behaviour in the sampling step as well.
  • Facts in favor of the second solution is that the sampler is keeping track of which nodes it has seen and may want to see again. However, the sampler does not know which information is in the cache and whether to retrieve from there or to refresh the information.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant