Skip to content

tirprox/related_post_gen

 
 

Repository files navigation

Generate related posts based on tags. Sort by the number of shared tags.

Steps:

  • Read the posts JSON file.
  • Iterate over the posts and populate a map containing: tag -> List<Post> that has that tag
  • Iterate over the posts and for each post:
    • Create a map: Post -> int to track the number of shared tags
    • For each tag, Iterate over the posts that have that tag
    • For each post, increment the shared tag count in the map.
  • Sort the related posts by the number of shared tags.
  • Write the top 5 related posts for each post to a new JSON file.

Run Benchmark

./run.sh go | rust | python | all

Results (on my machine)

Language Time (avg) Time (minus I/O) Details
Rust 4.5s - Initial
Rust v2 2.60s - Replace std HashMap with fxHashMap by phazer99
Rust v3 1.28s - Preallocate and reuse map and unstable sort by vdrmn and Darksonn
Rust v4 0.13s - Use Post index as key instead of Pointer and Binary Heap by RB5009
Rust v5 52ms 38ms Rm hashing from loop and use vec[count] instead of map[index]count by RB5009
Rust v6 36ms 23ms Optimized Binary Heap Ops by scottlamb
Rust Rayon 22ms 9ms Parallelize by masmullin2000
Go 1.5s - Initial
Go v2 80ms - Add rust optimizations
Go v3 70ms 56ms Use goccy/go-json
Go v3 55ms 34ms Use generic binaryheap by DrBlury
Go v4 50ms 26ms Replace binary heap with custom priority queue
Go Con 33ms 10ms Go concurrency by tirprox and DrBlury
Go Con v2 29ms 5ms Use arena, use waitgroup, rm binheap by DrBlury
Python 7.81s - Initial
Python v2 1.53s 1.35s Add rust optimizations by dave-andersen
Numpy 0.85s 0.57s Numpy implementation by Copper280z
Crystal 96ms 50ms Inital w/ previous optimizations
Crystal 72ms 33ms Replace binary heap with custom priority queue

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Rust 24.5%
  • Go 22.7%
  • Python 21.3%
  • Shell 16.0%
  • Crystal 13.6%
  • jq 1.9%