This repository has been archived by the owner on Aug 25, 2021. It is now read-only.
Releases: fasten-project/mvn-crawler
Releases · fasten-project/mvn-crawler
Maven Crawler 0.1.0
Added
- Adds command-line arguments for running the crawler with various options.
- Sends Maven coordinates to a Kafka topic.
- Sends the extracted Maven coordinates as a JSON-compatible string.
- The extracted Maven coordinates have a timestamp based on Unix epochs.
- Avoids downloading and processing POM files if already exists on the disk.
- A non-recursive approach to extract the Maven coordinates.
- Saves and loads the crawled pages from a queue.
- Adds the URL of an extracted POM file to its JSON string.
- Users can set a limit to extract a specified number of Maven coordinates.
- In the case of no Kafka server, the extracted Maven coordinates can be saved in a file.
- Adds a
setup.py
script to install the crawler as a command-line tool.
Fixed
- Solved the issue of
URLError
while crawling Maven repositories. - Solved an issue where the artifactID of parent package is extracted from a POM file.
- Solved an issue where a wrong groupID is extracted from a POM file.
- Solved an issue where the JSON string of some Maven coordinates have newlines, tabs or spaces.