Various improvements #332
Open
Various improvements #332
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix deprecated in build.gradle
Users can use gradle (gradlew.bat or gradlew) without installing gradle. Example on linux: ./gradlew assemble Example on windows: gradlew.bat assemble
…slator service. This improves speed and reliability of English parsing.
…e for English names. Grammarchecker.net is unreliable. Instead, we scrape DMM's English version.
…he site instead of both versions. This should improve scraping speed.
…if DMM is slow for you. 2. Upgrade jsoup to 1.9.2 in build.gradle 3. In the case where DMM English scraping gives Japanese (due to server caching), we will do another scraping. 4. add new UserAgent 5. some more code refactors
Dmm improve
� Conflicts: � build.gradle � src/main/java/moviescraper/doctord/controller/siteparsingprofile/specific/DmmParsingProfile.java
Increase accuracy of search result matches by matching movie ID with search results Label field first. If no matches found, then try matching to URL as before.
…ed, XStream is probably vulnerable."
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change contains some improvements mainly to DMM scraping.
When scraping with DMM for English, we will now scrape the website's English version instead of getting Japanese and translating it via grammarchecker.com. This should improve scraping reliability and speed.
Add menu option to disable DMM scraping for actress. Skipping actress scraping on DMM will improve speed. Getting actress data from R18 is better.
Increase accuracy of search result matches by matching movie ID with search results Label field first. If no matches found, then try matching to URL as before. For example,
search for URE-051 on R18 will no longer return PURE-051.
search for DV-1195 on JavBus will no longer return HODV-21195.
Add Gradle v5.6.4 wrapper. Enable use of gradle without installing gradle (using gradlew.bat or gradlew).
Example on linux: ./gradlew assemble
Example on windows: gradlew.bat assemble
Known issue with DMM scraping. Some times we still get JP version even though we're requesting EN. This may be due to web server caching. If this happens, DMM scraper will wait 5 seconds and retry again. Occasionally, the 2nd try will still get JP version. If that happens, just restart the scrape.