Skip to content

Commit

Permalink
chrono-analyzer project
Browse files Browse the repository at this point in the history
AlwaysBCoding committed Feb 16, 2020
0 parents commit 0c10517
Showing 45 changed files with 16,811 additions and 0 deletions.
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.DS_Store
scraper/images/
server/myappenv/
ui/node-modules/
Binary file added models/classifier.pkl
Binary file not shown.
Binary file added models/regression.pkl
Binary file not shown.
756 changes: 756 additions & 0 deletions notebooks/Chrono Analyzer - Classification Model.ipynb

Large diffs are not rendered by default.

638 changes: 638 additions & 0 deletions notebooks/Chrono Analyzer - Regression Model.ipynb

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions scraper/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
ruby '2.5.3'

gem 'nokogiri'
gem 'watir'
gem 'pry'
35 changes: 35 additions & 0 deletions scraper/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
GEM
specs:
childprocess (0.9.0)
ffi (~> 1.0, >= 1.0.11)
coderay (1.1.2)
ffi (1.11.1)
method_source (0.9.2)
mini_portile2 (2.4.0)
nokogiri (1.10.2)
mini_portile2 (~> 2.4.0)
pry (0.12.2)
coderay (~> 1.1.0)
method_source (~> 0.9.0)
regexp_parser (1.3.0)
rubyzip (1.2.2)
selenium-webdriver (3.141.0)
childprocess (~> 0.5)
rubyzip (~> 1.2, >= 1.2.2)
watir (6.16.5)
regexp_parser (~> 1.2)
selenium-webdriver (~> 3.6)

PLATFORMS
ruby

DEPENDENCIES
nokogiri
pry
watir

RUBY VERSION
ruby 2.5.3p105

BUNDLED WITH
2.0.2
299 changes: 299 additions & 0 deletions scraper/data/audemarspiguet.txt

Large diffs are not rendered by default.

303 changes: 303 additions & 0 deletions scraper/data/breitling.txt

Large diffs are not rendered by default.

300 changes: 300 additions & 0 deletions scraper/data/cartier.txt

Large diffs are not rendered by default.

300 changes: 300 additions & 0 deletions scraper/data/gucci.txt

Large diffs are not rendered by default.

304 changes: 304 additions & 0 deletions scraper/data/iwc.txt

Large diffs are not rendered by default.

304 changes: 304 additions & 0 deletions scraper/data/jaegerlecoultre.txt

Large diffs are not rendered by default.

300 changes: 300 additions & 0 deletions scraper/data/movado.txt

Large diffs are not rendered by default.

307 changes: 307 additions & 0 deletions scraper/data/omega.txt

Large diffs are not rendered by default.

303 changes: 303 additions & 0 deletions scraper/data/panerai.txt

Large diffs are not rendered by default.

303 changes: 303 additions & 0 deletions scraper/data/patekphilippe.txt

Large diffs are not rendered by default.

305 changes: 305 additions & 0 deletions scraper/data/rolex.txt

Large diffs are not rendered by default.

302 changes: 302 additions & 0 deletions scraper/data/seiko.txt

Large diffs are not rendered by default.

303 changes: 303 additions & 0 deletions scraper/data/zenith.txt

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions scraper/image_downloader.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
require 'csv'
require 'open-uri'

BRANDS = [
'rolex',
'audemarspiguet',
'breitling',
'iwc',
'jaegerlecoultre',
'omega',
'panerai',
'patekphilippe',
'cartier',
'gucci',
'seiko',
'movado',
'zenith'
]

BRANDS.each do |brand|

data = CSV.read("data/#{brand}.txt")

data.each_with_index do |item, index|
open(item[0]) do |image|
File.open("images/#{brand}-#{index+1}-#{item[1]}.jpg", "w+") do |file|
file.write(image.read)
end
end
end

end
60 changes: 60 additions & 0 deletions scraper/scraper.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
require "nokogiri"
require "watir"

BRANDS = [
'rolex',
'audemarspiguet',
'breitling',
'iwc',
'jaegerlecoultre',
'omega',
'panerai',
'patekphilippe',
'cartier',
'gucci',
'seiko',
'movado',
'zenith'
]

browser = Watir::Browser.new(:chrome)

BRANDS.each do |brand|
urls = [
"http://chrono24.com/#{brand}/index.htm",
"http://chrono24.com/#{brand}/index-2.htm",
"http://chrono24.com/#{brand}/index-3.htm",
"http://chrono24.com/#{brand}/index-4.htm",
"http://chrono24.com/#{brand}/index-5.htm"
]

urls.each do |url|
browser.goto(url)
sleep 2
15.times do |i|
browser.execute_script("window.scrollBy(0,500)")
sleep 2
end

doc = Nokogiri::HTML.parse(browser.html)

article_divs = doc.css(".article-item-container")
article_divs.each do |article_div|
image_div = article_div.at_css(".article-image-container .content img")
next if !article_div.at_css(".article-price strong")
price_text = article_div.at_css(".article-price strong").text
next if !image_div || !price_text

image_url = image_div['src']
price = price_text.gsub(/[^0-9]/, "")

next if image_url.empty? || price.empty?

File.open("data/#{brand}.txt", "a+") do |f|
f.puts("#{image_url},#{price}")
end
end

end

end
4 changes: 4 additions & 0 deletions server/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Flask>=1.1.1
fastai>=1.0
torch>=1.4.0
torchvision>=0.5.0
Binary file added server/seiko-monster.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
29 changes: 29 additions & 0 deletions server/server.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
from flask import Flask, request
import fastai.vision as fastai
app = Flask(__name__)

CLASSIFIER = fastai.load_learner("../models", "classifier.pkl")
REGRESSION = fastai.load_learner("../models", "regression.pkl")

@app.route("/classify", methods=["POST", "OPTIONS"])
def classify():
files = request.files
image = fastai.image.open_image(files['image'])
prediction = CLASSIFIER.predict(image)
price_prediction = REGRESSION.predict(image)
return {
"pricePrediction": round(float(price_prediction[1]), 2),
"brandPredictions": sorted(
list(
zip(
CLASSIFIER.data.classes,
[round(x,4) for x in map(float, prediction[2])]
)
),
key=lambda p: p[1],
reverse=True
)
}

if __name__ == "__main__":
app.run(host="0.0.0.0", port=8000, debug=True)
23 changes: 23 additions & 0 deletions ui/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.

# dependencies
/node_modules
/.pnp
.pnp.js

# testing
/coverage

# production
/build

# misc
.DS_Store
.env.local
.env.development.local
.env.test.local
.env.production.local

npm-debug.log*
yarn-debug.log*
yarn-error.log*
1 change: 1 addition & 0 deletions ui/Procfile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
web: node server.js
68 changes: 68 additions & 0 deletions ui/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).

## Available Scripts

In the project directory, you can run:

### `yarn start`

Runs the app in the development mode.<br />
Open [http://localhost:3000](http://localhost:3000) to view it in the browser.

The page will reload if you make edits.<br />
You will also see any lint errors in the console.

### `yarn test`

Launches the test runner in the interactive watch mode.<br />
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.

### `yarn build`

Builds the app for production to the `build` folder.<br />
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.<br />
Your app is ready to be deployed!

See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.

### `yarn eject`

**Note: this is a one-way operation. Once you `eject`, you can’t go back!**

If you aren’t satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.

Instead, it will copy all the configuration files and the transitive dependencies (Webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own.

You don’t have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it.

## Learn More

You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).

To learn React, check out the [React documentation](https://reactjs.org/).

### Code Splitting

This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting

### Analyzing the Bundle Size

This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size

### Making a Progressive Web App

This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app

### Advanced Configuration

This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration

### Deployment

This section has moved here: https://facebook.github.io/create-react-app/docs/deployment

### `yarn build` fails to minify

This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify
36 changes: 36 additions & 0 deletions ui/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"name": "ui",
"version": "0.1.0",
"private": true,
"dependencies": {
"@testing-library/jest-dom": "^4.2.4",
"@testing-library/react": "^9.3.2",
"@testing-library/user-event": "^7.1.2",
"axios": "^0.19.2",
"express": "^4.17.1",
"react": "^16.12.0",
"react-dom": "^16.12.0",
"react-scripts": "3.3.0"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": "react-app"
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
Binary file added ui/public/favicon.ico
Binary file not shown.
43 changes: 43 additions & 0 deletions ui/public/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="theme-color" content="#000000" />
<meta
name="description"
content="Web site created using create-react-app"
/>
<link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>React App</title>
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>
Binary file added ui/public/logo192.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ui/public/logo512.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
25 changes: 25 additions & 0 deletions ui/public/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
"short_name": "React App",
"name": "Create React App Sample",
"icons": [
{
"src": "favicon.ico",
"sizes": "64x64 32x32 24x24 16x16",
"type": "image/x-icon"
},
{
"src": "logo192.png",
"type": "image/png",
"sizes": "192x192"
},
{
"src": "logo512.png",
"type": "image/png",
"sizes": "512x512"
}
],
"start_url": ".",
"display": "standalone",
"theme_color": "#000000",
"background_color": "#ffffff"
}
2 changes: 2 additions & 0 deletions ui/public/robots.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# https://www.robotstxt.org/robotstxt.html
User-agent: *
14 changes: 14 additions & 0 deletions ui/server.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
var express = require('express')
var app = express()

app.set('port', (process.env.PORT || 3000))

app.use(express.static(__dirname + '/build'))

app.get('*', function(request, response) {
response.sendFile(__dirname + '/build/index.html')
})

app.listen(app.get('port'), function() {
console.log("Express server started on port", app.get('port'))
})
45 changes: 45 additions & 0 deletions ui/src/App.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
.file-dropzone {
height: 210px;
width: 210px;
background-color: mistyrose;
border: 2px dashed gray;
}

.App {
text-align: center;
}

.App-logo {
height: 40vmin;
pointer-events: none;
}

@media (prefers-reduced-motion: no-preference) {
.App-logo {
animation: App-logo-spin infinite 20s linear;
}
}

.App-header {
background-color: #282c34;
min-height: 100vh;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
font-size: calc(10px + 2vmin);
color: white;
}

.App-link {
color: #61dafb;
}

@keyframes App-logo-spin {
from {
transform: rotate(0deg);
}
to {
transform: rotate(360deg);
}
}
104 changes: 104 additions & 0 deletions ui/src/App.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
import React from 'react';
import axios from 'axios';
import './App.css';

const API_ENDPOINT = "http://3.227.8.3/"
const API_CLIENT = axios.create({
baseURL: API_ENDPOINT,
timeout: 10000
})

class App extends React.Component {

state = {
predictions: {
pricePrediction: undefined,
brandPredictions: []
},
imgSrc: ""
}

_onDragOver(e) {
e.preventDefault()
}

_onDragLeave(e) {
e.preventDefault()
}

_onDrop(e) {
e.preventDefault()
var targetFile = e.dataTransfer.files[0]
var reader = new FileReader()
reader.readAsDataURL(targetFile)
reader.onloadend = (e) => { this.setState({ imgSrc: reader.result })}
var data = new FormData()
data.append('image', targetFile)
API_CLIENT.post('/classify', data, {headers: {"Content-Type": targetFile.type}})
.then((response) => { this.setState({predictions: response.data}) })
.catch((error) => { console.log(error) })
}

render() {
var ImagePreview
if(this.state.imgSrc) {
ImagePreview = (<img src={this.state.imgSrc} alt="img-of-a-watch" />)
}

var Predictions = []
this.state.predictions.brandPredictions.forEach((item, index) => {
Predictions.push(
<p key={`item-${index}`}>{item[0]}: {item[1]}</p>
)
})

return (
<div className="App">
<div
className='file-dropzone'
onDragOver={(e) => { this._onDragOver(e) }}
onDragLeave={(e) => { this._onDragLeave(e) }}
onDrop={(e) => { this._onDrop(e) }}>
{ImagePreview}
</div>

<div className='predictions'>
{Predictions}
</div>
</div>
)
}
}

export default App;






























// SOME COMMENT
9 changes: 9 additions & 0 deletions ui/src/App.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import React from 'react';
import { render } from '@testing-library/react';
import App from './App';

test('renders learn react link', () => {
const { getByText } = render(<App />);
const linkElement = getByText(/learn react/i);
expect(linkElement).toBeInTheDocument();
});
13 changes: 13 additions & 0 deletions ui/src/index.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}

code {
font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New',
monospace;
}
12 changes: 12 additions & 0 deletions ui/src/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import App from './App';
import * as serviceWorker from './serviceWorker';

ReactDOM.render(<App />, document.getElementById('root'));

// If you want your app to work offline and load faster, you can change
// unregister() to register() below. Note this comes with some pitfalls.
// Learn more about service workers: https://bit.ly/CRA-PWA
serviceWorker.unregister();
7 changes: 7 additions & 0 deletions ui/src/logo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
137 changes: 137 additions & 0 deletions ui/src/serviceWorker.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
// This optional code is used to register a service worker.
// register() is not called by default.

// This lets the app load faster on subsequent visits in production, and gives
// it offline capabilities. However, it also means that developers (and users)
// will only see deployed updates on subsequent visits to a page, after all the
// existing tabs open on the page have been closed, since previously cached
// resources are updated in the background.

// To learn more about the benefits of this model and instructions on how to
// opt-in, read https://bit.ly/CRA-PWA

const isLocalhost = Boolean(
window.location.hostname === 'localhost' ||
// [::1] is the IPv6 localhost address.
window.location.hostname === '[::1]' ||
// 127.0.0.0/8 are considered localhost for IPv4.
window.location.hostname.match(
/^127(?:\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}$/
)
);

export function register(config) {
if (process.env.NODE_ENV === 'production' && 'serviceWorker' in navigator) {
// The URL constructor is available in all browsers that support SW.
const publicUrl = new URL(process.env.PUBLIC_URL, window.location.href);
if (publicUrl.origin !== window.location.origin) {
// Our service worker won't work if PUBLIC_URL is on a different origin
// from what our page is served on. This might happen if a CDN is used to
// serve assets; see https://github.com/facebook/create-react-app/issues/2374
return;
}

window.addEventListener('load', () => {
const swUrl = `${process.env.PUBLIC_URL}/service-worker.js`;

if (isLocalhost) {
// This is running on localhost. Let's check if a service worker still exists or not.
checkValidServiceWorker(swUrl, config);

// Add some additional logging to localhost, pointing developers to the
// service worker/PWA documentation.
navigator.serviceWorker.ready.then(() => {
console.log(
'This web app is being served cache-first by a service ' +
'worker. To learn more, visit https://bit.ly/CRA-PWA'
);
});
} else {
// Is not localhost. Just register service worker
registerValidSW(swUrl, config);
}
});
}
}

function registerValidSW(swUrl, config) {
navigator.serviceWorker
.register(swUrl)
.then(registration => {
registration.onupdatefound = () => {
const installingWorker = registration.installing;
if (installingWorker == null) {
return;
}
installingWorker.onstatechange = () => {
if (installingWorker.state === 'installed') {
if (navigator.serviceWorker.controller) {
// At this point, the updated precached content has been fetched,
// but the previous service worker will still serve the older
// content until all client tabs are closed.
console.log(
'New content is available and will be used when all ' +
'tabs for this page are closed. See https://bit.ly/CRA-PWA.'
);

// Execute callback
if (config && config.onUpdate) {
config.onUpdate(registration);
}
} else {
// At this point, everything has been precached.
// It's the perfect time to display a
// "Content is cached for offline use." message.
console.log('Content is cached for offline use.');

// Execute callback
if (config && config.onSuccess) {
config.onSuccess(registration);
}
}
}
};
};
})
.catch(error => {
console.error('Error during service worker registration:', error);
});
}

function checkValidServiceWorker(swUrl, config) {
// Check if the service worker can be found. If it can't reload the page.
fetch(swUrl, {
headers: { 'Service-Worker': 'script' }
})
.then(response => {
// Ensure service worker exists, and that we really are getting a JS file.
const contentType = response.headers.get('content-type');
if (
response.status === 404 ||
(contentType != null && contentType.indexOf('javascript') === -1)
) {
// No service worker found. Probably a different app. Reload the page.
navigator.serviceWorker.ready.then(registration => {
registration.unregister().then(() => {
window.location.reload();
});
});
} else {
// Service worker found. Proceed as normal.
registerValidSW(swUrl, config);
}
})
.catch(() => {
console.log(
'No internet connection found. App is running in offline mode.'
);
});
}

export function unregister() {
if ('serviceWorker' in navigator) {
navigator.serviceWorker.ready.then(registration => {
registration.unregister();
});
}
}
5 changes: 5 additions & 0 deletions ui/src/setupTests.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom/extend-expect';
10,771 changes: 10,771 additions & 0 deletions ui/yarn.lock

Large diffs are not rendered by default.

0 comments on commit 0c10517

Please sign in to comment.