willhaben real estate analysis

1. extract information (pysider)
2. store results (elasticsearch)
3. analyse (kibana)

How to run

1. start docker containers

   cd willhaben
   docker-compse up
   

2. create and run pyspider project

• go to http://localhost:5000
• click "create" and paste content of pyspider_scripts/willhaben_hauser_kaufen_ooe_uu.py
• on overview page: change status to "RUNNING" and click "Run"

3. check kibana

• go to http://localhost:5601
• import saved setting from docs/assets/kibana_export.json
• check "Discover" or "Visualizations" or "Dashboard"

Stack

docker

docker is used as reproduceable easy-to-use runtime environment.

everything can be started via docker-compose up

pyspider

crawls website and extracts information

• pyspider documentation
• supported databases for usage of undocumented databases search in GitHub e.g. elasticsearch+resultdb

elasticsearch

to store results in a kibana-friendly way

• elasticsearch website

kibana

to browse and analyse results

• kibana website

import configuration via docs/kibana_export.json