Skip to content

Customizing Maxmind IP Geo DB for Internal Networks

Jump to bottom
Jason Trost edited this page Nov 5, 2015 · 3 revisions

This howto walks through creating a customer Maxmind IP geo DB for use with MHN and honeymap so this database can be used to geolocate hosts that reside in private IP space. As time permits, this will be incorporated into MHN as a more fully supported software feature. For now, it is manual.

1. Download and install mmutils:

git clone https://github.com/mteodoro/mmutils.git
cd mmutils
virtualenv env
source env/bin/activate
pip install -r requirements.txt

2. Download the Maxmind CSV files:

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip
wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum2.zip
unzip GeoLiteCity-latest.zip
unzip GeoIPASNum2.zip

3. Use mmutils to convert the GeoLiteCity CSVs into a single flat CSV:

NOTE: your GeoLiteCity_20151006 will likely be named something different since it is based on the date the file was generated.

time python csv2dat.py -l GeoLiteCity_20151006/GeoLiteCity-Location.csv  flat GeoLiteCity_20151006/GeoLiteCity-Blocks.csv > GeoLiteCity-flat.csv

4. Create a CSV file named "mynetworks.csv" of your IP space mapped to its geographic location:

The CSV should have the following columns: startIP,endIP,country,region,city,postalCode,latitude,longitude,metroCode,areaCode.

  • startIP, endIP, Country, City, Latitude and Longitude are required. The others are optional. Missing values should be left blank.
  • startIP and endIP are the integer representation of your IPs. Here is a simple python command to convert IPs to their integer represenation:
$ python -c 'exec("import sys,socket\nfor ip in sys.argv[1:]:\n    print ip, int(socket.inet_aton(ip).encode(\"hex\"),16)\n")' 10.0.0.1 10.255.255.255
10.0.0.1 167772161
10.255.255.255 184549375

You may be able to use the Maxmind DB to find the latitude/longitude of the city by grepping "GeoLiteCity-Location.csv". You can copy all the fields starting with the country code. They map to the following fields: country,region,city,postalCode,latitude,longitude,metroCode,areaCode. Example:

$ grep "Redwood City" GeoLiteCity_20151006/GeoLiteCity-Location.csv
2700,"US","CA","Redwood City","94063",37.4914,-122.2110,807,650
2715,"US","CA","Redwood City","94065",37.5331,-122.2471,807,650
5709,"US","CA","Redwood City","94062",37.4096,-122.2991,807,650
11563,"US","CA","Redwood City","94064",37.4852,-122.2364,807,650
15030,"US","CA","Redwood City","94061",37.4852,-122.2364,807,650
39708,"US","CA","Redwood City","",37.4852,-122.2364,807,650
114927,"US","CA","Redwood City","94068",37.4742,-122.2494,807,650
421543,"US","CA","Redwood City","66666",37.4852,-122.2364,807,666

Example "mynetworks.csv" where 10.0.0.0/8 is mapped to Redwood City:

startIP,endIP,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
167772161,184549375,"US","CA","Redwood City","94063",37.4914,-122.2110,807,650

5. Create a new CSV that is the "GeoLiteCity-flat.csv" and "mynetworks.csv" combined:

cat mynetworks.csv GeoLiteCity-flat.csv > GeoLiteCity-and-mynetworks.csv

Verify that the resulting file looks ok.

$ head GeoLiteCity-and-mynetworks.csv 
startIP,endIP,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
167772161,184549375,"US","CA","Redwood City","94063",37.4914,-122.2110,807,650
16777216,16777471,AU,,,,-27.0000,133.0000,,
16777472,16778239,CN,,,,35.0000,105.0000,,
16778240,16779263,AU,,,,-27.0000,133.0000,,
16779264,16781311,CN,30,Guangzhou,,23.1167,113.2500,,
16781312,16785407,JP,40,Tokyo,100-0001,35.6850,139.7514,,
16785408,16793599,CN,30,Guangzhou,,23.1167,113.2500,,
16793600,16798975,JP,11,Hiroshima,730-0011,34.3963,132.4594,,
16798976,16799231,JP,31,Okayama,700-0824,34.6617,133.9350,,

6. Convert the flat file to Maxmind's binary format:

Note: this command took ~2 min to run for me.

python csv2dat.py -w mmcity.dat mmcity GeoLiteCity-and-mynetworks.csv

7. Verify it works:

$ ipython
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
Python 2.7.10 (default, Jul 14 2015, 19:46:27) 
Type "copyright", "credits" or "license" for more information.

IPython 3.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import pygeoip

In [2]: geo = pygeoip.GeoIP('mmcity.dat')

In [3]: print json.dumps(geo.record_by_addr('10.0.0.1'), indent=4, sort_keys=True)
{
    "area_code": 650, 
    "city": "Redwood City", 
    "continent": "NA", 
    "country_code": "US", 
    "country_code3": "USA", 
    "country_name": "United States", 
    "dma_code": 807, 
    "latitude": 37.4914, 
    "longitude": -122.211, 
    "metro_code": "San Francisco, CA", 
    "postal_code": "94063", 
    "region_code": "CA", 
    "time_zone": "America/Los_Angeles"
}

8. Copy mmcity.dat to /opt/GeoLiteCity.dat:

sudo cp /opt/GeoLiteCity.dat /opt/GeoLiteCity.dat.bak
sudo cp mmcity.dat /opt/GeoLiteCity.dat

9. Restart the services that use this database:

sudo supervisorctl restart honeymap mhn-uwsgi
Clone this wiki locally