#Introduction I had just (22 december 2013 01:58 UTC+00) forked https://github.com/oikomi/mygocrawl The commands below assume you have a working Go 1.2 compiler, and you have set the GOPATH enviroment variable.

Go getting it

go get github.com/larsth/mygocrawl

#Getting the repository using Git

Using a bash shell, the commands are (should be):

mkdir -p $GOPATH/src/github.com/larsth

cd $GOPATH/src/github.com/larsth

git clone https://github.com/larsth/mygocrawl.git

Install it

go install github.com/larsth/mygocrawl

Important notes

Compiles with the Go 1.2 compiler. Also I had not done any tests, while the program is running (behaveiour tests, does it crash?, etc.).

A little bit too simple.

It will probably hammering a web server, because there is no time limit betwen each request to a web server. Also, it does not ask a server for a robots.txt file, so it will ask for any file from a web server.

I other words - it is a bad web citizen, so I don't recommend using it unmodified.