Merge pull request #448 from zmap/phillip/lower-thread-default

Lowers default thread count from 1000 to 100

README.md

@@ -279,12 +279,20 @@ routines. This architecture has several caveats:
   of CPU cores, you can do so by including the `--go-processes=n` flag or setting
   the `GOMAXPROCS` environment variable.
 
- * It's difficult to recommend a precise amount of `--threads` as it depends on several
-   factors. Empirically, we've found that increasing threads while having a small number of 
-   `--name-servers` can lead to rate limiting and an increase in `TIMEOUT` or `ITERATIVE_TIMEOUT`
-   errors. Even if you're using `--iterative`, you'll still need to query the root/TLD servers
-   for each domain. We recommend investigating with your query parameters to find the number of threads
-   with acceptable performance without too many `TIMEOUT` or `ITERATIVE_TIMEOUT` issues. 
+* It's difficult to recommend a precise amount of `--threads` as it depends on several
+  factors. The graph below shows how a sample workflow has lower runtime but higher rates of name resolution failure as the number of threads increases.
+
+<img src="./docs/threads_vs_runtime.png" alt="threads_vs_runtime" width="700"/>
+
+  Much of the performance that you'll see depends on your workflow, hardware, and how many name servers the load is spread out on. If you're looking to maximize performance
+  for your workflow/hardware, we recommend starting at 100 threads and increasing until you start to see an increase in name resolution failures.
+  To help with this, you can use `--output-file=output.jsonl` and `grep -v "NOERROR" output.jsonl | wc -l` to count the number of names that failed to resolve.
+  Flags that may be of use in tuning performance are:
+
+  * `--timeout` The maximum amount of time ZDNS will spend on a single name
+  * `--iteration-timeout` The maximum amount of time ZDNS will spend on a single iteration step (ex: resolving google.com at the .com layer)
+  * `--retries` The number of retries ZDNS will make against a single nameserver before giving up for that name
+  * `--name-servers` The list of nameservers to use for lookups, mostly useful with `--iterative=false`
 
 
 Output Verbosity

src/cli/cli.go

@@ -52,7 +52,7 @@ type GeneralOptions struct {
 	UseNanoseconds       bool   `long:"nanoseconds" description:"Use nanosecond resolution timestamps in output"`
 	DisableFollowCNAMEs  bool   `long:"no-follow-cnames" description:"do not follow CNAMEs/DNAMEs in the lookup process"`
 	Retries              int    `long:"retries" default:"1" description:"how many times should zdns retry query if timeout or temporary failure"`
-	Threads              int    `short:"t" long:"threads" default:"1000" description:"number of lightweight go threads"`
+	Threads              int    `short:"t" long:"threads" default:"100" description:"number of lightweight go threads"`
 	Timeout              int    `long:"timeout" default:"15" description:"timeout for resolving a individual name, in seconds"`
 	Version              bool   `long:"version" short:"v" description:"Print the version of zdns and exit"`
 }