Skip to content

Commit

Permalink
Allow domains to be passed in dig-style, and give ZDNS multi-module c…
Browse files Browse the repository at this point in the history 
…apability (#428)

* added dig-style domain passing integration tests

* added ability to pass domains in after command

* able to parse .ini, used hidden Application Options group

* all global options parsed

* able to parse all module types

* working multi-module

* all tests passing

* prevent using a module multiple times in .ini

* added integration tests

* reset zdns exe in integration tests

* lint

* PR cleanup

* update README and fix URL's to be clickable

* README formatting

* Proofreading README

* provide description for MULTIPLE for --help

* Output output from multiple modules in the same output line, similar to Grab (#429)

* only add 'lookup_module' if we're using multiple modules to save storage space

* collect output

* sort imports

* fixed up integration tests to match new schema

* edit todo

* update README.md
  • Loading branch information
@phillip-stephens
phillip-stephens authored Aug 23, 2024
1 parent b31b9f7 commit 9563594
Show file tree
Hide file tree
Showing 16 changed files with 834 additions and 424 deletions.
194 changes: 117 additions & 77 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,27 +31,22 @@ Usage
=====

ZDNS was originally built as a CLI tool only. Work has been done to convert
this into a library with a CLI that calls this library. Currently, the library
has been separated out and a new, separate CLI has been added. Work is ongoing
to clean up the interface between the CLI (or any other client program of the
ZDNS library) and the ZDNS library itself.
this [library](github.com/zmap/zdns/src/zdns) into a standalone library and let a separate [CLI](github.com/zmap/zdns/src/cli) wrap the library.

The ZDNS library lives in `github.com/zmap/zdns/pkg/zdns`. A function there,
`zdns.Run()`, is used to start the ZDNS tool and do the requested lookups.
Currently, this tool is intended to accept a `zdns.GlobalConf` object, `plfag`
flags, and other information, but this interface is undergoing revisions to be
more generally usable and continue to decouple the CLI from the library.

The CLI for this library lives in `github.com/zmap/zdns` under the main
package. Its functionality is described below.
The library consists of a `ResolverConfig` struct which will contain all config options for all lookups made.
The `ResolverConfig` is used to create 1+ `Resolver` struct(s) which will make all lookups. A `Resolver`
should only make a single lookup at a time (it is not thread-safe) and multiple `Resolver` structs should be
used for parallelism. See our [examples](github.com/zmap/zdns/examples) for how to use the
library. [Modules](github.com/zmap/zdns/src/modules) are used to define the behavior of the lookups.

ZDNS provides several types of modules:

- *Raw DNS modules* provide the raw DNS response from the server similar to dig,
but in JSON. There is a module for (nearly) every type of DNS record

- *Lookup modules* provide more helpful responses when multiple queries are
required (e.g., completing additional `A` lookup if a `CNAME` is received)
required (e.g., completing additional `A` lookup for IP addresses if a `NS` is
received in `NSLOOKUP`)

- *Misc modules* provide other additional means of querying servers (e.g.,
`bind.version`)
Expand All @@ -76,40 +71,42 @@ For example, the command:
returns:
```json
{
"name": "censys.io",
"class": "IN",
"status": "NOERROR",
"data": {
"answers": [
{
"ttl": 300,
"type": "A",
"class": "IN",
"name": "censys.io",
"data": "216.239.38.21"
"name": "censys.io",
"results": {
"A": {
"data": {
"additionals": [
{
"flags": "",
"type": "EDNS0",
"udpsize": 512,
"version": 0
}
],
"answers": [
{
"answer": "104.18.10.85",
"class": "IN",
"name": "censys.io",
"ttl": 300,
"type": "A"
},
{
"answer": "104.18.11.85",
"class": "IN",
"name": "censys.io",
"ttl": 300,
"type": "A"
}
],
"protocol": "udp",
"resolver": "[2603:6013:9d00:3302::1]:53"
},
"duration": 0.285295416,
"status": "NOERROR",
"timestamp": "2024-08-23T13:12:43-04:00"
}
],
"additionals": [
{
"ttl": 34563,
"type": "A",
"class": "IN",
"name": "ns-cloud-e1.googledomains.com",
"data": "216.239.32.110"
},
],
"authorities": [
{
"ttl": 53110,
"type": "NS",
"class": "IN",
"name": "censys.io",
"data": "ns-cloud-e1.googledomains.com."
},
],
"protocol": "udp",
"resolver": "30.128.52.190:53"
}
}
}
```

Expand All @@ -119,13 +116,12 @@ Lookup Modules
Raw DNS responses frequently do not provide the data you _want_. For example,
an MX response may not include the associated A records in the additionals
section requiring an additional lookup. To address this gap and provide a
friendlier interface, we also provide several _lookup_ modules: `alookup` and
`mxlookup`.
friendlier interface, we also provide several _lookup_ modules: `alookup`,
`mxlookup`, and `nslookup`.

`mxlookup` will additionally do an A lookup for the IP addresses that
correspond with an exchange record. `alookup` acts similar to nslookup and will
follow CNAME records. `nslookup` will additionally do an A/AAAA lookup for IP
addresses that correspond with an NS record
`alookup` acts similar to nslookup and will follow CNAME records.
`mxlookup` will additionally do an A lookup for the IP addresses that correspond with an exchange record.
`nslookup` will additionally do an A/AAAA lookup for IP addresses that correspond with an NS record

For example,

Expand All @@ -134,32 +130,38 @@ For example,
returns:
```json
{
"name": "censys.io",
"status": "NOERROR",
"data": {
"exchanges": [
{
"name": "aspmx.l.google.com",
"type": "MX",
"class": "IN",
"preference": 1,
"ipv4_addresses": [
"74.125.28.26"
],
"ttl": 288
},
{
"name": "alt1.aspmx.l.google.com",
"type": "MX",
"class": "IN",
"preference": 5,
"ipv4_addresses": [
"64.233.182.26"
],
"ttl": 288
"name": "censys.io",
"results": {
"MXLOOKUP": {
"data": {
"exchanges": [
{
"class": "IN",
"ipv4_addresses": [
"209.85.202.27"
],
"name": "alt1.aspmx.l.google.com",
"preference": 5,
"ttl": 300,
"type": "MX"
},
{
"class": "IN",
"ipv4_addresses": [
"142.250.31.26"
],
"name": "aspmx.l.google.com",
"preference": 1,
"ttl": 300,
"type": "MX"
}
]
},
"duration": 0.154786958,
"status": "NOERROR",
"timestamp": "2024-08-23T13:10:11-04:00"
}
]
}
}
}
```

Expand Down Expand Up @@ -283,6 +285,44 @@ There is a feature available to perform a certain DNS query against all nameserv

```echo "google.com" | ./zdns A --all-nameservers```

Dig-style Domain Input
----------------------
Similiar to dig, zdns can take a domain as input and perform a lookup on it. The only requirement is that the module is
the first argument and domains follow.
For example:

```
./zdns A google.com
```

Multiple Lookup Modules
-----------------------
ZDNS supports using multiple lookup modules in a single invocation. For example, let's say you want to perform an A,
AAAA, and MXLOOKUP for a set of domains and you want to perform them with iterative resolution. You will need to use the
`MULTIPLE` module and provide a config file with the modules and module-specific flags you want to use.

Please see `./zdns --help` and `./zdns <MODULE_NAME> --help` for Global and Module-specific options that can be used in the config file.

For example:

```
cat 1000k_domains.txt | ./zdns MULTIPLE --multi-config-file="./multiple.ini"
```
Where `multiple.ini` is a file that looks like:
```
; Specify Global Options here
[Application Options]
iterative=true
; List out modules and their respective module-specific options here. A module can only be listed once
[MXLOOKUP]
ipv4-lookup = true
; You can use default values and just list modules if you don't need to specify any options
[A]
[AAAA]
```

A sample `multiple.ini` file is provided in [github.com/zmap/zdns/cli/multiple.ini](github.com/zmap/zdns/cli/multiple.ini)

Running ZDNS
------------

Expand Down
10 changes: 8 additions & 2 deletions benchmark/stats.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,14 @@ func updateStats(line string, s *Stats) {
log.Panicf("failed to unmarshal JSON (%s): %v", line, err)
}
domainName := res.Name
resolveTime := time.Duration(res.Duration * float64(time.Second))
// TODO - this will only work for a single module benchmark, we'll need to adjust this if we want to benchmark multi-module
var duration float64
var status zdns.Status
for _, moduleResult := range res.Results {
duration = moduleResult.Duration
status = zdns.Status(moduleResult.Status)
}
resolveTime := time.Duration(duration * float64(time.Second))

if resolveTime < s.MinResolveTime || s.MinResolveTime == 0 {
s.MinResolveTime = resolveTime
Expand Down Expand Up @@ -173,7 +180,6 @@ func updateStats(line string, s *Stats) {
}
s.numberOfResolutions++

status := zdns.Status(res.Status)
if status == zdns.StatusNoError {
s.SuccessfulResolutions++
} else if status == zdns.StatusTimeout || status == zdns.StatusIterTimeout {
Expand Down
Loading

0 comments on commit 9563594

Please sign in to comment.