Allow domains to be passed in dig-style, and give ZDNS multi-module capability #428

phillip-stephens · 2024-08-22T18:02:34Z

Closes #234 and #266

Description

let user pass domains similar to dig
- ./zdns A google.com yahoo.com
let user use multiple modules
- ./zdns MULTIPLE -c ./src/cli/multiple.ini google.com`
add new lookup_module field to zdns.Result to disambiguate what lookup module caused what output line
add integration tests for both the above
add new lookups field in metadata. names will be incremented for every domain name processed, lookups incremented for every domain + lookup module pair processed. Matters when querying with multiple modules at once.
update README with new usage

New `--help` flag

-c, --multi-config-file= config file path for multiple module

`MULTIPLE`'s `--help`

$ ./zdns MULTIPLE -h
Usage:
  zdns [OPTIONS] <command> [DOMAINS]
  zdns [OPTIONS] MULTIPLE

MULTIPLE is a lookup module used from the CLI to use multiple lookup modules at once with the help of a configuration file provided with --multi-config-file/-c. See README.md/Multiple Lookup Modules for more information.

Testing

Normal Usage

$ echo "cloudflare.com" | ./zdns A
{"name":"cloudflare.com","results":{"A":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":512,"version":0}],"answers":[{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":20,"type":"A"},{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":20,"type":"A"}],"protocol":"udp","resolver":"192.168.1.1:53"},"duration":0.03728875,"status":"NOERROR","timestamp":"2024-08-23T14:39:49-04:00"}}}

Dig-style Domains

$  ./zdns A google.com yahoo.com cloudflare.com
{"name":"google.com","results":{"A":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":512,"version":0}],"answers":[{"answer":"142.250.190.110","class":"IN","name":"google.com","ttl":104,"type":"A"}],"protocol":"udp","resolver":"192.168.1.1:53"},"duration":0.042185917,"status":"NOERROR","timestamp":"2024-08-23T14:40:12-04:00"}}}
{"name":"cloudflare.com","results":{"A":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":512,"version":0}],"answers":[{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":119,"type":"A"},{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":119,"type":"A"}],"protocol":"udp","resolver":"192.168.1.1:53"},"duration":0.04169675,"status":"NOERROR","timestamp":"2024-08-23T14:40:12-04:00"}}}
{"name":"yahoo.com","results":{"A":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":512,"version":0}],"answers":[{"answer":"74.6.231.21","class":"IN","name":"yahoo.com","ttl":770,"type":"A"},{"answer":"74.6.143.26","class":"IN","name":"yahoo.com","ttl":770,"type":"A"},{"answer":"98.137.11.164","class":"IN","name":"yahoo.com","ttl":770,"type":"A"},{"answer":"98.137.11.163","class":"IN","name":"yahoo.com","ttl":770,"type":"A"},{"answer":"74.6.143.25","class":"IN","name":"yahoo.com","ttl":770,"type":"A"},{"answer":"74.6.231.20","class":"IN","name":"yahoo.com","ttl":770,"type":"A"}],"protocol":"udp","resolver":"[2603:6013:9d00:3302::1]:53"},"duration":0.058911625,"status":"NOERROR","timestamp":"2024-08-23T14:40:12-04:00"}}}

Multi-Module

; Specify Global Options here
[Application Options]
iterative=true
prefer-ipv6-iteration="true"
; List out modules and their respective module-specific options here. A module can only be listed once
[ALOOKUP]
ipv4-lookup = true
; You can use default values and just list modules if you don't need to specify any options
[A]
[AAAA]
[CNAME]

$ echo "cloudflare.com" | ./zdns MULTIPLE -c ./src/cli/multiple.ini 
{"name":"cloudflare.com","results":{"A":{"data":{"answers":[{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"},{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"}],"protocol":"","resolver":""},"duration":0.0000605,"status":"NOERROR","timestamp":"2024-08-23T14:40:34-04:00"},"AAAA":{"data":{"answers":[{"answer":"2606:4700::6810:85e5","class":"IN","name":"cloudflare.com","ttl":300,"type":"AAAA"},{"answer":"2606:4700::6810:84e5","class":"IN","name":"cloudflare.com","ttl":300,"type":"AAAA"}],"protocol":"udp","resolver":"[2400:cb00:2049:1::a29f:21]:53"},"duration":0.1496895,"status":"NOERROR","timestamp":"2024-08-23T14:40:34-04:00"},"ALOOKUP":{"data":{"ipv4_addresses":["104.16.132.229","104.16.133.229"]},"duration":0.124313875,"status":"NOERROR","timestamp":"2024-08-23T14:40:34-04:00"},"CNAME":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":1232,"version":0}],"authorities":[{"class":"IN","expire":604800,"mbox":"dns.cloudflare.com","min_ttl":300,"name":"cloudflare.com","ns":"ns3.cloudflare.com","refresh":10000,"retry":2400,"serial":2349866729,"ttl":300,"type":"SOA"}],"protocol":"udp","resolver":"[2400:cb00:2049:1::a29f:21]:53"},"duration":0.129645792,"status":"NOERROR","timestamp":"2024-08-23T14:40:34-04:00"}}}

Metadata, new `lookup` field and `ActiveModuleNames`/`ActiveModules` listed at end

Same run as before,

$  echo "cloudflare.com" | ./zdns MULTIPLE -c ./src/cli/multiple.ini --metadata-file="-"
{"name":"cloudflare.com","results":{"A":{"data":{"answers":[{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"},{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"}],"protocol":"","resolver":""},"duration":0.000021541,"status":"NOERROR","timestamp":"2024-08-23T14:40:53-04:00"},"AAAA":{"data":{"answers":[{"answer":"2606:4700::6810:84e5","class":"IN","name":"cloudflare.com","ttl":300,"type":"AAAA"},{"answer":"2606:4700::6810:85e5","class":"IN","name":"cloudflare.com","ttl":300,"type":"AAAA"}],"protocol":"udp","resolver":"[2400:cb00:2049:1::a29f:21]:53"},"duration":0.199524708,"status":"NOERROR","timestamp":"2024-08-23T14:40:53-04:00"},"ALOOKUP":{"data":{"ipv4_addresses":["104.16.132.229","104.16.133.229"]},"duration":0.165888875,"status":"NOERROR","timestamp":"2024-08-23T14:40:53-04:00"},"CNAME":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":1232,"version":0}],"authorities":[{"class":"IN","expire":604800,"mbox":"dns.cloudflare.com","min_ttl":300,"name":"cloudflare.com","ns":"ns3.cloudflare.com","refresh":10000,"retry":2400,"serial":2349866729,"ttl":300,"type":"SOA"}],"protocol":"udp","resolver":"[2400:cb00:2049:1::a29f:21]:53"},"duration":4.177546792,"status":"NOERROR","timestamp":"2024-08-23T14:40:58-04:00"}}}
{"names":1,"lookups":4,"statuses":{"NOERROR":4},"start_time":"2024-08-23T14:40:53-04:00","end_time":"2024-08-23T14:40:58-04:00","name_servers":null,"timeout":15,"retries":1,"conf":{"LookupAllNameServers":false,"CacheSize":10000,"GoMaxProcs":0,"IterationTimeout":4,"IterativeResolution":true,"MaxDepth":10,"NameServerMode":false,"NameServersString":"","UseNanoseconds":false,"DisableFollowCNAMEs":false,"Retries":1,"Threads":1000,"Timeout":15,"Version":false,"IPv4TransportOnly":false,"IPv6TransportOnly":false,"LocalAddrString":"","LocalIfaceString":"","DisableRecycleSockets":false,"PreferIPv4Iteration":false,"PreferIPv6Iteration":true,"TCPOnly":false,"UDPOnly":false,"AlexaFormat":false,"BlacklistFilePath":"","DNSConfigFilePath":"/etc/resolv.conf","MultipleModuleConfigFilePath":"./src/cli/multiple.ini","IncludeInOutput":"","InputFilePath":"-","LogFilePath":"-","MetadataFilePath":"-","MetadataFormat":false,"OutputFilePath":"-","NameOverride":"","NamePrefix":"","ResultVerbosity":"normal","Verbosity":3,"CheckingDisabled":false,"ClassString":"INET","ClientSubnetString":"","Dnssec":false,"UseNSID":false,"OutputGroups":["normal",""],"TimeFormat":"2006-01-02T15:04:05Z07:00","NameServers":null,"Domains":null,"LocalAddrSpecified":false,"LocalAddrs":null,"ClientSubnet":null,"InputHandler":{},"OutputHandler":{},"CLIModule":"MULTIPLE","ActiveModuleNames":["ALOOKUP","A","AAAA","CNAME"],"ActiveModules":{"A":{"IsIterative":true,"LookupAllNameServers":false,"DNSType":1,"DNSClass":1,"Description":""},"AAAA":{"IsIterative":true,"LookupAllNameServers":false,"DNSType":28,"DNSClass":1,"Description":""},"ALOOKUP":{"IPv4Lookup":true,"IPv6Lookup":false},"CNAME":{"IsIterative":true,"LookupAllNameServers":false,"DNSType":5,"DNSClass":1,"Description":""}},"Class":1},"zdns_version":"1.1.0"}%

Multiple domains, dig-style, with `MULTIPLE`

$ ./zdns MULTIPLE -c ./src/cli/multiple.ini --metadata-file="-" google.com yahoo.com 
{"name":"yahoo.com","results":{"A":{"data":{"answers":[{"answer":"74.6.231.20","class":"IN","name":"yahoo.com","ttl":1800,"type":"A"},{"answer":"74.6.231.21","class":"IN","name":"yahoo.com","ttl":1800,"type":"A"},{"answer":"74.6.143.25","class":"IN","name":"yahoo.com","ttl":1800,"type":"A"},{"answer":"98.137.11.164","class":"IN","name":"yahoo.com","ttl":1800,"type":"A"},{"answer":"98.137.11.163","class":"IN","name":"yahoo.com","ttl":1800,"type":"A"},{"answer":"74.6.143.26","class":"IN","name":"yahoo.com","ttl":1800,"type":"A"}],"protocol":"udp","resolver":"[2001:4998:1b0::7961:686f:6f21]:53"},"duration":0.212285875,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"},"AAAA":{"data":{"answers":[{"answer":"2001:4998:44:3507::8000","class":"IN","name":"yahoo.com","ttl":1800,"type":"AAAA"},{"answer":"2001:4998:24:120d::1:1","class":"IN","name":"yahoo.com","ttl":1800,"type":"AAAA"},{"answer":"2001:4998:24:120d::1:0","class":"IN","name":"yahoo.com","ttl":1800,"type":"AAAA"},{"answer":"2001:4998:124:1507::f001","class":"IN","name":"yahoo.com","ttl":1800,"type":"AAAA"},{"answer":"2001:4998:124:1507::f000","class":"IN","name":"yahoo.com","ttl":1800,"type":"AAAA"},{"answer":"2001:4998:44:3507::8001","class":"IN","name":"yahoo.com","ttl":1800,"type":"AAAA"}],"protocol":"udp","resolver":"[2001:4998:1b0::7961:686f:6f21]:53"},"duration":0.131647125,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"},"ALOOKUP":{"data":{"ipv4_addresses":["74.6.231.20","74.6.231.21","74.6.143.25","98.137.11.164","98.137.11.163","74.6.143.26"]},"duration":0.0001275,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"},"CNAME":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":1272,"version":0}],"authorities":[{"class":"IN","expire":1814400,"mbox":"hostmaster.yahoo-inc.com","min_ttl":600,"name":"yahoo.com","ns":"ns1.yahoo.com","refresh":3600,"retry":300,"serial":2024082304,"ttl":600,"type":"SOA"}],"protocol":"udp","resolver":"[2001:4998:1b0::7961:686f:6f21]:53"},"duration":0.108211125,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"}}}
{"name":"google.com","results":{"A":{"data":{"answers":[{"answer":"142.250.190.110","class":"IN","name":"google.com","ttl":300,"type":"A"}],"protocol":"","resolver":""},"duration":0.000064417,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"},"AAAA":{"data":{"answers":[{"answer":"2607:f8b0:4009:80b::200e","class":"IN","name":"google.com","ttl":300,"type":"AAAA"}],"protocol":"udp","resolver":"[2001:4860:4802:34::a]:53"},"duration":0.121261709,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"},"ALOOKUP":{"data":{"ipv4_addresses":["142.250.190.110"]},"duration":0.172249333,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"},"CNAME":{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":512,"version":0}],"authorities":[{"class":"IN","expire":1800,"mbox":"dns-admin.google.com","min_ttl":60,"name":"google.com","ns":"ns1.google.com","refresh":900,"retry":900,"serial":666268564,"ttl":60,"type":"SOA"}],"protocol":"udp","resolver":"[2001:4860:4802:34::a]:53"},"duration":0.173076875,"status":"NOERROR","timestamp":"2024-08-23T14:41:35-04:00"}}}
{"names":2,"lookups":8,...

…to Grab (#429) * only add 'lookup_module' if we're using multiple modules to save storage space * collect output * sort imports * fixed up integration tests to match new schema * edit todo * update README.md

phillip-stephens added 17 commits August 21, 2024 13:06

added dig-style domain passing integration tests

6efedc4

added ability to pass domains in after command

26a2c40

able to parse .ini, used hidden Application Options group

49d1b68

Merge branch 'main' into phillip/dig-domains-and-multi-modules

342e01c

all global options parsed

86ca8d8

able to parse all module types

269cff7

working multi-module

4eaca02

all tests passing

940008b

prevent using a module multiple times in .ini

0c3c4d1

added integration tests

ba7c1aa

reset zdns exe in integration tests

b078165

lint

8b19a1a

PR cleanup

a0cce87

update README and fix URL's to be clickable

aea8812

README formatting

cc62bf9

Proofreading README

1f4abb7

provide description for MULTIPLE for --help

de9e417

phillip-stephens marked this pull request as ready for review August 22, 2024 19:31

phillip-stephens requested a review from a team as a code owner August 22, 2024 19:31

phillip-stephens requested a review from zakird August 22, 2024 19:32

phillip-stephens assigned zakird Aug 22, 2024

Output output from multiple modules in the same output line, similar …

a69ae6c

…to Grab (#429) * only add 'lookup_module' if we're using multiple modules to save storage space * collect output * sort imports * fixed up integration tests to match new schema * edit todo * update README.md

zakird approved these changes Aug 23, 2024

View reviewed changes

zakird merged commit 9563594 into main Aug 23, 2024
3 checks passed

zakird deleted the phillip/dig-domains-and-multi-modules branch August 23, 2024 18:48

zakird mentioned this pull request Aug 23, 2024

ZDNS should accept dig-like input #266

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow domains to be passed in dig-style, and give ZDNS multi-module capability #428

Allow domains to be passed in dig-style, and give ZDNS multi-module capability #428

phillip-stephens commented Aug 22, 2024 •

edited

Loading

Allow domains to be passed in dig-style, and give ZDNS multi-module capability #428

Allow domains to be passed in dig-style, and give ZDNS multi-module capability #428

Conversation

phillip-stephens commented Aug 22, 2024 • edited Loading

Description

New --help flag

MULTIPLE's --help

Testing

Normal Usage

Dig-style Domains

Multi-Module

Metadata, new lookup field and ActiveModuleNames/ActiveModules listed at end

Multiple domains, dig-style, with MULTIPLE

phillip-stephens commented Aug 22, 2024 •

edited

Loading

New `--help` flag

`MULTIPLE`'s `--help`

Metadata, new `lookup` field and `ActiveModuleNames`/`ActiveModules` listed at end

Multiple domains, dig-style, with `MULTIPLE`