Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add years option to fetchnvd sucommand. Fix README #8

Merged
merged 1 commit into from
Apr 14, 2016
Merged

Add years option to fetchnvd sucommand. Fix README #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 118 additions & 23 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
# go-cve-dictionary

HTTP API server to get CVE information stored in SQLite3 on localhost.
Current data sources are NVD(English) and JVN(Japanese).
This is tool to build a local copy of the NVD (National Vulnerabilities Database) [1]
and the Japanese JVN [2], which contain security vulnerabilities according to their
CVE identifiers [3] including exhaustive information and a risk score. The local
copy is generated in sqlite format, and the tool has a server mode for easy querying.

[1] https://en.wikipedia.org/wiki/National_Vulnerability_Database
[2] https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures
[3] http://jvndb.jvn.jp/apis/termsofuse.html

## Install requirements

Vuls requires the following packages.
go-cve-dictionary requires the following packages.

- sqlite
- git
Expand Down Expand Up @@ -46,18 +52,18 @@ $ sudo chmod 700 /var/log/vuls
$ go get github.com/kotakanbe/go-cve-dictionary
```

Start go-cve-dictionary as server mode.
For the first time, go-cve-dictionary fetches vulnerability data from NVD.
Fetch Vulnerability data from NVD.
It takes about 10 minutes (on AWS).

```bash
$ go-cve-dictionary server
... Fetching ...
$ for i in {2002..2016}; do ./go-cve-dictionary fetchnvd -years $i; done
... snip ...
$ ls -alh cve.sqlite3
-rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3
```

Now we has vulnerbility data, So start as server mode again.
Now we have vulnerability data.
Start go-cve-dictionary as server mode.
```bash
$ go-cve-dictionary server
[Mar 24 15:21:55] INFO Opening DB. datafile: /home/ec2-user/cve.sqlite3
Expand Down Expand Up @@ -142,10 +148,85 @@ $ curl http://127.0.0.1:1323/cves/CVE-2014-0160 | jq "."

```

# Hello Ruby on Rails 4.0.2

```
$ curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"name": "cpe:/a:rubyonrails:ruby_on_rails:4.0.2:-"}' http://localhost:1323/cpes | jq "."
[
{
"ID": 345,
"CreatedAt": "2016-04-10T10:52:26.196610454+09:00",
"UpdatedAt": "2016-04-10T10:52:26.196610454+09:00",
"DeletedAt": null,
"CveInfoID": 0,
"CveID": "CVE-2016-0751",
"Nvd": {
"ID": 345,
"CreatedAt": "2016-04-10T10:52:26.196853826+09:00",
"UpdatedAt": "2016-04-10T10:52:26.196853826+09:00",
"DeletedAt": null,
"CveDetailID": 345,
"Summary": "actionpack/lib/action_dispatch/http/mime_type.rb in Action Pack in Ruby on Rails before 3.2.22.1, 4.0.x and 4.1.x before 4.1.14.1, 4.2.x before 4.2.5.1, and 5.x before 5.0.0.beta1.1 does not properly restrict use of the MIME type cache, which allows remote attackers to cause a denial of service (memory consumption) via a crafted HTTP Accept header.",
"Score": 5,
"AccessVector": "NETWORK",
"AccessComplexity": "LOW",
"Authentication": "NONE",
"ConfidentialityImpact": "NONE",
"IntegrityImpact": "NONE",
"AvailabilityImpact": "PARTIAL",
"Cpes": null,
"References": [
{
"ID": 486,
"CreatedAt": "2016-04-10T10:52:26.217958168+09:00",
"UpdatedAt": "2016-04-10T10:52:26.217958168+09:00",
"DeletedAt": null,
"JvnID": 0,
"NvdID": 345,
"Source": "MLIST",
"Link": "https://groups.google.com/forum/message/raw?msg=ruby-security-ann/9oLY_FCzvoc/5CDXbvpYEgAJ"
},
{
"ID": 487,
"CreatedAt": "2016-04-10T10:52:26.218175571+09:00",
"UpdatedAt": "2016-04-10T10:52:26.218175571+09:00",
"DeletedAt": null,
"JvnID": 0,
"NvdID": 345,
"Source": "MLIST",
"Link": "http://www.openwall.com/lists/oss-security/2016/01/25/9"
}
],
"PublishedDate": "2016-02-15T21:59:05.877-05:00",
"LastModifiedDate": "2016-03-18T21:02:43.817-04:00"
},
"Jvn": {
"ID": 0,
"CreatedAt": "0001-01-01T00:00:00Z",
"UpdatedAt": "0001-01-01T00:00:00Z",
"DeletedAt": null,
"CveDetailID": 0,
"Title": "",
"Summary": "",
"JvnLink": "",
"JvnID": "",
"Score": 0,
"Severity": "",
"Vector": "",
"References": null,
"Cpes": null,
"PublishedDate": "0001-01-01T00:00:00Z",
"LastModifiedDate": "0001-01-01T00:00:00Z"
}
},
... snip ...
]
```

# Usage:

```
./go-cve-dictionary -help
$ go-cve-dictionary -help
Usage: go-cve-dictionary <flags> <subcommand> <subcommand args>

Subcommands:
Expand Down Expand Up @@ -182,17 +263,21 @@ go-cve-dictionary has four subcommands
- server
Start HTTP server

# Usage: Update NVD Data.
# Usage: Fetch NVD Data.

```
$ go-cve-dictionary fetchnvd -h
$ ./go-cve-dictionary fetchnvd -help
fetchnvd:
fetchnvd
[-last2y]
[-years] 2015 2016 ...
[-dbpath=/path/to/cve.sqlite3]
[-debug]
[-debug-sql]

For the first time, run the blow command to fetch data for entire period. (It takes about 10 minutes)
$ for i in {2002..2016}; do go-cve-dictionary fetchnvd -years $i ; done

-dbpath string
/path/to/sqlite3 (default "$PWD/cve.sqlite3")
-debug
Expand All @@ -201,25 +286,35 @@ fetchnvd:
SQL debug mode
-last2y
Refresh NVD data in the last two years.
-years
Refresh NVD data of specific years.
```

- Fetch data of the entire period
- Fetch data in the last two years

```
$ go-cve-dictionary fetchnvd -entire
$ go-cve-dictionary fetchnvd -last2y
```

- Fetch data last 2 years
- Fetch data of specific years

```
$ go-cve-dictionary fetchnvd -last2y
$ go-cve-dictionary fetchnvd -years 2002 2003 2016
```

- Fetch NVD data for entire period.
```
for i in {2002..2016}; do ./go-cve-dictionary fetchnvd -years $i; done

```


----

# Usage: Update JVN Data.
# Usage: Fetch JVN Data.

```
./go-cve-dictionary fetchjvn -h
$ go-cve-dictionary fetchjvn -h
fetchjvn:
fetchjvn
[-dump-path=/path/to/cve.json]
Expand All @@ -231,7 +326,7 @@ fetchjvn:
[-debug-sql]

-dbpath string
/path/to/sqlite3 (default "/Users/kotakanbe/go/src/github.com/kotakanbe/go-cve-dictionary/cve.sqlite3")
/path/to/sqlite3 (default "$PWD/cve.sqlite3")
-debug
debug mode
-debug-sql
Expand Down Expand Up @@ -270,7 +365,7 @@ $ go-cve-dictionary fetchnvd -week
# Usage: Run HTTP Server.

```
./go-cve-dictionary server -h
$ go-cve-dictionary server -h
server:
server
[-bind=127.0.0.1]
Expand All @@ -282,7 +377,7 @@ server:
-bind string
HTTP server bind to IP address (default: loop back interface) (default "127.0.0.1")
-dbpath string
/path/to/sqlite3 (default : /Users/kotakanbe/go/src/github.com/kotakanbe/go-cve-dictionary/cve.sqlite3) (default "/Users/kotakanbe/go/src/github.com/kotakanbe/go-cve-dictionary/cve.sqlite3")
/path/to/sqlite3 (default : $PWD/cve.sqlite3)
-debug
debug mode (default: false)
-debug-sql
Expand All @@ -307,8 +402,8 @@ Use job scheduler like Cron (with -last2y option).

- How to cross compile
```bash
$ cd /path/to/your/local-git-reporsitory/vuls
$ GOOS=linux GOARCH=amd64 go build -o vuls.amd64
$ cd /path/to/your/local-git-reporsitory/go-cve-dictionary
$ GOOS=linux GOARCH=amd64 go build -o cvedict.amd64
```

- Logging
Expand All @@ -330,7 +425,7 @@ Run with --debug, --sql-debug option.

# Authors

kotakanbe ([@kotakanbe](https://twitter.com/kotakanbe)) created vuls and [these fine people](https://github.com/future-architect/vuls/graphs/contributors) have contributed.
kotakanbe ([@kotakanbe](https://twitter.com/kotakanbe)) created go-cve-dictionary and [these fine people](https://github.com/future-architect/go-cve-dictionary/graphs/contributors) have contributed.

----

Expand Down
50 changes: 47 additions & 3 deletions commands/fetchnvd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package commands
import (
"flag"
"os"
"strconv"
"time"

"github.com/google/subcommands"
c "github.com/kotakanbe/go-cve-dictionary/config"
Expand All @@ -18,7 +20,9 @@ type FetchNvdCmd struct {
debugSQL bool

dbpath string

last2Y bool
years bool
}

// Name return subcommand name
Expand All @@ -29,14 +33,17 @@ func (*FetchNvdCmd) Synopsis() string { return "Fetch Vulnerability dictionary f

// Usage return usage
func (*FetchNvdCmd) Usage() string {
//TODO
return `fetchnvd:
fetchnvd
[-last2y]
[-years] 2015 2016 ...
[-dbpath=/path/to/cve.sqlite3]
[-debug]
[-debug-sql]

For the first time, run the blow command to fetch data for entire period. (It takes about 10 minutes)
$ for i in {2002..2016}; do go-cve-dictionary fetchnvd -years $i; done

`
}

Expand All @@ -52,6 +59,9 @@ func (p *FetchNvdCmd) SetFlags(f *flag.FlagSet) {

f.BoolVar(&p.last2Y, "last2y", false,
"Refresh NVD data in the last two years.")

f.BoolVar(&p.years, "years", false,
"Refresh NVD data of specific years.")
}

// Execute execute
Expand All @@ -65,13 +75,47 @@ func (p *FetchNvdCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface
}

c.Conf.DBPath = p.dbpath
c.Conf.FetchNvdLast2Y = p.last2Y

if !c.Conf.Validate() {
return subcommands.ExitUsageError
}

entries, err := nvd.FetchFiles()
years := []int{}
thisYear := time.Now().Year()

switch {
case p.last2Y:
for i := 0; i < 2; i++ {
years = append(years, thisYear-i)
}
case p.years:
if len(f.Args()) == 0 {
log.Errorf("Specify years to fetch (from 2002 to %d)", thisYear)
return subcommands.ExitUsageError
}
for _, arg := range f.Args() {
year, err := strconv.Atoi(arg)
if err != nil || year < 2002 || time.Now().Year() < year {
log.Errorf("Specify years to fetch (from 2002 to %d), arg: %s", thisYear, arg)
return subcommands.ExitUsageError
}
found := false
for _, y := range years {
if y == year {
found = true
break
}
}
if !found {
years = append(years, year)
}
}
default:
log.Errorf("specify -last2y or -years")
return subcommands.ExitUsageError
}

entries, err := nvd.FetchFiles(years)
if err != nil {
log.Error(err)
return subcommands.ExitFailure
Expand Down
22 changes: 5 additions & 17 deletions commands/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ import (
c "github.com/kotakanbe/go-cve-dictionary/config"
"github.com/kotakanbe/go-cve-dictionary/db"
log "github.com/kotakanbe/go-cve-dictionary/log"
"github.com/kotakanbe/go-cve-dictionary/nvd"
server "github.com/kotakanbe/go-cve-dictionary/server"
"golang.org/x/net/context"
)
Expand Down Expand Up @@ -52,7 +51,7 @@ func (p *ServerCmd) SetFlags(f *flag.FlagSet) {

pwd := os.Getenv("PWD")
f.StringVar(&p.dbpath, "dbpath", pwd+"/cve.sqlite3",
fmt.Sprintf("/path/to/sqlite3 (default : %s)", pwd+"/cve.sqlite3"))
fmt.Sprintf("/path/to/sqlite3"))

f.StringVar(&p.bind,
"bind",
Expand Down Expand Up @@ -99,21 +98,10 @@ func (p *ServerCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{}
}

if count == 0 {
log.Info("Fetching vulnerability data from NVD because no NVD data found in DB.")
entries, err := nvd.FetchFiles()
if err != nil {
log.Error(err)
return subcommands.ExitFailure
}

if err := db.InsertNvd(entries); err != nil {
log.Errorf("Failed to insert. dbpath: %s, err: %s",
c.Conf.DBPath, err)
return subcommands.ExitFailure
}

// Exit because fetching NVD data costs huge memory and keeping...
log.Info("Success")
log.Info("No Vulnerability data found. Run the below command to fetch data from NVD")
log.Info("")
log.Info(" for i in {2002..2016}; do go-cve-dictionary fetchnvd -years $i ; done")
log.Info("")
return subcommands.ExitSuccess
}

Expand Down
2 changes: 0 additions & 2 deletions config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ type Config struct {
FetchJvnEntire bool
FetchJvnPeriodChar string

FetchNvdLast2Y bool

Bind string `valid:"ipv4"`
Port string `valid:"port"`

Expand Down
Loading