Merge pull request osquery#13 in CLOUD/osquery from sync/osquery-4.5.…

…1 to master * commit 'dc2c7d0ef21f7913448c664ae0ebddd034afc977': (43 commits) Changelog 4.5.1 (osquery#6692) Fix incorrect stat return checking within process_events (osquery#6694) tests: Reduce flakiness of test_osqueryi (osquery#6688) Flush stdout with --help (osquery#6693) Enable cppcheck target in macOS builds (osquery#6685) Fix dirPathsAreEqual the documented way (osquery#6690) Add broad exception catching for table execution (osquery#6689) Authenticode table with catalog file info (osquery#6677) Document max interval for scheduled queries (osquery#6683) Fix container overflow in curl_certificate (osquery#6664) Update documentation around build steps (osquery#6681) Incorporate suggested changes on PR 5789 from Directionless Copy-edit and Markdown lint the remaining deployment docs Copy-edit and Markdown lint, clarify section headers for HTTP API doc Copy-edit and Markdown lint, remove old comment about CI Copy-edit and Markdown lint many deployment docs Fixed handling of invalid array bound error with EvtNext function (osquery#6660) Copy-edit and Markdown lint (just nits) remaining developer docs Copy-edit and Markdown lint (just nits) several developer docs Copy-edit and Markdown lint the configuration plugin guidance; minor clarification ...
aikuchin · Nov 19, 2020 · db584cf · db584cf
2 parents a7a150b + dc2c7d0
commit db584cf
Show file tree

Hide file tree

Showing 146 changed files with 1,711 additions and 1,297 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,10 +1,69 @@
+# osquery Changelog
+
+<a name="4.5.1"></a>
+## [4.5.1](https://github.com/osquery/osquery/releases/tag/4.5.1)
+
+[Git Commits](https://github.com/osquery/osquery/compare/4.5.0...4.5.1)
+
+### Under the Hood improvements
+
+- Improve carver tests by faking `postCarve` ([#6659](https://github.com/osquery/osquery/pull/6659))
+- Emit an error during carving, if the `carve` SQL function is disabled ([#6658](https://github.com/osquery/osquery/pull/6658))
+- Update `carves` specs to allow full scan ([#6657](https://github.com/osquery/osquery/pull/6657))
+- Update `carves` table to use JSON ([#6656](https://github.com/osquery/osquery/pull/6656))
+- Improve performance and accuracy of Windows `registry` querying ([#6647](https://github.com/osquery/osquery/pull/6647))
+- Refactor `ephemeral` database plugin into core and simplify tests ([#6648](https://github.com/osquery/osquery/pull/6648))
+
+### Table Changes
+
+- Support for Office MRU (most recently used) entries ([#6587](https://github.com/osquery/osquery/pull/6587))
+- Implement configurable timeout through WHERE clause on `curl_certificate` ([#6641](https://github.com/osquery/osquery/pull/6641))
+- Add `atom_packages` table spec to window ([#6649](https://github.com/osquery/osquery/pull/6649))
+- Add signature information to `authenticode` table on windows ([#6677](https://github.com/osquery/osquery/pull/6677))
+- Add additional AWS regions ([#6666](https://github.com/osquery/osquery/pull/6666))
+
+### Bug Fixes
+
+- Fix container overflow in `curl_certificate` ([#6664](https://github.com/osquery/osquery/pull/6664))
+- Fix handling of invalid array bound error with `EvtNext` function ([#6660](https://github.com/osquery/osquery/pull/6660))
+- Fix `wmi_bios_info` table searching ([#5246](https://github.com/osquery/osquery/pull/5246))
+- Fix `image` column within `drivers` table on Windows ([#6652](https://github.com/osquery/osquery/pull/6652))
+- Fix windows `dirPathsAreEqual` to use the documented way ([#6690](https://github.com/osquery/osquery/pull/6690))
+- Fix incorrect `stat()` return checking within process_events ([#6694](https://github.com/osquery/osquery/pull/6694))
+- Always flush `stdout` when called with `--help` ([#6693](https://github.com/osquery/osquery/pull/6693))
+
+### Documentation
+
+- Document max scheduled query interval ([#6683](https://github.com/osquery/osquery/pull/6683))
+- Update documentation around build steps ([#6681](https://github.com/osquery/osquery/pull/6681))
+- Documentation copy editing
+  ([#6676](https://github.com/osquery/osquery/pull/6676),
+  [#6665](https://github.com/osquery/osquery/pull/6665),
+  [#6662](https://github.com/osquery/osquery/pull/6662))
+- Add 4.5.0 CHANGELOG ([#6646](https://github.com/osquery/osquery/pull/6646))
+- Add 4.5.1 CHANGELOG ([#6692](https://github.com/osquery/osquery/pull/6692))
+
+### Build
+
+- Improve flaky python test handling ([#6654](https://github.com/osquery/osquery/pull/6654))
+- Restore `test_osqueryi` ([#6631](https://github.com/osquery/osquery/pull/6631))
+- Limit `osqueryd` CPU usage to 20% in systemd unit file ([#6644](https://github.com/osquery/osquery/pull/6644))
+- Improve flaky `test_osqueryi` ([#6688](https://github.com/osquery/osquery/pull/6688))
+- Add `cppcheck` support to macOS ([#6685](https://github.com/osquery/osquery/pull/6685))
+
+### Hardening
+
+- Add exception catching for table execution ([#6689](https://github.com/osquery/osquery/pull/6689))
+
 <a name="4.5.0"></a>
 ## [4.5.0](https://github.com/osquery/osquery/releases/tag/4.5.0)
 
 [Git Commits](https://github.com/osquery/osquery/compare/4.4.0...4.5.0)
 
-We would like to thank all of the contributors working on bootstrapping the ARM64/AARCH64 support and Windows 32bit support.
-Additionally, we want to thank those working on Unicode support and all the bug fixes, documentation improvements, and new features.
+We would like to thank all of the contributors working on
+bootstrapping the ARM64/AARCH64 support and Windows 32bit support.
+Additionally, we want to thank those working on Unicode support and
+all the bug fixes, documentation improvements, and new features.
 Thank you! :clap:
 
 ### New Features
@@ -263,20 +322,24 @@ Thank you! :clap:
 [Git Commits](https://github.com/osquery/osquery/compare/4.1.2...4.2.0)
 
 ### New Features / Under the Hood improvements
+
 - TLS Testing infrastructure has been overhauled ([#6170](https://github.com/osquery/osquery/pull/6170))
 - Boost regex has been replaced with std ([#6236](https://github.com/osquery/osquery/pull/6236))
 - `community_id_v1` added as a SQL function ([#6211](https://github.com/osquery/osquery/pull/6211))
 
 ### Build
+
 - Fix format checking on Windows ([#6188](https://github.com/osquery/osquery/pull/6188))
 - Fix format folder exclusions for build checks ([#6201](https://github.com/osquery/osquery/pull/6201))
 - Fix the linking for extensions in build ([#6219](https://github.com/osquery/osquery/pull/6219))
 - Fix build to include windows optional features table ([#6207](https://github.com/osquery/osquery/pull/6207))
 
 ### Security Issues
+
 - [CVE-2020-1887] osquery does not properly verify the SNI hostname ([#6197](https://github.com/osquery/osquery/pull/6197))
 
 ### Bug Fixes
+
 - Carver no longer returns empty carves for hidden files ([#6183](https://github.com/osquery/osquery/pull/6183))
 - Address a race in the Dispatcher logic ([#6145](https://github.com/osquery/osquery/pull/6145))
 - Fix validation in 'last' table ([#6147](https://github.com/osquery/osquery/pull/6147))
@@ -288,6 +351,7 @@ Thank you! :clap:
 - Fix heap buffer overflow in callDoubleFunc and powerFunc ([#6225](https://github.com/osquery/osquery/pull/6225))
 
 ### Table Changes
+
 - Added table `firefox_addons` to All Platforms ([#6200](https://github.com/osquery/osquery/pull/6200))
 - Added table `ssh_configs` to All Platforms ([#6161](https://github.com/osquery/osquery/pull/6161))
 - Added table `user_ssh_keys` to All Platforms ([#6161](https://github.com/osquery/osquery/pull/6161))
@@ -452,7 +516,6 @@ It features a heavily reworked build system. This aims to provide flexibility an
 - macOS query pack: detect SearchAwesome malware ([#5713](https://github.com/osquery/osquery/pull/5713))
 - macOS query pack: detect when a process is tapping keyboard event ([#5345](https://github.com/osquery/osquery/pull/5345))
 
-
 ### Build
 
 - Refactor CMake build ([#5604](https://github.com/osquery/osquery/pull/5604), [#5627](https://github.com/osquery/osquery/pull/5627), [#5630](https://github.com/osquery/osquery/pull/5630), ([#5618](https://github.com/osquery/osquery/pull/5618)), ([#5619](https://github.com/osquery/osquery/pull/5619)))
@@ -463,7 +526,6 @@ It features a heavily reworked build system. This aims to provide flexibility an
 - Update MSI package to install to `Program Files` on Windows ([#5579](https://github.com/osquery/osquery/pull/5579))
 - Linux custom toolchain integration ([#5759](https://github.com/osquery/osquery/pull/5759))
 
-
 ### Hardening
 
 - Link binaries with Full RELRO on Linux ([#5748](https://github.com/osquery/osquery/pull/5748))
@@ -521,16 +583,16 @@ It features a heavily reworked build system. This aims to provide flexibility an
 - Added table `running_apps` on macOS ([#5216](https://github.com/osquery/osquery/pull/5216))
 - Added table `atom_packages` on macOS and Linux ([6d159d40](https://github.com/osquery/osquery/commit/6d159d40))
 - Remove EC2 tables on Windows ([#5657](https://github.com/osquery/osquery/pull/5657))
-- Added column `win_timestamp` to `time` table on Windows ([3bbe6c51](https://github.com/osquery/osquery/commit/3bbe6c51))
-- Added column `is_hidded` to `users` and `groups` table on macOS ([#5368](https://github.com/osquery/osquery/pull/5368))
-- Added column `profile` to `chrome_extensions` table ([#5213](https://github.com/osquery/osquery/pull/5213))
-- Added column `epoch` to `rpm_packages` table on Linux ([#5248](https://github.com/osquery/osquery/pull/5248))
-- Added column `sid` to `logged_in_users` table on Windows ([#5454](https://github.com/osquery/osquery/pull/5454))
-- Added column `registry_hive` to `logged_in_users` table on Windows ([#5454](https://github.com/osquery/osquery/pull/5454))
-- Added column `sid` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
-- Added column `store_location` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
-- Added column `store` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
-- Added column `username` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
-- Added column `store_id` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
-- Added column `product_version`  to `file` table on Windows ([#5431](https://github.com/osquery/osquery/pull/5431))
-- Added column `source` to `sudoers` table on POSIX systems ([#5350](https://github.com/osquery/osquery/pull/5350))
+- Add column `win_timestamp` to `time` table on Windows ([3bbe6c51](https://github.com/osquery/osquery/commit/3bbe6c51))
+- Add column `is_hidden` to `users` and `groups` table on macOS ([#5368](https://github.com/osquery/osquery/pull/5368))
+- Add column `profile` to `chrome_extensions` table ([#5213](https://github.com/osquery/osquery/pull/5213))
+- Add column `epoch` to `rpm_packages` table on Linux ([#5248](https://github.com/osquery/osquery/pull/5248))
+- Add column `sid` to `logged_in_users` table on Windows ([#5454](https://github.com/osquery/osquery/pull/5454))
+- Add column `registry_hive` to `logged_in_users` table on Windows ([#5454](https://github.com/osquery/osquery/pull/5454))
+- Add column `sid` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
+- Add column `store_location` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
+- Add column `store` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
+- Add column `username` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
+- Add column `store_id` to `certificates` table on Windows ([#5631](https://github.com/osquery/osquery/pull/5631))
+- Add column `product_version`  to `file` table on Windows ([#5431](https://github.com/osquery/osquery/pull/5431))
+- Add column `source` to `sudoers` table on POSIX systems ([#5350](https://github.com/osquery/osquery/pull/5350))
diff --git a/cmake/flags.cmake b/cmake/flags.cmake
@@ -270,6 +270,7 @@ function(setupBuildFlags)
       wevtapi.lib
       shell32.lib
       gdi32.lib
+      mswsock.lib
     )
 
     set(osquery_windows_common_defines

diff --git a/cmake/modules/Findcppcheck.cmake b/cmake/modules/Findcppcheck.cmake
@@ -19,7 +19,12 @@ function(FindCppcheck)
         PATHS "${OSQUERY_TOOLCHAIN_SYSROOT}"
       )
     endif()
-
+  elseif("${CMAKE_SYSTEM_NAME}" STREQUAL "Darwin")
+    set(executable_name "cppcheck")
+
+    set(optional_path_suffix_list 
+      PATH_SUFFIXES usr/local/bin
+    )
   else()
     set(executable_name "cppcheck.exe")
   endif()

diff --git a/docs/wiki/deployment/anomaly-detection.md b/docs/wiki/deployment/anomaly-detection.md
@@ -1,6 +1,8 @@
+# Anomaly detection with osquery
+
 An osquery deployment can help you establish an infrastructural baseline, allowing you to detect malicious activity using scheduled queries.
 
-This approach will help you catch known malware ([WireLurker](https://bits.blogs.nytimes.com/2014/11/05/malicious-software-campaign-targets-apple-users-in-china/), IceFog, Imuler, etc.), and more importantly, unknown malware. Let's look at MacOS startup items for a given laptop using [osqueryi](../introduction/using-osqueryi.md):
+This approach will help you catch known malware ([WireLurker](https://bits.blogs.nytimes.com/2014/11/05/malicious-software-campaign-targets-apple-users-in-china/), IceFog, Imuler, etc.), and more importantly, unknown malware. Let's look at macOS startup items for a given laptop using [osqueryi](../introduction/using-osqueryi.md):
 
 ```sh
 $ osqueryi

diff --git a/docs/wiki/deployment/aws-logging.md b/docs/wiki/deployment/aws-logging.md
@@ -1,4 +1,6 @@
-As of version 1.7.4, osquery can log results directly to Amazon AWS [Kinesis Streams](https://aws.amazon.com/kinesis/streams/) and [Kinesis Firehose](https://aws.amazon.com/kinesis/firehose/). For users of these services, `osqueryd` can eliminate the need for a separate log forwarding daemon running in your deployments.
+# Logging osquery to AWS
+
+As of osquery version 1.7.4, osquery can log results directly to Amazon AWS [Kinesis Streams](https://aws.amazon.com/kinesis/streams/) and [Kinesis Firehose](https://aws.amazon.com/kinesis/firehose/). For users of these services, `osqueryd` can eliminate the need for a separate log forwarding daemon running in your deployments.
 
 ## Configuration
 
@@ -31,20 +33,21 @@ When working with AWS, osquery will look for credentials and region configuratio
 4. `default` profile in the AWS config files
 5. Profile from the EC2 Instance Metadata Service
 
-All of the STS configuration flags are optional.  However, if `aws_sts_arn_role` is set, you can utilize temporary credentials via assume role with the [AWS Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+All of the STS configuration flags are optional. However, if `aws_sts_arn_role` is set, you can utilize temporary credentials via assume role with the [AWS Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
 
 ### Kinesis Streams
 
 When logging to Kinesis Streams, the stream name must be specified with `aws_kinesis_stream`, and the log flushing period can be configured with `aws_kinesis_period`.  
 
-Setting aws_kinesis_random_partition_key to true will use random partition keys when sending data to Kinesis. Using random values will load balance over stream shards if you are using multiple shards in a stream.  Note that using this setting will result in the logs of each host distributed across shards, so do not use it if you need logs from each host to be processed by a consistent shard.  The default for this setting is "false".
+Setting `aws_kinesis_random_partition_key` to `true` will use random partition keys when sending data to Kinesis. Using random values will load balance over stream shards if you are using multiple shards in a stream. Note that using this setting will result in the logs of each host distributed across shards, so do not use it if you need logs from each host to be processed by a consistent shard. The default for this setting is `false`.
 
 ### Kinesis Firehose
 
 Similarly for Kinesis Firehose delivery streams, the stream name must be specified with `aws_firehose_stream`, and the period can be configured with `aws_firehose_period`.
 
 ### Sample Config File
-```
+
+```JSON
 {
   "options": {
     "host_identifier": "hostname",
@@ -66,4 +69,4 @@ Similarly for Kinesis Firehose delivery streams, the stream name must be specifi
 }
 ```
 
-**Note**: Kinesis services have a maximum 1MB record size. Result logs bigger than this will not be forwarded by **osqueryd** as they will be rejected by the Kinesis services.
+**Note**: Kinesis services have a maximum 1MB record size. Result logs bigger than this will not be forwarded by `osqueryd` as they will be rejected by the Kinesis services.