Cannot search a directory EVTX files stored on deduplicated volume

[StevenShockley]

I have a bunch of event logs (a couple of terabytes uncompressed) to search through for a specific event. Event logs deduplicate pretty well, so I've stored them on a Server 2019 machine with Windows deduplication turned on for one drive. (Deduplication reduces 1 TB logs down to ~150 GB, which is nice.)

When I run evtxecmd -d D:\path it exits with no results. If I run evtxecmd -f d:\path\file.evtx, it successfully parses the file, so it's something with giving it a directory rather than a file.

In Program.cs, if I remove DirectoryEnumerationOptions.SkipReparsePoints from dirEnumOptions (line 606) it succeeds. IIRC deduplication uses reparse points, so this makes sense. Is there a specific reason to exclude reparse points?

Unrelated, but you may want to output an error/warning if any unexpected options are provided on the command line. Otherwise, typos can drive you nuts.

Thanks!

[EricZimmerman]

-d expects a directory

evtxecmd.exe -d d:\path

you should not be using -d with a single file. thats what -f is for

you do not have to do anything extra with reparse points. thats not the issue.

if you do not want to deal with typos, use kape to automate! there is no way for me to warn about that except to possibly look for any extra/unexpected ones (i dont know how easy this would be) and then report it.

this works fine for me:

what version are you using, and show me your exact command line and any output. --debug may be helpful switch too

[EricZimmerman]

i can only assume that D:\path is NOT an reparse point. if it IS, do not do it that way =)

there are reasons for that under the hood. again, you can automate collection AND processing with KAPE. would be a one liner:

.\kape.exe --tsource c --target eventlogs --tdest C:\Temp\tout --tflush --mdest C:\Temp\mout --module evtxecmd

you will end up with csv file to load into TLE under c:\temp\mout

[StevenShockley]

Thanks for the reply. I probably should be using KAPE to archive the logs in the first place, but I created this process before KAPE existed and I've only just discovered KAPE.

Running 0.6.0.0 on a deduplicated drive:
C:\Temp>EvtxECmd.exe -d D:\LogArchive\2020-01\Logs --debug
EvtxECmd version 0.6.0.0

Author: Eric Zimmerman ([email protected])
https://github.com/EricZimmerman/evtx

Command line: -d D:\LogArchive\2020-01\Logs --debug

Loading maps from 'C:\Temp\Maps'
'Application-Audit-CVE_1.map' is valid. Adding to maps...
[other maps removed]
Maps loaded: 114
Looking for event log files in 'D:\LogArchive\2020-01\Logs'

Clearing map collection since no output specified

Processed 0 files in 0.9570 seconds

Running on the same files but non-deduplicated:
C:\Temp>EvtxECmd.exe -d c:\temp\logs --debug
EvtxECmd version 0.6.0.0

Author: Eric Zimmerman ([email protected])
https://github.com/EricZimmerman/evtx

Command line: -d c:\temp\logs --debug

Loading maps from 'C:\Temp\Maps'
'Application-Audit-CVE_1.map' is valid. Adding to maps...
[other maps removed]
Maps loaded: 114
Looking for event log files in 'c:\temp\logs'

Clearing map collection since no output specified

Processing 'c:\temp\logs\Archive-Security-2020-01-01-04-14-51-849.evtx'...
Chunk count: 2,048, Iterating records...
Processing chunk at offset 0x1000. Events found so far: 0
Record #: 14568819886 (timestamp: 2020-01-01 03:26:00.8045100): Warning! Time just went backwards! Last seen time before change: 2020-01-01 03:26:02.5694770
Processing chunk at offset 0x11000. Events found so far: 84
Processing chunk at offset 0x21000. Events found so far: 151
[etc...]

Deduplication definitely does use reparse points.

What goes wrong if reparse points are enabled?

[EricZimmerman]

show me what is in the D:\LogArchive\2020-01\Logs directory

how does c:\temp\logs differ from D:\LogArchive\2020-01\Logs?

if one works, so should the other, unless D:\LogArchive\2020-01\Logs is a reparse. did you make it a reparse point?