MacOS: scripts fail due to use of linux features #103
Assignees
Comments
|
Thanks for the report. for now I'll look at the bash scripts to bring them into compatibility with Mac (and/or to use the coreutils versions when available). I'll take the second suggestion under advisement as well. |
mmguero
added a commit
that referenced
this issue
Jan 21, 2020
…icable (issue #103) A few uses of "grep" and "find" use flags unique to the GNU versions of those tools. As GNU coreutils was already required for a few other utilities I've done the same thing to detect and use ggrep and gfind when needed. As suggested, it might be a good idea to rewrite these to use Python instead to be more portable (although I'll have to take care to make them work with both python 2/3 for various platforms).
|
Commit a0cbe89 modified some of the scripts (see the development branch at https://github.com/idaholab/Malcolm/tree/development/scripts) to use the GNU version of grep when needed. I looked over the other bash scripts there and fixed a few other places where we needed to use GNU utils. Hopefully that ought to get you up and running. Let me know how it goes. I'll consider rewriting the control scripts in python like you suggested, that may make it more platform portable (although I still do have to target both python 2 and 3 as 2 will be hanging around for a while yet). |
|
Thanks for the swift response and patch! I'll try it out when I'm back in the office tomorrow. Great work! |
mmguero
added a commit
that referenced
this issue
Jan 21, 2020
…n python rather than bash for portability (see issue #103). have not as of yet removed the bash versions, as I am still testing these new implementations. for now the auth_setup.sh and malcolm_appliance_packager.sh are still in Bash as well
|
79c39c0 should address the issue with your |
|
In the development branch I've replaced the bash scripts with python scripts that should be compatible on any system with either python 2 or python 3. If you want to try them out you're welcome to pull that branch and give it a shot. |
mmguero
added a commit
that referenced
this issue
Apr 8, 2020
Tons of stuff. * bump moloch to 2.2.0 * reduce log noise * bump version for development to 1.8.2 * set elastalert index settings for a single node * fix issue #97, when using tcpdump the capture files are named .pcap.pcap * check moloch viewer status page periodically for docker container health check * fix docker-compose log verbosity * made kibana_index_refresh.py more robust as suggested by @fabrie in issue #100 use a _find API to get the index ID for a given index name instead of just the saved_objects/index-pattern API Example in test environment: Before adding new fields (dry run): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 465 fields success (dry run only, no write performed) ``` After adding new fields (dry run): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-n', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=True, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 481 fields success (dry run only, no write performed) ``` After adding new fields (update index mapping): ``` /home/user/devel/github/malcolm/kibana/scripts/kibana_index_refresh.py Arguments: ['-v', '-k', 'http://192.168.0.11:5601/kibana'] Arguments: Namespace(debug=True, dryrun=False, index='sessions2-*', url='http://192.168.0.11:5601/kibana') Kibana version is 7.5.1 Index ID for sessions2-* is sessions2-* sessions2-* would have 481 fields success ``` * added plugin for detecting cve-2020-0601 * work on issue #102, log access to Malcolm web interface(s) to Elasticsearch for analysis in Kibana * nginx/php adjustments for issue #101, uploading very large pcap files may fail * fix a few of the control bash scripts to use GNU coreutils where applicable (issue #103) A few uses of "grep" and "find" use flags unique to the GNU versions of those tools. As GNU coreutils was already required for a few other utilities I've done the same thing to detect and use ggrep and gfind when needed. As suggested, it might be a good idea to rewrite these to use Python instead to be more portable (although I'll have to take care to make them work with both python 2/3 for various platforms). * bump malcolm version to 1.9.0, moloch version to 2.2.1 * work on implementing control scripts (start,stop,restart,wipe,logs) in python rather than bash for portability (see issue #103). have not as of yet removed the bash versions, as I am still testing these new implementations. for now the auth_setup.sh and malcolm_appliance_packager.sh are still in Bash as well * have ISO use new scripts * bump elastic to 7.5.2 * update iso build scripts to use new python scripts for install * compatibility fixes for scripts under linux * don't source missing files * more reworking of scripts from bash -> python (not complete yet, may be in a broken state) * more work on auth_setup * remove reference to files we're not longer using * Revert "bump elastic to 7.5.2" This reverts commit 440c859. * fix default for external password question * handle missing python package in windows * documentation updates * tweak some codenames * use specified path (rather than absolute path) for compose file * make python3 the default * Revert "make python3 the default" This reverts commit 52e53f4. * fixes for new control scripts * fix logs script hanging * pin filters by default in kibana * put a hack/fix in for vagrant not liking dhcp nat in 6.1 * create a zeek.service_version field to track protocol version in a single place; also, move password up to the zeek root level * added security overview dashboard wip * added freq.Dockerfile to detecting string entropy * if designated by the FREQ_LOOKUP (true) environment variable, look up DNS query hostnames using freq_server.py * update docker ignore ifle * ask about string freq lookup in install.py * added security overview dashboard wip * added security overview dashboard wip * use a ruby block rather than an http filter in order to better handle arrays * fix volume mapping for local.zeek in docker-compose.yml for testing * fix volume mapping for local.zeek in docker-compose.yml for testing * clean up symlinks as well * initial code for generating and parsing smb_cmd.log * initial code for generating and parsing smb_cmd.log * more work on smb command mapping * more work on smb * more work on smb * bump version to 2.0.0 * some field normalization for 2.0.0 - restored kibana swimlane visualization as it has been fixed for 7.5.x - remove some unused fields from records (agent.ephemeral_id, input.type, path portion of log.file.path) - remove "_jsonparsefailure" tag on cleanup - change some places where we were doing calculations to get count values when we already know the count is "1" - normalization of "action" or "command" values to "zeek.action" field - normalization of mime type, file names, fuids, and service version * fixes to SMB action mapping * remove useless prefix before smb action * exclude some domains from freq. analysis * utility script to repackage zeek logs for upload: * fix issue #111, moloch/etc mount in docker-compose.yml causes custom Zeek fields not to be loaded * fix Malcolm issue #110, submitting hunt job crashes viewer unless Zeek logs are filtered out (temporary patch of fix for Moloch issue 1374, arkime/arkime#1374) * Added smb_cmd fields to WISE * fix dashboard referring to zeek_smb.action -> zeek.action * remove tunnel:: prefix from tunnel type * added 'action' panel to overview * added security overview dashboard (wip) to directory * more work on issue #108, create security overview dashboard in kibana * working on issue #109, create ICS security overview dashboard * added ipv4/ipv6 * working on issue #109, create ICS security overview dashboard * working on issue #109, create ICS security overview dashboard * added network layer to connections * fix max font size * bring sensor local.zeek up to match malcolm's * fixed spacing of navigation menu * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * fix issue #112, region maps not working because of incorrect redirect * comments * ignore logs that have been renamed and are in transit being archived * updates to dashboards * fix kibana_index_refresh.py for python2 * fix non-ics/iot protocols dashboard * bump version to 7.6.0 for elastic * working with es 7.6, but elastalert had to be temporarily disabled. will work on fixing this next * don't include known_certs in outdated/insecure protocols * Tons of work refining dashboards * tweak connections view * improved maps * remove warnings * improvements to how notices can be used througout the other dashboards * improvements to how notices can be used througout the other dashboards * do frequency analysis on zeek_ssl.server_name * merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113) * experimenting with creating a merged network.mac_oui field that looks like this: ... "network": { "type": "ipv4", "mac_oui": { "00:10:db:ff:10:01": "Juniper Networks", "8c:85:90:65:85:8f": "Apple, Inc." } }, ... however, I may revert this for now because although this works kibana does't really play nicely with the data in visualizations * Revert "experimenting with creating a merged network.mac_oui field that looks like this:" This reverts commit 8bdcefa. * Revert "merge src/dst mac/oui fields into network.mac and network.oui arrays, respectively to provide aggregated fields that can be used for asset inventory (issue 113)" This reverts commit ae60cf2. * make installer work better for vms * build virtualbox guest debs in a clean environment * fixed vagrantfile for malcolm build * only keep vmware/virtualbox guest packages in the right environments * increase build memory requirements * fix typo * updating sensor-iso to match malcolm-iso * fix relative path * removed docker-gen in nginx container, we're not using it any more * update software saved search * Fixed installation of elastalert kibana plugin, but still broken due to this issue: bitsensor/elastalert-kibana-plugin#141 * fix issue #104, Upload without trailing slash redirects to incorrect hostname and scheme this fix includes a few things: - modifying the index.html page to prepend "upload/" before relative HREF/SRC references - removing some useless code in the file-upload default nginx config - added the trailing slash to the proxy-pass directive for the upload section of the main nginx proxy - handle "/server/php" as a separate proxy redirect as that's the XHR where the uploads seem to go - remove the unused Moloch upload page * update copyright * update style of upload screen to match the rest of the app * working on network diff code (wip) * work in progress on network time diff, viewer.js not actually used yet * bump moloch to 2.2.2 * work in progress for network diff * network diff work in progress * comments/work in progress * some test files * update zeek to 3.0.2 * fix reference to zeek::af_packet * install zeek::af_packet with zkg * Revert "install zeek::af_packet with zkg" This reverts commit a20fa9b. * added docker files for running moloch regression tests * added vim to test harnest * fix af_packet zeek build * added promotional poster: * switch test harness branch * temporarily pull from https://github.com/mmguero-dev/moloch fork for issue #2 development rather than aol/moloch releases * update elastic to 7.6.1 for security and bug fixes * update moloch to 2.2.3 * update moloch to 2.2.3 * update zeek to 3.0.3 * update psutil to fix security alert GHSA-qfc5-mcwq-26q8 * zeek updated website, fix broken link * restore stuff for generating web documentation * fix URL for relocated MITRE ATTACK BZAR plugin * fix broken links for build of Zeek, MITRE ATT&CK BZAR plugin * fixed duplicate plugin URL in script * update moloch version in docs to 2.2.3 * remove files no longer needed for testing * remove files no longer needed for testing and update moloch version in documentation * should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files. * should fix issue #114. I discovered that even though moloch-capture isn't writing the PCAP files, the pcapDir and maxFileSizeG values still matter for viewer to be able to delete managed pcap files. * proof of concept for a segment mapping form * work in progress on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * more work on the segment mapping ui * apply tooltip for table columns * scroll back and forth to selected item * beautify with icons * basic validation client-side * more work on the segment mapping ui (integration with malcolm scripts on logstash startup) * more work on the segment mapping ui (creation of docker image, integration with malcolm's nginx reverse proxy) * Added new icon to malcolm iso for subnet mapping editor * documentation updates * start logstash under supervisord in order to add a process that will watch for changes to the name matching * more work on name-map-ui, allow uploading of the JSON file so it can be pushed to the docker image volume automatically * map location of host/subnet mapping to correct location under name-map-ui container * integrate upload with name-map-ui * add the ability to signal logstash from the net-map-ui container * clear out previous maps between restarts * add ability to save net-map.json from web ui * basic control for restarting logstash via ui controls * put save/restart confirmations in UI * added import button to name map ui * send save-state post value to restart-logstash.php * update documentation * update documentation * remove unused variable * documentation updates * use fonts-symbola instead of fonts-noto-color-emoji * re-enable swimlane visualization * update elasticsearch to 7.6.2; also, fix issue #119 * use default theme in elastalert kibana editor * update kibana plugin version * add user to vboxsf group for using shared folders * the 'run a separate instance of Zeek locally' use case isn't really a big enough use case to have a whole separate docker-compose file for it; especially with the ISO and live capture methods. * ensure all services have a health check * reduce verbosity of health checks in logs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Affected Versions:
Master & v1.8.1 release
Details:
I notice that some of the scripts in
./scriptsdon't seem compatible with the default MacOS tools. For instance:./scripts/auth_setup.shappears to terminate silently here. If I addset +ebefore the line, or comment out the line, it works. I assume this is due to different handling of errors on Mac, as it looks like the script wants to ignore any error from failing to source the file?./scripts/start.shstarts the containers and then dumps out agrepusage error:Looking at the script, it calls
/scripts/logs.shwhich passes a-Pflag to grep, which does not appear in themanpage forgrepon Mac, but does appear in discussions ofgrepon linux distros.I tried following the MacOS set up guide but that seems to be largely about configuring Docker (which I already have), I also installed
coreutilsandgrepvia homebrew under the assumption that these would be more like the linux/GNU versions. So I have 1 questiona nd 1 suggestion:Question: What do I need to get these script running out of the box on Mac?
Suggestion: Is it worth writing these scripts in a more cross platform language like python rather than bash?
Great job on Malcolm by the way - the only reason I'm raising this issue is because I think the project looks awesome and want to use it 👍
The text was updated successfully, but these errors were encountered: