From 3fcd973a7fa08e2721163d28e5c47d02b918054b Mon Sep 17 00:00:00 2001 From: Anthony Truskinger Date: Fri, 22 Dec 2017 03:03:25 +1000 Subject: [PATCH] Added a ton of documentation Closes #130 This incorporates @towsey's two manual documents into a docs folder that will eventually be a microsite. The documentation is far far far from finished - but it is a good a decent start. Closing #130 because we'll open more specific issues as we go on. --- AnalysisProgramsREADME.md | 194 ------------------------- doc/.vscode/settings.json | 6 + doc/README.MD | 30 ++++ doc/actions.md | 41 ++++++ doc/actions/analyze_long_recording.md | 65 +++++++++ doc/actions/colour_spectrogram.md | 61 ++++++++ doc/actions/concatenate_index_files.md | 50 +++++++ doc/analyses/acoustic_indices.md | 41 ++++++ doc/cli.md | 103 +++++++++++++ doc/code_paths.md | 27 ++++ doc/config_files.md | 26 ++++ doc/developing.md | 0 doc/faq.md | 40 +++++ doc/formats.md | 18 +++ doc/installing.md | 38 +++++ doc/media/README.md | 2 +- doc/supported_platforms.md | 51 +++++++ doc/versioning.md | 37 +++++ doc/workflows.md | 5 + scripts/.gitkeep | 1 + 20 files changed, 641 insertions(+), 195 deletions(-) delete mode 100644 AnalysisProgramsREADME.md create mode 100644 doc/.vscode/settings.json create mode 100644 doc/README.MD create mode 100644 doc/actions.md create mode 100644 doc/actions/analyze_long_recording.md create mode 100644 doc/actions/colour_spectrogram.md create mode 100644 doc/actions/concatenate_index_files.md create mode 100644 doc/analyses/acoustic_indices.md create mode 100644 doc/cli.md create mode 100644 doc/code_paths.md create mode 100644 doc/config_files.md create mode 100644 doc/developing.md create mode 100644 doc/faq.md create mode 100644 doc/formats.md create mode 100644 doc/installing.md create mode 100644 doc/supported_platforms.md create mode 100644 doc/versioning.md create mode 100644 doc/workflows.md create mode 100644 scripts/.gitkeep diff --git a/AnalysisProgramsREADME.md b/AnalysisProgramsREADME.md deleted file mode 100644 index 1dc8d6f51..000000000 --- a/AnalysisProgramsREADME.md +++ /dev/null @@ -1,194 +0,0 @@ -# AnalysisPrograms.exe - -The `AnalysisPrograms.exe` (abbreviated here on in to `AP`) contains many sub programs. -Usage is obtained by running the program without arguments: - - $ AnalysisPrograms.exe - -Which will produce output like: -``` -QUT Bioacoustic Analysis Program - version 16.10.123.456 (DEBUG build, 11/10/2016 11:07 AM) -Git branch-version: master-e2a88694390d39216bfab3a88a77d21f96be2f4a -Copyright QUT 2016 -An action is required to run the program, here are some suggestions: -Usage: AnalysisPrograms options - -Environment variables: - AP_PLAIN_LOGGING [true|false] Enable simpler logging - the default is value is `false` -Global options: - -d -debug [switch] * Do not show the debug prompt AND automatically attach a debugger. Has no effect in RELEASE builds - -n -nodebug [switch] * Do not show the debug prompt or attach a debugger. Has no effect in RELEASE builds - -l -loglevel [logverbosity] * Set the logging. Valid values: None = 0, Error = 1, Warn = 2, Info = 3, Debug = 4, Trace = 5, Verbose = 6, All = 7 - -v -verbose [switch] * Set the logging to be verbose. Equivalent to LogLevel = Debug = 4 - -vv -vverbose [switch] * Set the logging to very verbose. Equivalent to LogLevel = Trace = 4 - -vvv -vvverbose [switch] * Set the logging to very very verbose. Equivalent to LogLevel = ALL = 7 -Actions: - list - Prints the available program actions - << no arguments >> - help - Prints the full help for the program and all actions - EXAMPLE: help spt - will print help for the spt action - - -a -actionname [string] 1 -``` - - -A list of sub-programs can be obtained by running: - - $ AnalysisPrograms.exe list - -And help for a specific action: - - $ AnalysisPrograms.exe help audio2csv - -Importantly, to get a list of the available `IAnalyzer2` analyses (which are used by `audio2csv`), run: - - $ AnalysisPrograms.exe analysesavailable - -## Gotchas - - - **Never** finish a double quoted argument with a backslash (`\`). The parsing rules for such -cases are complicated and outside of our control. See -[here](https://msdn.microsoft.com/en-us/library/system.environment.getcommandlineargs.aspx) for -details. - - You can test arguments with the `echoargs.EXE` program - - If an input argument is an array (e.g. `directoryinfo[]`) any commas in the argument will delimit -the values. For example `"Y:\Results\abc, 123, doo-dah-dee"` will be parsed as -`"Y:\Results\abc"`, `" 123"`, `" doo-dah-dee"`. - -## Version numbering - -Our program uses an automatic version numbering system. A version number such as `16.06.3430.0` can be deciphered as: - -``` -<2-digit-year>.<2-digit-month>.. -``` - -It is important that before building releases that the local repository is clean and up-to-date - otherwise the -build number won't increment. - -The version information is also printed when the program first starts: - -``` -QUT Bioacoustic Analysis Program - version 16.06.123.456 (RELEASE build, 23/06/2016 11:29) -Git branch-version: master-16a11bad5e3c2423bb92386ec83773c700eb4be0 -``` - -In the above output you can see the: -``` - - version ( build, ) -Git branch-version: - -``` - - -## Sub-program types -There are, in broad terms, these types of sub-programs: - - - Main actions - - Process large amounts of information (like `audio2csv`) - - Development / small scale actions - - Small data / development entry points - - `eventrecognizer` which is a generic program for running different recognizers - - Utility actions - - DummyAnalyser - - audiocutter - - Meta actions - - help and documentation usage (`help` & `list`) - - `analysesavailable` - -### IAnalyzer[2] - -`IAnalyzer2` is a pattern code must adhere to in order to be run by `audio2csv`. `audio2csv` is our mass, -parallel, analysis runner that is used to analyze very long files. - -It is common for each analysis type to have **both** a _development sub-program type_ which is used for -testing and an _IAnalyzer_ implementation which is used in production. - -For example, the canetoad recognizer has: - -- Sub-program type: canetoad (for short testing recordings, <2min) - - ``` - $ AnalysisPrograms.exe canetoad ... -> CanetoadOld.Execute -> CanetoadOld.Analysis -> RhinellaMarina.Analysis - ``` -- audio2csv + IAnalyzer: Rhinella.Marina (for very long files, >2min) - - ``` - $ AnalysisPrograms.exe audio2csv ... -c Rhinella.Marina.yml ...-> AnalyseLongRecording.Execute -> RhinellaMarina.Analysis - ``` - - -**Warning:** Our newer event recognizers no longer have their own _sub-program types_. Instead they are run -in devleopment through a generic _sub-program_ named _eventrecognizer_ - -- eventrecognizer + IAnalyzer: Rhinella.Marina (for short testing recordings, <2min) - - ``` - $ AnalysisPrograms.exe eventrecognizer ... -c Rhinella.Marina.yml ... -> RecognizerEntry.Execute -> RecognizerBase.Analysis -> RhinellaMarina.Analysis - ``` - - -## Documentation for specific sub-programs - -### Concatenate Index Files - -This section describes the command line arguments required to concatenate csv data files output by -the index calculation software that works on files shorter than 24 hours. -This software concatenates csv files into blocks of 24 hours only. - - - -To get help: - - $ AnalysisPrograms.exe help concatenateindexfiles - -Output: - -``` -... -Usage: AnalysisPrograms options - -Environment variables: - AP_PLAIN_LOGGING [true|false] Enable simpler logging - the default value is `false` -Global options: - -d -debug [switch] * Do not show the debug prompt AND automatically attach a debugger. Has no effect in RELEASE builds - -n -nodebug [switch] * Do not show the debug prompt or attach a debugger. Has no effect in RELEASE builds - -l -loglevel [logverbosity] * Set the logging. Valid values: None = 0, Error = 1, Warn = 2, Info = 3, Debug = 4, Trace = 5, Verbose = 6, All = 7 - -v -verbose [switch] * Set the logging to be verbose. Equivalent to LogLevel = Debug = 4 - -vv -vverbose [switch] * Set the logging to very verbose. Equivalent to LogLevel = Trace = 4 - -vvv -vvverbose [switch] * Set the logging to very very verbose. Equivalent to LogLevel = ALL = 7 - -q -quietconsole [switch] * Reduce StdOut logging to WARN and ERROR. The default is false. -Action: - concatenateindexfiles - Calls ConcatenateIndexFiles.Execute(): Concatenates multiple consecutive index.csv files. - -input -inputdatadirectories [directoryinfo[]] One or more directories where the original csv files are located. - -inputd -inputdatadirectory [directory] One directory where the original csv files are located. This option exists as well as a hack to get around commas in paths conflicting with PowerArgs' array parsing feature. - -outp -outputdirectory [directory] Directory where the output is to go. - -di -directoryfilter [string] Used to get the required data.csv files, which are assumed to be in a matching dir or subdirectory. E.g. use name of audio file suffix e.g.: *.wav - -f -filestemname [string] File stem name for output files. - -sta -startdate [nullable`1] DateTimeOffset at which concatenation begins. If null, then start with earliest available file. Can parse an ISO8601 date. - -en -enddate [nullable`1] DateTimeOffset at which concatenation ends. If null, then will be set = today's date available file. Can parse an ISO8601 date. - -ti -timespanoffsethint [nullable`1] TimeSpan offset hint required if file names do not contain time zone info. NO DEFAULT IS SET - -in -indexpropertiesconfig [file] User specified file containing a list of indices and their properties. - -fa -falsecolourspectrogramconfig [file] Config file for drawing the false colour spectrograms. - -su -sunrisedatafile [file] User specified file containing times of sunrise & sunset for recording location. Must be format! - -con -concatenateeverythingyoucanlayyourhandson [switch] Set true only when concatenating more than 24-hours of data into one image - e.g. PNG/data. - -eve -eventdatadirectories [directoryinfo[]] One or more directories where the RECOGNIZER event scores are located in csv files. This optional - -even -eventfilepattern [string] Used only to get Event Recognizer files. -``` - -An example command: - - $ AnalysisPrograms.exe concatenateindexfiles -input "Y:\Results\2016Apr14-135123 - Yvonne, Towsey.Acoustic, #93" -output "C:\Temp\concatOutput" -directoryfilter "GympieNP" -filestemname "GympieNP" -timespanoffsethint "10:00:00" - -Other notes: -- `InputDataDirectories` (path): the directory containing all the files of interest - the required files can be in subdirectories of any depth -- `OutputDirectory` (path): the output directory. If it does not exist it will be created. If it exists contents will be overwritten. -- `DirectoryFilter` (string): Typically the recording siteName is used as the filter pattern to select directories. It can also be used for naming the output files -- `FileStemName` (string): the stem name given to the output files -- `StartDate`: A .Net DateTimeOffset object, start dateTime must be in format readable into a DateTimeOffset object, e.g. `2015-10-25T00:00:00+10:00` -- `EndDate`: A .Net DateTimeOffset object, end dateTime must be in format readable into a DateTimeOffset object `2015-10-25T00:00:00+10:00` -- `DrawImages` (boolean): **HARDCODED TO TRUE CURRENTLY** typically will set this true because want to obtain the false-colour spectrograms -- `IndexPropertiesConfig` (path): full path to the IndexPropertiesConfig file - required only if DrawImages = true -- `ConcatenateEverythingYouCanLayYourHandsOn` (bool): **NEVER CHANGE THIS - SHOULD be FALSE by default.** -- `TimeSpanOffsetHint` (timespan): should be set to +1000 hours for Queensland. = TimeSpan.FromHours(10), e.g. `10:00:00` - diff --git a/doc/.vscode/settings.json b/doc/.vscode/settings.json new file mode 100644 index 000000000..98ec960fe --- /dev/null +++ b/doc/.vscode/settings.json @@ -0,0 +1,6 @@ +{ + "cSpell.words": [ + "Ecoacoustics", + "recognizers" + ] +} \ No newline at end of file diff --git a/doc/README.MD b/doc/README.MD new file mode 100644 index 000000000..f3d8442b3 --- /dev/null +++ b/doc/README.MD @@ -0,0 +1,30 @@ +# QUT Ecoacoustics: AnalysisPrograms.exe manual + + +*QUT Ecoacoustics Analysis Programs* is a software package to perform a suite of analyses on audio recordings of the environment. Although the analyses are intended for long-duration recordings (1 – 24 hours approx), in fact they can be performed on any audio file in a format supported by the software. Typically, you would not use the software to analyse recordings shorter than one minute. Currently the software performs: + +- calculation of summary and spectral acoustic indices at variable resolutions +- produces long-duration, false-colour, multi-index spectrograms +- can calculate critical statistics of annotations downloaded from an Acoustic Workbench +- is capable of running various acoustic event recognizers + +All the analyses are performed by a single executable file, _AnalysisPrograms.exe) (henceforth abbreviated to _AP.exe_), with command-line arguments determining what analyses are to be done and on which files. + +## Table of contents + +- [FAQ](./faq) +- [Supported Platforms](./supported_platforms.md) +- [Downloading and Installing](./installing.md) AnalysisPrograms.exe +- [Introduction to running commands](./cli.md) +- [Actions](./actions.md) +- [Config Files](./config_files.md) +- [Supported Audio Formats](./formats.md) +- [List of Analyses](./analyses/) +- [Workflows](./workflows) +- [TODO: Citing the software] + +Supplementary: + +- [Version Numbers](./versioning.md) +- [TODO: Log Files] +- [Code paths](./code_paths) that explain how code is executed \ No newline at end of file diff --git a/doc/actions.md b/doc/actions.md new file mode 100644 index 000000000..442fbdccc --- /dev/null +++ b/doc/actions.md @@ -0,0 +1,41 @@ +# Actions + +## Types + +There are, in broad terms, these types of sub-programs: + +- Main actions + - Process large amounts of information (like `audio2csv`) +- Development / small scale actions + - Small data / development entry points + - `eventrecognizer` which is a generic program for testing different recognizers +- Utility actions + - `DummyAnalyser` - uses CPU for a while and does nothing useful + - `audiocutter` - cuts and converts long audio files up +- Meta actions + - help and documentation usage (`help` & `list`) + - `analysesavailable` + + +## Actions + + +### Analyze Long Recordings + +Processes large audio recording with the specified analysis. Can run recognizers, calculate indices, or do other things, for very long recordings. + +See [details on Analyze Long Recordings](./actions/analyze_long_recordings.md) + +### Colour Spectrogram + +This action produces a single false-colour (FC) spectrogram, taking as input the spectral indices produced by the _Acoustic Indices_ analysis on a single audio file. + +See [details on Colour Spectrogram](./actions/colour_spectrogram.md) + +### Concatenate Index Files + +This action joins together the results of several _Acoustic Indices_ analysis +result sets to produce data and images for 24-hour blocks of data. + +See [details on Concatenate Index Files](./actions/concatenate_index_files.md) + diff --git a/doc/actions/analyze_long_recording.md b/doc/actions/analyze_long_recording.md new file mode 100644 index 000000000..ddc5ae9ba --- /dev/null +++ b/doc/actions/analyze_long_recording.md @@ -0,0 +1,65 @@ +# Analyze Long Recordings (audio2csv) + +Action | audio2csv +Config file | \ + +This action applies an analysis to a long recording. It works well with recordings +that are up 1440 minutes long. + +This action takes a single long-duration audio recording as input, splits it into segments and calculates results and writes the output to several .csv files. +This analysis process supports **parallel** processing. + +Analyze long recordings can run many types of analyses. The provided config file +determines what analyses is run. + +To see a list of all analyses that can be used by _audio2csv_, execute: + +``` +AnalysisPrograms.exe analysesavailable +``` + +As the action name implies, the output of this analysis is a set of CSV files that can contain: + +- recognized acoustic events +- summary indices +- spectral indices + +## Usage + +This section describes the command line arguments required to, for example, calculate acoustic indices derived from a single audio file. + +To run the action, type: + +``` +$ AnalysisPrograms.exe audio2csv +``` + +Here is an example of a command line with abbreviated path names: + +``` +$ AnalysisPrograms.exe audio2csv -input "audioPath\\fileName.wav" -output "outputPath\\directoryName" -config "configPath\\fileName.yml" +``` + +The above three paths are required arguments (the program will return a fatal error if they are not found on the command line). + +A REMINDER: All path names should be double quoted, BUT make sure you use plain double quotes (`".."`) and NOT so called _smart quotes_ (`“..”`). The paths to directories should *never* end in a forward-slash or back-slash. + +## Options + +Typically, analyses will also include optional arguments that take default values if they are not found on the command line. + +Here is more detail about the command line options: + +- `-input`: The path to the audio file to process +- `-output`: The output will be placed the directory provided +- `-config`: + - **IMPORTANT**: The config file chosen will determine the analysis that is run + - The config file contain parameters that unique for the chosen analysis + - If the config file cannot be found, a fatal error will result + - Typically, if a required parameter is not found in the config file, a default value will be used or in certain cases, a fatal error may result. +- `-Channels`: + - The default value is `null` (not specified) which means that all channels are used. + - _AP.exe_ can process files with up to 8 channels. + - [TODO: CHECK THIS] Channels are numbered, 0, 1, 2, … In the usual case of stereo, left channel = 0 and right channel = 1. Note that, although the software should be able to accept recordings where channel number is greater than two, this use has not been debugged and results are undefined. +- `-MixDownToMono`: The default value is `true`. Typically, indices are calculated on the mixed down waveform. +- `-Parallel`: If you have access to a multi-core CPU you can set this option to true. Otherwise, the segments will be cut and analysed in sequence. \ No newline at end of file diff --git a/doc/actions/colour_spectrogram.md b/doc/actions/colour_spectrogram.md new file mode 100644 index 000000000..0b0534f70 --- /dev/null +++ b/doc/actions/colour_spectrogram.md @@ -0,0 +1,61 @@ +# Colour Spectrogram + +Action | colourSpectrogram +Config file | SpectrogramFalseColourConfig.yml +Config file2 | IndexPropertiesConfig.yml + +This action produces a single false-colour (FC) spectrogram, taking as input the spectral indices produced by the _Acoustic Indices_ analysis on a single audio file. + +One FC spectrogram is produced per audio file. To learn more about long-duration, false-colour spectrograms, check out our tutorial at . + +## Usage + +This section describes the command line arguments required to draw a false-colour spectrogram derived from matrices of spectral acoustic indices extracted from a single long-duration audio recording. + +To run the analysis, type: + +``` +$ AnalysisPrograms.exe colourSpectrogram ; +``` + +The output of this analysis is a set of false-colour and greyscale long-duration index spectrograms. + +Here is an example of a command line with abbreviated path names: + +``` +$ AnalysisPrograms.exe colourSpectrogram –inputdatadirectory "path\directoryName1" –outputdirectory "path\directoryName2" - spectrogramconfigpath "path\fileName3.yml" -indexpropertiesconfig "path\fileName4.yml" +``` + +The above four paths are obligatory arguments (the program will return a fatal error if they are not found on the command line). + +## Options + +- `-inputdatadirectory`: Directory where the input matrices of spectral acoustic indices are located. +- `-outputdirectory`: Directory where the output spectrogram images are to go. +- `-spectrogramconfigpath`: Path to the config file specifying other parameters. +- `-indexpropertiesconfig`: Path to the config file containing a list of indices and their display properties, including information about the brightness and contrast of the RGB channels in the FC spectrograms. These values are extremely important in determining the amount of detail that can be seen in FC spectrograms. + +## Config file parameters (SpectrogramFalseColourConfig.yml) + +The colourSpectrogram config file gives you access to various parameters that control drawing of long-duration false-colour spectrograms. Here are the available parameters with default values. + + +Here is some additional information about the more important parameters: + +- `ColorMap1`: "ACI-ENT-EVN". You can experiment with various combinations of indices but we find these to work well. See the tutorial at for more information about the indices to which abbreviations refer. +- `ColorMap2`: "BGN-PMN-R3D" +- `ColourFilter`: This parameter determines the extent to which low index values are emphasized or de-emphasized in their colour representation. Its purpose is either to give emphasis to low intensity features or to de-emphasise them. This parameter applies a function that lies between y=x^-2 and y=x^2, i.e. between the square-root and the square. + - When filterCoeff = 1.0, small values are maximally emphasized, i.e. y=sqrt(x). + - When filterCoeff = 0.0, the matrix remains unchanged, that is, y=x. + - When filterCoeff =-1.0, small values are maximally de-emphasized, i.e. y=x^2. + - Generally, usage suggests that a value of -0.25 is suitable. i.e. a slight de-emphasis. +- `FreqScale`: "Linear". This sets the type of y-axis or Hertz scale. + - Eventual options will be: + - Linear + - Mel + - Linear62Octaves31Nyquist11025 + - Linear125Octaves30Nyquist11025 + - Octaves24Nyquist32000 + - Linear125Octaves28Nyquist32000 + - [TODO: Update] Only "Linear", "Linear125Octaves7Tones28Nyquist32000" work at present. +- `YAxisTicInterval`: 1000. Horizontal grid lines will be placed every 1kHz interval. \ No newline at end of file diff --git a/doc/actions/concatenate_index_files.md b/doc/actions/concatenate_index_files.md new file mode 100644 index 000000000..17a80d64d --- /dev/null +++ b/doc/actions/concatenate_index_files.md @@ -0,0 +1,50 @@ +# Concatenate Index Files + +Action | concatenateIndexfiles +Config file | SpectrogramFalseColourConfig.yml +Config file2 | IndexPropertiesConfig.yml + +This action also produces false-colour spectrograms. However instead of taking the spectral indices from a single audio file, it takes the total output from multiple runs of the *audio2csv* action, and concatenates them to produce one or more concatenated false-colour index spectrograms. + +Typically, this action is used to produce a sequence of one or more 24-hour false-colour spectrograms. 24-hour false-colour spectrograms are easier to interpret because sound-marks, such as the morning chorus, evening chorus, and insect tracks are easier to identify. Typically, environmental recordings are anything from 30 minutes to 24 hours in duration. False-colour index spectrograms shorter than about 3 hours are difficult to interpret due to the lack of soundscape context. + +There is also an option with this action to concatenate the false-colour spectrograms of every audio recording that can be found in a specified directory into one large dataset/image. Due to memory constraints however, one would not usually concatenate more than about 48 hours of recordings. + +It is **strongly** recommended you only run this action on files from a single +acoustic sensor deployment - don't mix recordings from different sites or deployments! + +## Usage + +This section describes the command line arguments required to concatenate the output from multiple runs of the audio2csv action on shorter duration audio recordings, that is less than 24-hours. Typically, the output is a sequence of one or more 24-hour FC spectrograms. + +To run the action, type: + +``` +$ AnalysisPrograms.exe concatenateIndexfiles +``` + +## Options + +Some of the option are obligatory (the program will return fatal error if they are not found on the command line) and some are optional. + +- `-inputdatadirectories`: An array of one or more directories where the original csv files are located. The required files can be in subdirectories to any depth +- `-inputdatadirectory`: A single directory where *all* the original csv files are located. This option exists (in addition to the above) as a hack to get around commas in paths conflicting with PowerArgs' array parsing feature. +- `-outputdirectory`: The directory where the all the output is to go. + - If it does not exist it will be created. If it exists contents will be overwritten. +- `-directoryfilter`: Used as a pattern matcher to collect the required data CSV files, which are assumed to be in a matching directory or subdirectory. We often place the output in directories with same name as the recording file (including the extension). + - Typically the recording siteName is used as the filter pattern to select directories. It can also be used for naming the output files +- `-fileStemName`: User defined file stem name for the output files. +- `-startdate`: A date at which concatenation is to begin. If null, then start with earliest available file. Can parse an ISO8601 date. +- `-enddate`: A date at which concatenation ends. If null, then will be set equal to today's date available file. Can parse an ISO8601 date. +- `-timeSpanOffsetHint`: A TimeSpan offset hint required if file names do not contain time zone offset info. NO DEFAULT IS SET. +- `-indexpropertiesconfig`: User specified file as for the colourSpectrogram action. +- `-falsecolourspectrogramconfig`: Config file for drawing the false colour spectrograms. +- `-concatenateeverythingyoucanlayyourhandson`: Set true only when concatenating more than 24-hours of data into one image + +## Other notes + +- StartDate: A .Net DateTimeOffset object, start dateTime must be in format readable into a DateTimeOffset object, e.g. 2015-10-25T00:00:00+10:00 + +- EndDate: A .Net DateTimeOffset object, end dateTime must be in format readable into a DateTimeOffset object 2015-10-25T00:00:00+10:00 + +- TimeSpanOffsetHint (timespan): should be set to +1000 hours for Queensland. = TimeSpan.FromHours(10), e.g. 10:00:00 diff --git a/doc/analyses/acoustic_indices.md b/doc/analyses/acoustic_indices.md new file mode 100644 index 000000000..21ee03312 --- /dev/null +++ b/doc/analyses/acoustic_indices.md @@ -0,0 +1,41 @@ +# Acoustic Indices + +Action | audio2csv +Config file | Towsey.Acoustic.yaml + +Generates acoustic indices (AKA summary indices) and spectral indices for given audio recording. To learn more about the spectral and summary indices calculated by this action, download “The calculation of acoustic indices derived from long-duration recordings of the natural environment”, Michael Towsey, 2017. at . + + +```yml +IndexCalculationDuration: 60.0 +# BGNoiseNeighbourhood: 5 +ResampleRate: 22050 +FrameLength: 512 +LowFreqBound: 1000 +MidFreqBound: 8000 +HighFreqBound: 11000 +FrequencyScale: Linear +# SaveIntermediateWavFiles: Never +# SaveIntermediateCsvFiles: false +SaveSonogramImages: Never +# SaveSonogramData: false +ParallelProcessing: false +# TileImageOutput: false +# RequireDateInFilename: false +# IndexPropertiesConfig: './IndexPropertiesConfig.yml' +# EventThreshold: 0.2 +``` + +Here is some additional information about the more important parameters: + +- IndexCalculationDuration: Typically, a long duration audio recordings will be divided into 60 second segments and indices calculated for each segment. However, this timespan can be shortened. A minimum value is 0.1 seconds. Note however, that you would not do this for very long-duration recordings. + +- ResampleRate: You should only increase this, from the default of 22050, if you have high sample rate recording and you are interested in acoustic content above 11kHz. Conversely, you would only decrease this value if you have very long recordings (thereby decreasing computation time) and are only interested in low-frequency content. + +- FrameLength: Typically, we find 512 or 1024 to be good values for producing FC spectrograms. However, where the sample rate > 44Khz, frame size can be increased to 2048. + +- LowFreqBound, MidFreqBound, HighFreqBound: Birds calls are typically found in the 1- 8 kHz frequency band. It can be useful to obtain indices for this band and to compare the acoustic energy in the three bands. If your bands of interest are different, you can reset the bounds here. + +- FrequencyScale: Currently the only options are Linear or Octave. + +- SaveSonogramImages: Best to keep the default value of *Never*. However, it may be useful to save spectrograms for debugging purposes. The other (case-sensitive) options are: `[True/Always | WhenEventsDetected]` \ No newline at end of file diff --git a/doc/cli.md b/doc/cli.md new file mode 100644 index 000000000..661edda04 --- /dev/null +++ b/doc/cli.md @@ -0,0 +1,103 @@ +# Introduction To Running Commands + +## You need a shell + +On Windows we recommend using a PowerShell prompt, but CMD will also work. +On Unix systems any terminal, like BaSH will work. + +## Example syntax + +For examples of how to call the executable from the command line, we use the `$` symbol to denote a command line prompt. Do not type the `$` symbol yourself. + +On Windows machines the standard prompt is a `>` character. We use a `$` sign in these examples only for consistency. + +Special mentions: + +- Powershell + - All executables must be prefixed with a `./` +- Unix + - All executions must be prefixed with `mono ` <-- note the space there + +## Your first execution + +In your shell, type the following and press Enter. + +``` +$ AnalysisPrograms.exe +``` + +This will produce several lines of output, including the program version number and instructions on how to set debug levels and verbosity. These options have sensible defaults and you do not need to change them. + +## Seeing what _AP.exe_ can do + +``` +$ AnalysisPrograms.exe list +``` + +Will print a list of the available *actions*. Every action has an action-name by which it is invoked. Some actions perform acoustic analyses. + +When constructing a command line, the first argument after the executable file name must be the action name, which will typically be followed by a list of options for that action: + +``` +$ AnalysisPrograms [options...] +``` + +To obtain help with the options of a particular action, type: + +``` +$ AnalysisPrograms.exe help +``` + +For example: + +``` +$ AnalysisPrograms.exe help audio2csv +``` + +In this command line, ‘help’ is an *action* and the action-name, *audio2csv*, is an option. + +## Options + +The command line options (all prefixed with a hyphen (`-`)), have a short and long form. The short form is always shown to the left of the long form and using either is equivalent. The value for the option should follow the option name, separated by a space. Use double quotes to group values together. + + We use long form options in this manual for clarity. See output from the `help` action for all options. + +## Global options + +Here is a short description of some of global options. As mentioned previously, you can ignore these options until you need them. + +### Console and Log Verbosity + +Verbosity of the logging output can be set by appending the `loglevel` options to the command line: + +Valid verbosity values are: + +- `None` = 0 - show nothing +- `Error` = 1 - show only errors +- `Warn` = 2 - show only warnings +- `Info` = 3 - the standard level +- `Debug` = 4 - print some debug statements that show variable state and exta information +- `Trace` = 5 - print many more debugging statements with detailed variable values +- `Verbose` = 6 - print all stdout and stderr from associated tools +- `All` = 7 - print absolutely everything + +For example: `-l 4`, will give you the `debug` level of verbosity. + +Alternatively, you can append one of the following switches to the command line: + +- `-v` Sets the logging to verbose. Equivalent to LogLevel = `Debug` = 4 +- `-vv` Sets the logging to very verbose. Equivalent to LogLevel = `Trace` = 5 +- `-vvv` Sets the logging to extremely verbose. Equivalent to LogLevel = `All` = 7 + +### Environment variables + +- `AP_PLAIN_LOGGING`: `[true|false]`-- Enables simpler logging and no color output--the default is value is `false` +- [NOT YET IMPLEMENTED] `AP_DISABLE_METRICS` -- if defined will not send performance metrics back to the developers + +## Beware these Syntax Gotchas** + +- **Never** finish a double quoted string argument with a backslash (\\). In particular, do not end directory names like this: “C:\\\\Path\\OutputDirectory\\”. + The parsing rules for such cases are complicated and outside of our control. See [here](https://msdn.microsoft.com/en-us/library/system.environment.getcommandlineargs.aspx) for details. +- You can test arguments on Windows with the `echoargs.EXE` program +- The arguments used are one of the first lines logged in _AP.exe_ log file +- If an input argument is an array (e.g. directoryinfo\[\]), any commas in the argument will delimit the values. For example, "Y:\\Results\\abc, 123, doo-dah-dee" will be parsed as "Y:\\Results\\abc", " 123", " doo-dah-dee". \ No newline at end of file diff --git a/doc/code_paths.md b/doc/code_paths.md new file mode 100644 index 000000000..0d2ae53a0 --- /dev/null +++ b/doc/code_paths.md @@ -0,0 +1,27 @@ +# Code Paths + +[INCOMPLETE] + +![Common Actions](./media/AnalysisFlow.svg) + +**IAnalyzer\[2\]** + +IAnalyzer2 is a pattern code must adhere to in order to be run by audio2csv. audio2csv is our mass, parallel, analysis runner that is used to analyze very long files. + +It is common for each analysis type to have **both** a *development sub-program type* which is used for testing and an *IAnalyzer* implementation which is used in production. + +For example, the canetoad recognizer has: + +- Sub-program type: canetoad (for short testing recordings, <2min) + +- $ AnalysisPrograms.exe canetoad ... -> CanetoadOld.Execute -> CanetoadOld.Analysis -> RhinellaMarina.Analysis + +- audio2csv + IAnalyzer: Rhinella.Marina (for very long files, >2min) + +- $ AnalysisPrograms.exe audio2csv ... -c Rhinella.Marina.yml ...-> AnalyseLongRecording.Execute -> RhinellaMarina.Analysis + +**Note:** All event recognizers are run through a generic *sub-program* named *eventrecognizer* + +- eventrecognizer + IAnalyzer: Rhinella.Marina (for short testing recordings, <2min) + +- $ AnalysisPrograms.exe eventrecognizer ... -c Rhinella.Marina.yml ... -> RecognizerEntry.Execute -> RecognizerBase.Analysis -> RhinellaMarina.Analysis \ No newline at end of file diff --git a/doc/config_files.md b/doc/config_files.md new file mode 100644 index 000000000..ed0b18ab7 --- /dev/null +++ b/doc/config_files.md @@ -0,0 +1,26 @@ +# Config Files + +Most actions require a configuration file which gives you access to various parameters whose values change the outcome of the analysis. + +## Format + +The config file must be in strict YAML format and should have the file extension `.yml`. + +Comments in the config file give further information. All comment lines must start with a hash symbol `#`. Be careful with the syntax. Incorrect syntax can lead to errors that are difficult to trace. Typically, you will only need to adjust a subset of the available parameters. They all have default values. + +You can find an introduction to YAML here: + +You can validate YAML files (to check for syntax errors) here: + +## Location + +All config files are packaged with _AP.exe_ releases. Inside the package you will +find a `ConfigFiles` folder that contains all the config files. + +**IMPORTANT**: Avoid editing these files directly. If you want to change a value: + +1. Copy the file to another directory +1. Rename the file to describe the changes + - e.g. `Towsey.Acoustic.yml` might become `Towsey.Acoustic.HighResolution.yml` +1. Change the values in the file +1. Remember to update the path to the config file when you run _AP.exe_ diff --git a/doc/developing.md b/doc/developing.md new file mode 100644 index 000000000..e69de29bb diff --git a/doc/faq.md b/doc/faq.md new file mode 100644 index 000000000..ace6a5bf2 --- /dev/null +++ b/doc/faq.md @@ -0,0 +1,40 @@ +# FAQ + +## Who is this program designed for? + +Short answer: computers. + +_AP.exe_ is designed primarily to be used by people that have significant computing +expertise. It designed specifically to be used by **other programs** for any real workloads. +It is not designed to be human friendly. + +We encourage anyone to give it a go--don't be daunted by these docs--but keep in +mind the target audience. You're in the right ballpark if: + +- If your workload involves thousands of files +- If you need to use a script just to use _AP.exe_ +- If you have more RAM or CPU than you know what to do with + +More than likely if you're stuck we can help 😊. + +## What is a _binary_? What is an _executable_? What does _compiling_ mean? + +Unlike R, Python, Ruby, or JavaScript, some programming languages can not just be run straight from source code. + +We call such languages _compiled_ programming languages because a special program, called a _compiler_ +is required to transform the text-based programming source code to a low-level set of instructions that a computer understands. + +Some programming languages that need to be compiled include C++, C, Java, and C#. + +This compilation step is discrete and happens before the code is run. Compiling is often referred to as _building_. + +The result of compilation is one or more _binary_ files. We call these files binaries because the code in them +is no longer readable text - rather it is just blobs of binary data. + +Binaries that can be run as programs are often called _executables_. + +## What is an _action_? + +Our program is a monolith--a very large structure. To support doing different +things we have various sub-programs that can be run by themselves. We call each +of these sub-programs an _action_. \ No newline at end of file diff --git a/doc/formats.md b/doc/formats.md new file mode 100644 index 000000000..6d9062568 --- /dev/null +++ b/doc/formats.md @@ -0,0 +1,18 @@ +# Formats + +_AP.exe_ supports: + +- WAVE (`.wav`) + - any sample rate + - integer PCM only + - bit depths of 8, 16, 24, and 32 + - multiple channels +- MP3 (`.mp3`) + - any sample rate + - VBR or Fixed + - **NOTE**: MP3 compression creates artefacts in the reconstituted audio file to which some indices (especially ACI) are highly sensitive. For this reason, we discourage the use of MP3 recordings. +- Ogg Vorbis (`.ogg`) +- FLAC (`.flac`) +- Wave Pack (`.wv`) +- WebM (`.webm`) +- WMA (`.wma`) \ No newline at end of file diff --git a/doc/installing.md b/doc/installing.md new file mode 100644 index 000000000..e72ccf705 --- /dev/null +++ b/doc/installing.md @@ -0,0 +1,38 @@ +# Installing + +[![Build status](https://ci.appveyor.com/api/projects/status/ntf6vtuy5wnlww37/branch/master?svg=true)](https://ci.appveyor.com/project/QUTEcoacousticsResearchGroup/audio-analysis/branch/master) + + +## General + +1. Choose what version of _AP.exe_ that you want + - Stable: Well tested code, used on production servers, usually months old + - Weekly: Automatic build based off the master branch. + - Has many more features and many more bug fixes + - Has many more bugs + - Continuous: Builds that are done by the continuous integration system + - Bleeding edge, always up to date + - Bleeding edge, very likely will break +1. Download a package + - Click on the appropriate link + - [Stable](https://github.com/QutBioacoustics/audio-analysis/releases/latest) + - [Weekly](https://github.com/QutBioacoustics/audio-analysis/releases) + - [Continuous](https://ci.appveyor.com/project/QUTEcoacousticsResearchGroup/audio-analysis/build/artifacts) + - Download the _Release_ version - it will be a ZIP file that starts with _Release_ +1. Extract the contents to a directory on your computer + - It can be installed in any directory + - We typically extract to a directory named `C:\Temp\AP` +1. [Optionally] Add the directory to your system's `PATH` environment variable + - This makes it easier to type _AP.exe_ commands + +## Unix + +The following additional dependencies are required for Unix machines: + +- ffmpeg +- wavpack +- libsox-fmt-all +- sox +- shntool +- mp3splt +- libav-tools diff --git a/doc/media/README.md b/doc/media/README.md index 8f41f062a..608ef2b6e 100644 --- a/doc/media/README.md +++ b/doc/media/README.md @@ -2,5 +2,5 @@ ## Diagrams -The SVG disgrams in this folder were created with the tool. +The SVG diagrams in this folder were created with the tool. You can import them directly into the tool to edit and modify them. diff --git a/doc/supported_platforms.md b/doc/supported_platforms.md new file mode 100644 index 000000000..4c57a1d41 --- /dev/null +++ b/doc/supported_platforms.md @@ -0,0 +1,51 @@ +# Supported Execution Platforms + +## Execution + +### Windows + +The _AP.exe_ binary will run on any Windows computer with v4.6.2 of the .NET Framework installed. +Practically, it means that these variants are supported: + +- Client SKUs + - ✓ 10 Anniversary Update + - + 10 November Update + - + 10 + - + 8.1 + - + 7 +- Server SKUs + - ✓ 2016 + - + 2012 R2 + - + 2012 + - + 2008 R2 SP1 + +The ticks (✓) mean the framework can be installed, the plusses (+) indicate the framework should be installed. +The above was taken from + +### Unix + +To run _AP.exe_ on Unix systems, the Mono runtime needs to be installed. +We require **Mono Version 5.5** or greater to be installed. + +We support the following distributions: + +- Ubuntu 16.04 (used in production) +- Debian 9 (not tested) +- Raspian (not tested) + +To run a .NET Framework binary with Mono, simply prefix the command you would use on Windows with the `mono` keyword. +For example, in Windows: + +```cmd +> AnalysisPrograms.exe list +``` + +In Linux: + +``` +$ mono AnalysisPrograms.exe list +``` + +### MacOS + +Should the same as the Unix requirments, however, we have never run or tested the code on MacOS systems. \ No newline at end of file diff --git a/doc/versioning.md b/doc/versioning.md new file mode 100644 index 000000000..80a3b17a1 --- /dev/null +++ b/doc/versioning.md @@ -0,0 +1,37 @@ +# Versioning + +## Obtaining the version of _AP.exe_ + +The software version number can be obtained by simply typing: + +``` +$ AnalysisPrograms.exe +``` + +The version number is also embedded in the: + +- whenever the program runs +- in the log file every time the program is run +- metadata of the file +- the filename of the release +- the tag of the release on GitHub + +## Interpreting the version + +The output looks like: + +``` + QUT Bioacoustic Analysis Program - version 16.10.123.456 (DEBUG build, 1/10/2016 11:07 AM) + + Git branch-version: master-e2a88694390d39216bfab3a88a77d21f96be2f4a + + Copyright QUT 2016 +``` + +Our program uses an automatic version numbering system. A version number such as 16.06.3430.0 can be deciphered as: + +<2-digit-year>.<2-digit-month>.<build-server-count>.<commits-since-last-release>. + +The Git branch information can be deciphered as: + +<git-branch-when-built>-<latest-commit-hash-when-built> \ No newline at end of file diff --git a/doc/workflows.md b/doc/workflows.md new file mode 100644 index 000000000..4e52e9b59 --- /dev/null +++ b/doc/workflows.md @@ -0,0 +1,5 @@ +# Workflows + +[INCOMPLETE] + +Performing more than one action in sequence. diff --git a/scripts/.gitkeep b/scripts/.gitkeep new file mode 100644 index 000000000..8b1378917 --- /dev/null +++ b/scripts/.gitkeep @@ -0,0 +1 @@ +