All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Types of changes
Addedfor new features.Changedfor changes in existing functionality.Deprecatedfor soon-to-be removed features.Removedfor now removed features.Fixedfor any bug fixes.Securityin case of vulnerabilities.
Addedmaskpartitionsto handle fields containing different types of values by applying distinct transformationsAddedmasksegmentsto allow transformations on specific parts of a field's value using regular expressions to capture subgroups
Fixedmock command ignores global seed flagFixedmissing flagbuffer-sizeinmock,xmlandplaycommandsFixedmissing flagload-cacheinmockcommandFixedremove unused flags frommock,xml,parquet,jsonschema,flowandplaycommands
Addedmaskapplyto externalize masksFixedmock command fails to parse JSON body when root is an array
Fixedmock command fails when log level is not TRACE
Addednew action for parquet files (experimental feature)Addedmock command to intercept HTTP requests/responses to a web service and apply maskingsAddedtime in JSON logs generated by--log-jsonflag
Addedparametermaxstrlentosha3andhashInCSVmasksFixedtype casting issues when usingadd,add-transient,choice,weightedChoiceorhashto create complex structuresFixedperformance issue withpipemaskFixedperformance issue with selectorsFixedURI resources will be loaded in-memory only once instead of once per maskFixedtimelineshould not overwrite existing data
Fixedregression in pimo play WASM (web version)
Fixedperformance issues on JSON serialization
Addedpropertypreserve-listto masking definition to be able to ignore masking specific values
Addedmasksha3to generate collision resistant identifiersAddedoptionidentifierto maskhashInCSVto include a collision resistant identifier in the result
Fixedtemplate functionMaskSequencedoes not increment correctly
Addedmaskff1preserve specific domainAddedmaskff1log field value as WARN when an error is catched byonErrorAddedmasksequenceto generate sequenced ID of any format
Addedmasktimelineto generate timelines from rules and constraints between dates
Fixedload data for unique cache
Fixedmemory leak with context logs
Added--pprofflag to create acpuormemorypprof fileAddednew featuresxmlmask to to manipulate XML content within JSON values - the proposed syntax aims to align with existing masking conventions for ease of use
Addednew featuresfindInCSVmask to get one or multiple csv lines which matched with Json entry value from CSV filesAddednew flag--serveto start a HTTP server that apply masking to JSON body of requestsAddednew flag--buffer-sizeto increase the buffer used to load files from URIFixedsome bugs with selectors
Addednew features for parsing and masking XML files using the following command:cat XMLfile | pimo xml --subscriber <parent tag name>=<mask name> > outputXMLfile- this feature supports all level 1 elements that are not arrays
Addednew features for ff1 mask :domain,preserveandonErrorAddedpossibility to use.in the jsonpath field of selectorsAddedoption--skip-log-file <filename>to log skipped lines on error when using--skip-line-on-error
Fixedlocal seed parameter should be consistent between different jsonpathFixedPIMO Play handle panics and recover gracefully when panics occure
AddedrandomChoiceInCSVandhashInCSVmask to get values from CSV filesFixedSeedoption on randomized mask use global configuredseedand the hash of the valueFixed--seedflag allow the-1value
AddedEnvironment variablePIMO_STATS_TEMPLATEto control the format of generated statsAddedEnvironment variablePIMO_STATS_URLto generate a stat file or HTTP POST
Addedflag to declare seed in command lineAddedflag--statsto generate a stat file or HTTP POSTAddedflag--statsTemplateto control the format of generated statsFixedcache now apply once on whole masking item, instead of each mask in the block
Securityin PIMO Play : disallow usage of mask Command inmaskspropertyAddedfunctions section in masking configurationAddedpossibility to use masks as template functions (most mask are available asMask<mask_name>: MaskRandDate, MaskRegex, ...)Addedin PIMO Play: adds a tab to visualize a mermaid graph flow of the maskingFixedpreserve notInCache enum in the jsonschemaFixedfunctions call from a pipe mask crash pipeline
Fixedclick on examples does not reload webpage on the PIMO Play websiteFixedverbosity in pimo play command
Addedpossibility to share URL directly from the PIMO Play websiteAddedprecrafted examples in the PIMO Play websiteAddedloading screen when JS is loading on PIMO Play websiteAddedloading spinner when refreshing output on PIMO Play websiteAddeda refresh button on PIMO Play websiteFixedjsonschema forrandomDecimalandrandomDurationmasks
Addedseed option on all randomized masks, another way to preserve data coherenceAddedtranscode maskAddedplay command
Fixeduse fromcache after a mask which causes a change in the type of the value
Addedmarkov mask to generate pseudo text based on a sample textAddedcommand to export maskings as a mermaid flow chartAddedpossibility to use optionpreserve: notInCachewith maskfromCacheAddedflag to mask input while a declared condition is metAddedflag to mask input until a declared condition is metAddedflag to declare a simple mask definition in command lineAddedpossibility to declare multiple selectors for a masking definitionFixedpimo doesn't panic anymore with fromjson combined with pipe mask, or fromjson combined with another fromjson mask with nested selectorsFixedmaskreplacementwith nested selectorsFixedusing fromCache, the line is not deleted if the jsonpath in not present in the documentAddedoption incachessection for use reverse cacheAddedunixEpoch format with maskdateParser
Addedoption preserve in masking configuration.Fixedcache with maskfluxUri.
Addedluhn mask to generate valid checksums using the Luhn algorithm.
FixedJSON Schema validation formasksproperty (only required ifmaskis not set).
Addedpossibility to use a template string withrandomChoiceInUrimaskAddednew maskadd-transientsame asaddbut the field is not exported in the jsonline outputAddedpossibility to use a template string directly withaddmaskAddedpossibility to chain multiple masks in YAML configuration on the same jsonpath with themasksproperty
Addednew maskfromjsonto convert a string to object model
Addednew masktemplate-eachto mask each value of an array with a templateFixedpimo doesn't crash anymore when nulls values occurs in the middle of a path
FixedProtect masks against null values (dateparser, duration, hash, randdura, range)
FixedRange over array in template mask
FixedPipe mask with repeat flag failed in panic (#34)Addedjsonschema command to generate schema of masking.yml fileAddedflag to enable or disable coloring in output logs (--color [yes|no|auto])
Changedorder of keys on JSON objects will now be preserved on the output
Addednew functions intemplatemask via sprig dependency v3 (durationRound, numerous, toRawJson, htpasswd, duration, seq, randInt, fromJson, mustFromJson, bcrypt, randBytes, dig, regexQuoteMeta, osBase, osDir, osExt, osClean, osIsAbs, and, all, addf, add1f, subf, divf, mulf, maxf, and minf, chunk, and more...) and improve others (get)Addedstrutured logging with-vand--log-jsonflagsAddeddebug option--debug, warning it's slow do not use in productionFixedflag--skip-line-on-errorFixedflag--skip-field-on-errorFixedhandling of null values by the theff1mask
Addedff1 mask to meet the requirement of re-identification from a single secret key.Addedpipe mask to handle jsons with complex structure (nested arrays of objects)Fixeduse same random mask on different path produce the same value.Fixedmasking values in nested arrays generate panic error.
AddedFirst public version released