PowerFlows.jl unified develop branch (#53)

* add raw psy -> psse exporter code

* Run formatter

* Code cleanup: remove machine-specific paths, add dependencies

* Code cleanup: minor edits to make `Write_Sienna2PSSE` run

* Add prefixes, adjust imports to make standalone scripts work

* Add `DISABLED_TEST_FILES` option like in PowerSimulations

* Add compat for new dependencies

* Begin work on a formal exporter interface

* Add rough draft tests for exporter interface

* Add `update_exporter!(..., System)`, extensively refactor tests

* PSSE exporter: test AC powerflow on reimport, minor test refactor

* Begin implementing routine to update a `System` from a `PowerFlowData`

* Finish first draft of `update_system!`

* Exporter: add round trip generator name remapping

* Exporter: add round trip *bus* name remapping

* Add round trip branch name remapping, RTS_GMLC tests, more test tweaks

* Add load zone name mapping, test verbosity; make RTS_GMLC tests pass

* Implement `update_exporter!` for `PowerFlowData`, build out RTS tests

* Reimplement Case Identification Data and Bus Data export (untested)

Those sections being

* Reimplement Load Data and Fixed Bus Shunt Data export (untested)

* Reimplement Generator Data and Non-Transformer Branch Data export

(untested)

* Reimplement Transformer Data export (first draft, untested)

* Implement Zone Data export, keep track of unimplemented sections

* Add option for customary comments at end of group, etc.

* Support `PowerLoad`s

* Use new `IS.compare_values` interface

* Revamp reimport remappers, etc. so tests fully pass with both systems

* Remove old exporter implementation

* Address minor PR comments, migrate from JSON.jl to JSON3.jl

* Minor `end_group` refactor

* Multiply dispatch on export groups: part 1

* Multiply dispatch on export groups: part 2

* Add `write_comments` test

* Remove more old code

* Fix power limits units, fix redistribution calls, power flow is broken

* Remove old dependency and TODOs, fix some power flow testing

* add abstract type

* add branch types for the PowerFlowData

* time_step clean up

* add methods and types

* add clear method

* Do not copy time series when `deepcopy`ing for PSS/E export

* Reduce arguments to `write_export`

* Add optional extra data to `PowerFlowData`

* Factor out egregious `PowerFlowData` constructor code duplication

* Add overwrite option to PSS/E export

* Create a `PSSEExportPowerFlow` `PowerFlowData` using `extra_data`

* Fix `clear_injection_data!` bug, other minor code cleanup

* Create `PowerFlowContainer` type hierarchy, supporting infrastructure

* Remove `extra_data` from `PowerFlowData`

* Make `PowerFlowData` tests actually test something

* `PSSEExporter`: Add `name`, `step`, and `overwrite` fields

* Better accommodate multi period in PSS/E export

* Misc code cleanup following self-review

* Minor test fixes

* Implement `get_total_p`, `q` for `InterruptiblePowerLoad`

* Exporter: allow for null area, load zone

* Strictly specify `vPTDFPowerFlowData`, fix bug 50

* Add detail to `deepcopy_system_no_time_series`

* Gracefully skip tests when file is missing

* Refactor test data path

* Exporter: tighten test comparisons, remove unused helper functions

* Refactor test export directory

* PSSE reimport: add `shunt_name_formatter`

* PSSE exporter: remove rest of old testing

* Make exporter more flexible

* PERF: reuse export buffer, cache component lists

* PERF: cache serialized exporter metadata

* PERF: use existing buffer for `show(::IOBuffer, ::Float64)`

* Fix `show` optimization compatibility

* PERF: unroll `joinln` loops by hand (ugh)

* WIP: implement NR power flow function, rename ACPowerFlow to NLSolveACPowerFlow, introduce KLUACPowerFlow for the use in the new NR power flow function

* WIP: some bugfixes in the PF function

* small changes

* KLU PF: use correct V0; return x in expected format; write results function supports KLU PF; implement tests for KLUACPowerFlow

* reformat code

* formatting

* Addressed review comments except for using functor

* formatter

* small fix ACPowerFlow type; add test for results consistency for 2000 bus system

* small change

* optimize NR code to reduce memory allocations

* trying to reduce memory allocation

* Use PSY's `with_units_base`

See NREL-Sienna/PowerSystems.jl#1232

* Use PSY's `fast_deepcopy_system`

* PERF: remove try/catch in tight loop, cache more

* NR PF non-allocating

* vectorized update for dSbus_dV

* fixes Add `solve_powerflow!` for AC power flow #49

* formatting

* WIP: multiperiod AC PF

* adrressing comments

* wip: multiperiod ac pf

* Modify `PowerFlowData` fields for multi-period AC PF

* Add `to_from` vs. `from_to` distinction to `PowerFlowData`

* Modify `PowerFlowData` constructors given new fields

* Make tests pass with new `PowerFlowData` fields

* wip: solve_powerflow!

* integrate previous changes to powersystemdata

* remove a println

* small change to _update_system!

* Implement power flow-specific (re)active power limits proxies (#79)

Fixes #39

* Fix `ACPowerFlow` method overwriting

* Remove old/obsolete TODOs

* add field "converged" to PowerFlowData, return NaN when not converged

* Fix typing of empty vector from `get_name.`

* wip: SolverData, penalty factors

* fix penalty factors function

* bugfix shape penalty factors

* Update dependencies

* Comment out broken tests

* Add fields to `PowerFlowEvaluationModel`s

* Address some exporter TODOs, misc cleanup

* implement yft, ytf calculations in solve_powerflow!

* adjust for the changes in PNM

* formatter

* Fine-tune exporter interface, minor cleanup

* Move reimporting code to PSY

* Reform export metadata file, make tests pass

* More thoroughly exercise new PSY reimporting interface

* Reduce manual computation by relying on a new PSY bugfix

Depends on NREL-Sienna/PowerSystems.jl#1270

* Fix `yft`, `ytf` typos

* clean up code for release

* time_step removed where not relevant

* add multi-period pf for nlsolve, test to compare results across solvers for multi-period pf

* formatter

* docstrings

* Add `Storage` to exporter, resolve minor TODOs

* Address PR comments

* address PR comments

---------

Co-authored-by: HaleyRoss <[email protected]>
Co-authored-by: Jose Daniel Lara <[email protected]>
Co-authored-by: Roman Bolgaryn <[email protected]>
Co-authored-by: Roman Bolgaryn <[email protected]>

.gitignore

@@ -1,4 +1,5 @@
 test_simulation_results/*
+test/test_exports
 #Files generated by invoking Julia with --code-coverage
 *.jl.cov
 *.jl.*.cov

Project.toml

@@ -5,19 +5,29 @@ version = "0.7.2"
 
 [deps]
 DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
+DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
+Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
 InfrastructureSystems = "2cd47ed4-ca9b-11e9-27f2-ab636a7671f1"
+JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
+KLU = "ef3ab10e-7fda-4108-b977-705223b18434"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
+Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
 NLsolve = "2774e3e8-f4cf-5e23-947b-6d7e65073b56"
 PowerNetworkMatrices = "bed98974-b02a-5e2f-9fe0-a103f5c450dd"
 PowerSystems = "bcd98974-b02a-5e2f-9ee0-a103f5c450dd"
 SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
 
 [compat]
 DataFrames = "1"
-InfrastructureSystems = "2"
+DataStructures = "0.18"
+Dates = "1"
+InfrastructureSystems = "^2.4"
+JSON3 = "1"
+KLU = "^0.6"
 LinearAlgebra = "1"
+Logging = "1"
 NLsolve = "4"
-PowerNetworkMatrices = "^0.11"
-PowerSystems = "^4"
+PowerNetworkMatrices = "^0.12"
+PowerSystems = "^4.5"
 SparseArrays = "1"
 julia = "^1.6"