Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: analyze curtailment #57

Merged
merged 5 commits into from
Dec 13, 2019
Merged

feat: analyze curtailment #57

merged 5 commits into from
Dec 13, 2019

Conversation

danielolsen
Copy link
Contributor

@danielolsen danielolsen commented Dec 12, 2019

This branch adds a script postreise/analyze/curtailment.py with a few useful functions:

  • calculate_curtailment_time_series() gives pandas DataFrames of time-series curtailment of selected resource(s), by plant.
  • calculate_curtailment_percentage() gives the overall curtailment percentage of selected resource(s).
  • summarize_curtailment_by_bus() gives total curtailment by resources and by bus.
  • summarize_curtailment_by_location() gives total curtailment by resources and by (lat, long).

These functions also have tests in test_curtailment.py. Adding tests for these required adding a bit more functionality to the MockAnalyze class.

This branch also fixes a bug in helpers.py which causes an groupby error if you try to sum a subset of plants (since there will be a mismatch in the length of the dataframe column headers and in the the Series from the full set of plants that you're using to group by).

> python -m unittest -v
test_carbon_calc_always_on (postreise.analyze.tests.test_carbon.TestCarbonCalculation) ... ok
test_carbon_calc_decommit (postreise.analyze.tests.test_carbon.TestCarbonCalculation) ... ok
test_carbon_calc_simple (postreise.analyze.tests.test_carbon.TestCarbonCalculation) ... ok
test_carbon_summarization (postreise.analyze.tests.test_carbon.TestCarbonSummarization) ... ok
test_calculate_congestion_surplus_single_time (postreise.analyze.tests.test_congestion_surplus.TestCalculateCongestionSurplus)
Congested case from Kirschen & Strbac Section 5.3.2.4 ... ok
test_calculate_congestion_surplus_three_times (postreise.analyze.tests.test_congestion_surplus.TestCalculateCongestionSurplus)
First: congested. Second: uncongested. Third: uncongested, fuzzy. ... ok
test_map_demand_to_buses (postreise.analyze.tests.test_congestion_surplus.TestMappingHelpers) ... ok
test_calculate_curtailment_percentage_default (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentPercentage) ... ok
test_calculate_curtailment_percentage_solar (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentPercentage) ... ok
test_calculate_curtailment_percentage_solar_wind (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentPercentage) ... ok
test_calculate_curtailment_percentage_wind (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentPercentage) ... ok
test_calculate_curtailment_time_series_default (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentTimeSeries) ... ok
test_calculate_curtailment_time_series_solar (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentTimeSeries) ... ok
test_calculate_curtailment_time_series_solar_wind (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentTimeSeries) ... ok
test_calculate_curtailment_time_series_wind (postreise.analyze.tests.test_curtailment.TestCalculateCurtailmentTimeSeries) ... ok
test_error_geothermal_curtailment (postreise.analyze.tests.test_curtailment.TestCheckResourceInScenario) ... ok
test_error_no_solar (postreise.analyze.tests.test_curtailment.TestCheckResourceInScenario) ... ok
test_summarize_curtailment_by_bus (postreise.analyze.tests.test_curtailment.TestSummarizeCurtailmentByBus) ... ok
test_summarize_curtailment_by_location (postreise.analyze.tests.test_curtailment.TestSummarizeCurtailmentByLocation) ... ok
test_summarize_all_buses_false (postreise.analyze.tests.test_helpers.TestSummarizePlantToBus) ... ok
test_summarize_all_buses_true (postreise.analyze.tests.test_helpers.TestSummarizePlantToBus) ... ok
test_summarize_default (postreise.analyze.tests.test_helpers.TestSummarizePlantToBus) ... ok
test_summarize_location (postreise.analyze.tests.test_helpers.TestSummarizePlantToLocation) ... ok
test_calculate_mw_miles_many_scaled (postreise.analyze.tests.test_mwmiles.TestCalculateMWMiles) ... ok
test_calculate_mw_miles_no_scale (postreise.analyze.tests.test_mwmiles.TestCalculateMWMiles) ... ok
test_calculate_mw_miles_one_line_scaled (postreise.analyze.tests.test_mwmiles.TestCalculateMWMiles) ... ok
test_calculate_mw_miles_one_transformer_scaled (postreise.analyze.tests.test_mwmiles.TestCalculateMWMiles) ... ok
test_classify_interstate_intrastate_bad_ct (postreise.analyze.tests.test_statelines.TestClassifyInterstateIntrastate) ... ok
test_classify_interstate_intrastate_empty_ct (postreise.analyze.tests.test_statelines.TestClassifyInterstateIntrastate) ... ok
test_classify_interstate_intrastate_none (postreise.analyze.tests.test_statelines.TestClassifyInterstateIntrastate) ... ok
test_classify_interstate_intrastate_several (postreise.analyze.tests.test_statelines.TestClassifyInterstateIntrastate) ... ok
test_classify_interstate_intrastate_two (postreise.analyze.tests.test_statelines.TestClassifyInterstateIntrastate) ... ok
test_mock_grid_failures (postreise.tests.test_mocks.TestMocks) ... ok
test_mock_grid_successes (postreise.tests.test_mocks.TestMocks) ... ok
test_mockpg_stored_properly (postreise.tests.test_mocks.TestMocks) ... ok

----------------------------------------------------------------------
Ran 35 tests in 0.388s

OK

Copy link
Collaborator

@merrielle merrielle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a bunch of advice but in general this looks awesome! I'm always on board for adding useful stuff that we can reuse when we need it.

"""Ensure that curtailment is a dict of dataframes, and that each key is
represented in at least one generator in grid.
:param dict curtailment: keys are resources, values are pandas.DataFrame.
:param powersimdata.input.grid grid: Grid instance.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pycharm is saying you need to use :param powersimdata.input.grid.Grid here and in the rest of the docstrings

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo

Copy link
Collaborator

@rouille rouille Dec 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the :param powersimdata.scenario.scenario.Scenario scenario:

_resource_func = {'solar': 'get_solar', 'wind': 'get_wind'}


def _check_scenario(scenario):
Copy link
Collaborator

@merrielle merrielle Dec 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally write custom checks on inputs when having the wrong input would cause a really confusing error or possibly a silent bug, for example when an array needs to be a certain length. So I'm not sure you need this -- I think calling if not isinstance(scenario.state, Analyze) should cause a pretty clear error on its own.

:param powersimdata.input.grid grid: Grid instance.
:return: (*None*).
"""
if not isinstance(curtailment, dict):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also recommend only validating inputs on public facing functions. If you want, you can check the inputs for private functions in tests instead. There's a lot of conflicting opinions here but in my experience most people follow these guidelines since it means a lot less work. Check it out: https://softwareengineering.stackexchange.com/questions/64926/should-a-method-validate-its-parameters/64929#64929

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is 'public' in this case? summarize_curtailment_by_bus() and summarize_curtailment_by_location() are both public functions of the module, so I'm checking their inputs but using a custom private function to avoid duplication (since the inputs to each are the same). Is the idea that these are our own analysis functions so they're not really public? All of the _check* functions in this module are doing this: avoiding duplication of code when checking the inputs of the public functions.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this particular function has an underscore at the beginning of the name so I'm assuming it's private.

indexed by (datetime, plant) where plant is only plants of matching type.
"""
_check_scenario(scenario)
_check_resources(resources)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like checking the resource types here is a great use case for input validation! :)

err_msg += '. Plant: ' + ', '.join(gentypes_in_grid)
raise ValueError(err_msg)

def calculate_curtailment_time_series(scenario, resources=('solar', 'wind')):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice job using an immutable tuple here - it avoids a lot of sneaky python bugs

curtailment = {}
for rentype, genpotential in rentype_genpotential.items():
ren_plants = list(genpotential.columns)
#gens_pg = pg[ren_plants]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think you missed deleting this

@rouille
Copy link
Collaborator

rouille commented Dec 13, 2019

Should we have a curtailment time series in relative difference (in %)? The calculate_curtailment_time_series function gives the time series of the curtailment in MWh (difference between generated and available). Also, do we want the time series by state / load zone? These are just suggestions. I kow that the above is available through AnalyzePG. Since, we want to refactor the latter, it could be a good opportunity to actually use your module to get every possible curtailment analysis and use it to make the curtailment plots in postreise.plot.

Copy link
Collaborator

@rouille rouille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice

@danielolsen
Copy link
Contributor Author

Should we have a curtailment time series in relative difference (in %)? The calculate_curtailment_time_series function gives the time series of the curtailment in MWh (difference between generated and available). Also, do we want the time series by state / load zone? These are just suggestions. I know that the above is available through AnalyzePG. Since, we want to refactor the latter, it could be a good opportunity to actually use your module to get every possible curtailment analysis and use it to make the curtailment plots in postreise.plot.

You raise a good point, this module does not replicate all of the functionality present in AnalyzePG. This is code that I had written that we needed to do some analysis for slides last week, and I figured that it should go into the codebase for the next time we need it. When we refactor AnalyzePG, we probably want to move more of the curtailment calculation processes into this script, with the methods in AnalyzePG pointing to the functions here.

@danielolsen danielolsen merged commit 5d6fb34 into develop Dec 13, 2019
rouille pushed a commit that referenced this pull request Jan 31, 2020
* feat: analyze curtailment by resource, bus

* refactor: use summarize_plant_to_bus helper, clean cruft

* fix: avoid error when summing subset of plants

* feat: add summarization of curtailment by location, input checking

* chore: cleaning up docstrings, PEP8, commented-out cruft
@danielolsen danielolsen deleted the analyze_curtailment branch March 23, 2020 22:54
@ahurli ahurli mentioned this pull request Mar 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants