[feat] Connect NYSERDA API and Download Data from NYISO #14

deenasun · 2024-10-14T02:03:28Z

What's new in this PR

Description

Created api/webscraper/utils module:

Contains a scraper_utils.py file with helper methods for parsing and filtering the data from our data sources
check_status checks the status of our projects
Any projects with "Completed" or "Operational" are considered "Operational"
Projects that have the status "Cancelled" are not included in our json files
Any other projects are classified as "Proposed" for now
geocode_lat_long uses the Google Maps API Geocoding function to get the approximate latitude and longitude information based on a city and state. This function is used for the small solar project data from NYSERDA, which contains a field for city_town but not latitude/longitude

Created api/webscraper/nyserda_scraper.py file:

This function fetches data from the NYSERDA Large-scale Renewable Projects database and NYSERDA Statewide Distributed Solar Projects database.
Filters for specific fields in the data, excluding any projects that have a "Cancelled" status
Dumps the data into nyserda_large.json and nyserda_small.json files

Created api/webscraper/scraper.py file:

Makes a get request to the NYISO url to download a xlsx file
For now, we simply load the fetched bytes into a Pandas dataframe

How to review

Standard procedure

git fetch origin deenasun/10-feat-connect-nyserda-api-and-download-data-from-nyiso
git checkout deenasun/10-feat-connect-nyserda-api-and-download-data-from-nyiso

NOTE: This function to parse and filter the data for the small solar projects NYSERDA data set makes calls to our Google Maps API! Don't run this file very frequently unless needed---you can also check the data already inside nyserda_large.json and nyserda_small.json.
To check if the scrapers for the NYSERDA data are working, you can run this command in your terminal:

python api/webscraper/nyserda_scraper.py

This will dump data into the nyserda_large.json and nyserda_small.json files inside the api/webscraper directory! If you want to see it in action, you can delete the data in there and run the command above and the json files should be repopulated.

If you run into any issues with dependencies, you may need to download certain python packages such as:

requests
json
python-dotenv
urllib
pandas
io

Run this command in your terminal:

pip install requests

for each of the needed dependencies!

Next steps

Download NYISO data using a webscraper to parse for the most up-to-date xlsx NYISO spreadsheet

Relevant links

Online sources

Related PRs

CC: @itsliterallymonique

…le geocoding api to retrieve latitude and longitude

itsliterallymonique

Looks good Deena. Please address the changes I suggested.

itsliterallymonique · 2024-10-16T20:30:39Z

api/webscraper/utils/scraper_utils.py

+google_maps_api_key = os.environ.get('NEXT_PUBLIC_GOOGLE_MAPS_API_KEY')
+
+def check_status(status):
+    if status is None:


In this case, let's return NULL for now.

Hi Monique! just wanted to clarify--in python, None is a special keyword used to refer to null objects. So do you want me to return a string "NULL"?

return None then ! But make sure when you write it in the database it will be NULL.

itsliterallymonique · 2024-10-16T20:35:19Z

api/webscraper/utils/scraper_utils.py

+        return 'Cancelled'
+    elif status.lower() == 'operational' or status.lower() == 'completed':
+        return 'Operational'
+    else:


let's return proposed only if status says under development

got it! what should we return if the status is missing? I can leave it as a null/None status and then add error handling for that in the other scraper files.

let's leave status as None if it is missing

itsliterallymonique · 2024-10-16T20:37:16Z

api/webscraper/nyserda_scraper.py

+                      'longitude': long,
+                      'data_through_date': item.get('data_through_date', None),
+      }
+      small_list.append(project_dict)


I am checking on the missing / ambiguous data and will have you fix anything once we get confirmation from the npo.

itsliterallymonique · 2024-10-16T20:38:46Z

api/webscraper/scraper.py

+from io import BytesIO
+import urllib
+
+nyiso = requests.get('https://www.nyiso.com/documents/20142/1407078/NYISO-Interconnection-Queue.xlsx')


next steps for this is to scrape the actual URL from the NYISO page

nehaahussain · 2024-10-17T06:14:43Z

api/webscraper/nyserda_scraper.py

+  response = requests.get('https://data.ny.gov/resource/dprp-55ye.json')
+  data = response.json()
+  filtered_list = []
+  for item in large_data:


data is created from the response.json but isn't used in this method. Do we want to go through the items in large_data or data from line 81?

nehaahussain · 2024-10-17T06:16:46Z

api/webscraper/nyserda_scraper.py

+                      'proposed_cod': item.get('year_of_delivery_start_date', None),
+                      'county': item.get('county_province', None),
+                      'region': item.get('redc', None),
+                      'zipcode:': item.get('zip_code', None),


nit: should be 'zipcode':

@nehaahussain ah yes! it's a little funky but that's the way they titled the "zipcode" field for the NYSERDA large-scale data

nehaahussain

Good job Deena! I added some comments:)

deenasun added 2 commits October 13, 2024 16:22

[feat] added check_status function for project queries from NYSERDA

5940d82

[feat] added util helper modules to check project status and use goog…

acaf507

…le geocoding api to retrieve latitude and longitude

deenasun linked an issue Oct 14, 2024 that may be closed by this pull request

[feat] Connect NYSERDA API and Download Data from NYISO #10

Closed

deenasun requested review from itsliterallymonique and nehaahussain October 14, 2024 05:32

itsliterallymonique requested changes Oct 16, 2024

View reviewed changes

[fix] removed unused imports, updated check_status util funciton

c6cc391

nehaahussain reviewed Oct 17, 2024

View reviewed changes

nehaahussain requested changes Oct 17, 2024

View reviewed changes

[fix] return None if status is None for scraper_utils

8356516

nehaahussain approved these changes Oct 18, 2024

View reviewed changes

deenasun requested a review from itsliterallymonique October 18, 2024 04:36

deenasun merged commit 8d52e3b into main Oct 18, 2024
2 checks passed

deenasun deleted the 10-feat-connect-nyserda-api-and-download-data-from-nyiso branch November 8, 2024 04:41

itsliterallymonique mentioned this pull request Nov 27, 2024

[chore] Web Scraper Fixes #90

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] Connect NYSERDA API and Download Data from NYISO #14

[feat] Connect NYSERDA API and Download Data from NYISO #14

deenasun commented Oct 14, 2024 •

edited

Loading

itsliterallymonique left a comment •

edited

Loading

itsliterallymonique Oct 16, 2024

deenasun Oct 16, 2024

itsliterallymonique Oct 17, 2024

itsliterallymonique Oct 16, 2024

deenasun Oct 16, 2024

itsliterallymonique Oct 17, 2024

itsliterallymonique Oct 16, 2024

itsliterallymonique Oct 16, 2024

nehaahussain Oct 17, 2024

nehaahussain Oct 17, 2024

deenasun Oct 17, 2024

nehaahussain left a comment

[feat] Connect NYSERDA API and Download Data from NYISO #14

[feat] Connect NYSERDA API and Download Data from NYISO #14

Conversation

deenasun commented Oct 14, 2024 • edited Loading

What's new in this PR

Description

How to review

Next steps

Relevant links

Online sources

Related PRs

itsliterallymonique left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nehaahussain left a comment

Choose a reason for hiding this comment

deenasun commented Oct 14, 2024 •

edited

Loading

itsliterallymonique left a comment •

edited

Loading