KeyError: 'gasDayStartedOn'

[KarelWintersky]

storage = pandas.read_json(prefix+"storage.json")
storage["timestamp"]=pandas.to_datetime(storage["gasDayStartedOn"]) + datetime.timedelta(days=1)
storage.sort_values(["timestamp"], inplace=True)
storage.set_index("timestamp", inplace=True)
storage=storage["2022-01-01":]
storage

KeyError                                  Traceback (most recent call last)
File ~/.local/lib/python3.9/site-packages/pandas/core/indexes/base.py:3629, in Index.get_loc(self, key, method, tolerance)
   3628 try:
-> 3629     return self._engine.get_loc(casted_key)
   3630 except KeyError as err:

File ~/.local/lib/python3.9/site-packages/pandas/_libs/index.pyx:136, in pandas._libs.index.IndexEngine.get_loc()

File ~/.local/lib/python3.9/site-packages/pandas/_libs/index.pyx:163, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:5198, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:5206, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'gasDayStartedOn'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In [14], line 2
      1 storage = pandas.read_json(prefix+"storage.json")
----> 2 storage["timestamp"]=pandas.to_datetime(storage["gasDayStartedOn"]) + datetime.timedelta(days=1)
      3 storage.sort_values(["timestamp"], inplace=True)
      4 storage.set_index("timestamp", inplace=True)

File ~/.local/lib/python3.9/site-packages/pandas/core/frame.py:3505, in DataFrame.__getitem__(self, key)
   3503 if self.columns.nlevels > 1:
   3504     return self._getitem_multilevel(key)
-> 3505 indexer = self.columns.get_loc(key)
   3506 if is_integer(indexer):
   3507     indexer = [indexer]

File ~/.local/lib/python3.9/site-packages/pandas/core/indexes/base.py:3631, in Index.get_loc(self, key, method, tolerance)
   3629     return self._engine.get_loc(casted_key)
   3630 except KeyError as err:
-> 3631     raise KeyError(key) from err
   3632 except TypeError:
   3633     # If we have a listlike key, _check_indexing_error will raise
   3634     #  InvalidIndexError. Otherwise we fall through and re-raise
   3635     #  the TypeError.
   3636     self._check_indexing_error(key)

KeyError: 'gasDayStartedOn'

there is no such value at any json files

[berthubert]

hi - I've just updated the Jupyter notebook in the repository, it lagged behind the .py version. Can you check if this solved your problem?

[KarelWintersky]

I cloned repo again.

Traceback (most recent call last):
  File "/home/arris/gazproject/gazmon.py", line 38, in <module>
    uzhgorod = pandas.read_json(prefix+"uzhgorod-data.json")
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 207, in wrapper
    return func(*args, **kwargs)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 612, in read_json
    return json_reader.read()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 746, in read
    obj = self._get_object_parser(self.data)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 768, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 880, in parse
    self._parse_no_numpy()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 1133, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None
ValueError: Expected object or value
ssh: connect to host 10.0.0.4 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(228) [sender=3.2.3]

after running update.sh

[KarelWintersky]

Also:

NameError                                 Traceback (most recent call last)
Cell In [2], line 10
      7     v.set_index("timestamp", inplace=True)
      8     print(v.index.min(), v.index.max())
---> 10 uzhgorod = pandas.read_json(prefix+"uzhgorod-data.json")
     11 nordstreamF = pandas.read_json(prefix+"nordstream-fluxsys-data.json")
     12 nordstreamO = pandas.read_json(prefix+"nordstream-opal-data.json")

NameError: name 'pandas' is not defined

NameError                                 Traceback (most recent call last)
Cell In [5], line 1
----> 1 plt.figure()
      2 plt.plot(1000*storage.gasInStorage/(365*24))   # TWh
      3 plt.ylabel("GW*year")

NameError: name 'plt' is not defined

but

/home/arris/gazproject$ pip install pandas
Requirement already satisfied: pandas in /home/arris/.local/lib/python3.9/site-packages (1.4.4)
Requirement already satisfied: numpy>=1.18.5 in /home/arris/.local/lib/python3.9/site-packages (from pandas) (1.23.3)
Requirement already satisfied: python-dateutil>=2.8.1 in /home/arris/.local/lib/python3.9/site-packages (from pandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3/dist-packages (from pandas) (2021.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)

[KarelWintersky]

/home/arris/gazproject$ python3 gazmon.py
2022-01-02 06:00:00 2022-09-20 06:00:00
2022-01-02 06:00:00 2022-09-20 06:00:00
2022-01-02 06:00:00 2022-09-20 06:00:00
2022-01-02 06:00:00 2022-09-21 06:00:00
2022-01-02 06:00:00 2022-09-21 06:00:00
2022-01-02 06:00:00 2022-09-20 06:00:00
2022-01-02 06:00:00 2022-09-20 06:00:00
Traceback (most recent call last):
  File "/home/arris/gazproject/gazmon.py", line 65, in <module>
    yamalK_h = pandas.read_json(prefix+"yamal-kondratki-data-intraday.json")
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 207, in wrapper
    return func(*args, **kwargs)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 612, in read_json
    return json_reader.read()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 746, in read
    obj = self._get_object_parser(self.data)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 768, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 880, in parse
    self._parse_no_numpy()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 1133, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None
ValueError: Expected object or value

of course, I changed prefix value to /home/arris/gazproject/

[KarelWintersky]

After downloading *intraday.json (get-data-intraday.sh) new issue appeared:

Traceback (most recent call last):
  File "/home/arris/gazproject/gazmon.py", line 82, in <module>
    storage = pandas.read_json(prefix+"storage-old.json")
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 207, in wrapper
    return func(*args, **kwargs)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 612, in read_json
    return json_reader.read()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 746, in read
    obj = self._get_object_parser(self.data)
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 768, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 880, in parse
    self._parse_no_numpy()
  File "/home/arris/.local/lib/python3.9/site-packages/pandas/io/json/_json.py", line 1133, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None
ValueError: Expected object or value