Add stack trace decoding [$35] #31

diimdeep · 2017-04-12T09:44:40Z

There is
https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/stack_dump.rst

https://github.com/me-no-dev/EspExceptionDecoder
https://github.com/littleyoda/EspStackTraceDecoder

There is a $35 open bounty on this issue. Add to the bounty at Bountysource.

psy0rz · 2017-04-24T20:50:22Z

we really need this..now i need to switch back to arduino ide, just do decode stack traces.

BBBSnowball · 2017-07-22T22:13:51Z

I have created some code that decodes the addresses and is integrated into miniterm as a filter:
espminiterm.py

The code is hardly tested at all, it isn't integrated with PlatformIO and all the paths are hardcoded so it is only a first step towards solving this issue. However, it is enough to continue working on my main project.

osos · 2018-01-21T11:34:51Z

Also look at: https://github.com/janLo/EspArduinoExceptionDecoder

Forum: https://community.platformio.org/t/could-we-add-support-for-stack-trace-decoding-in-the-esp8266/1017

JarekParal · 2018-06-27T20:56:26Z

Any progress in this issue?

JarekParal · 2018-06-28T12:06:57Z

Could you look at that @Tasssadar?

Misiu · 2018-08-17T13:45:02Z

@ivankravets any chance this might get added? It would be awesome to have as much debug information as possible

HenrikSte · 2018-11-14T10:48:44Z

Added 15,- to the bounty to make it 50. Anyone else?

stritti · 2019-04-02T19:32:16Z

maybe there could be just a button to copy&paste the trace to following command:
https://gist.github.com/stritti/b5f409963139af11676a5b39614aa36d

edhedges · 2019-04-25T15:20:46Z

I've used https://github.com/littleyoda/EspStackTraceDecoder as a CLI, but it'd be great if it automagically happened in VS Code. I'd like to add this for my own selfish purposes. Where could I get started at within the code / VS Code extension?

Example execution:
java -jar EspStackTraceDecoder.jar ~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-addr2line ~/PROJECT_DIR/.pioenvs/d1_mini_pro/firmware.elf ~/PROJECT_DIR/dump.txt

bzeeman · 2019-05-08T20:31:36Z

The Arduino one used to work for me a few weeks ago. Now it doesn't. Not sure what changed. I even reverted back to [email protected].
C'mon PIO.

HollisTech · 2019-12-19T14:21:54Z

I wrote a simple psh script to filter a log file with Backtrace: sections and decode them: https://gist.github.com/HollisTech/18f7bd2963e9c33fc8b630ff414e13a9

This filter can decode the crash traces emitted by Espressif devices. Add '--filter=esp_exception_decoder' to monitor_flags to use it. Fixes platformio/platform-espressif8266#31

Updates platformio/platform-espressif8266#31

Add --filter=esp8266_exception_decoder to monitor_flags to use it. Fixes platformio#31

* Implement mechanism for adding platform filters into miniterm Updates platformio/platform-espressif8266#31 * DeviceMonitorFilter: fixes for Windows and Python2

#197) * Esp8266ExceptionDecoder: add monitor filter to decode exception traces Add --filter=esp8266_exception_decoder to monitor_flags to use it. Fixes #31 * Esp8266ExceptionDecoder: ignore unknown addresses

ivankravets · 2020-03-13T10:49:11Z

Thanks to @Tasssadar 's great contribution this issue has been finally resolved! We need some feedback/testing before making the final release of PlatformIO Core 4.3 and this dev-platform. Please re-test:

Open PlatformIO IDE Terminal and run pio upgrade --dev.
Use the upstream version of this dev-platform. See docs https://docs.platformio.org/en/latest/platforms/espressif8266.html#upstream
Run pio update.

See documentation https://docs.platformio.org/en/latest/core/userguide/device/cmd_monitor.html#filters

cocus · 2020-03-16T00:07:20Z

@ivankravets I've tried the upstream version of the platform, and I've also used the dev version for pio.
After running into an issue in a firmware I was trying on an esp8266, I tried this and I got:

Exception (9):
epc1=0x402193d4 epc2=0x00000000 epc3=0x40218174 excvaddr=0x04a00423 depc=0xException in thread rx:
Traceback (most recent call last):
  File "C:\Users\cocus\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "C:\Users\cocus\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\cocus\.platformio\penv\lib\site-packages\serial\tools\miniterm.py", line 452, in reader
    text = transformation.rx(text)
  File "C:\Users\cocus\.platformio\platforms\espressif8266@src-eb7495f88eb0afa18fedff98bfb5e40f\monitor\filter_exception_decoder.py", line 160, in rx
    extra = self.process_line(line)
  File "C:\Users\cocus\.platformio\platforms\espressif8266@src-eb7495f88eb0afa18fedff98bfb5e40f\monitor\filter_exception_decoder.py", line 187, in process_line
    return self.process_exception_match(match)
  File "C:\Users\cocus\.platformio\platforms\espressif8266@src-eb7495f88eb0afa18fedff98bfb5e40f\monitor\filter_exception_decoder.py", line 225, in process_exception_match
    lines = self.get_lines([p[1] for p in pairs])
  File "C:\Users\cocus\.platformio\platforms\espressif8266@src-eb7495f88eb0afa18fedff98bfb5e40f\monitor\filter_exception_decoder.py", line 265, in get_lines
    subprocess.check_output(args + [addr.encode(enc)])
  File "C:\Users\cocus\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "C:\Users\cocus\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\cocus\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "C:\Users\cocus\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1119, in _execute_child
    args = list2cmdline(args)
  File "C:\Users\cocus\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 530, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: a bytes-like object is required, not 'str'

What I get from the serial console (when the esp8266 code crashes) is always this:


Exception (9):
epc1=0x402193d4 epc2=0x00000000 epc3=0x402092bc excvaddr=0x04a00423 depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3ffffcf0 end: 3fffffc0 offset: 01a0
3ffffe90:  3ffe8ac2 00000001 3ffee8d0 40206ac8
3ffffea0:  3ffe89cc 47c66a8d 3ffee8d0 40207820  
3ffffeb0:  00000020 3ffef3d0 3ffee8d0 40207bc8
3ffffec0:  426c9b00 00000020 3fffff60 40209474  
3ffffed0:  3fffff60 3fffff60 3ffe89cc 402094aa
3ffffee0:  426c9b00 47c66a8d 3ffee8d0 40201f33
3ffffef0:  35313031 6820392e 00006150 3ffeed34
3fffff00:  000000f4 000000b7 3ffeea09 4020ae30
3fffff10:  007a1200 33116a3d 3ffee900 402023ac  
3fffff20:  401059d5 002eba80 3ffeee3c 00000000
3fffff30:  3ffee2e0 3ffeee3c 00000064 3ffeed34
3fffff40:  3fffdad0 3ffeee3c 00000064 4020a227  
3fffff50:  4020af85 00000064 3ffeed34 4020af7a  
3fffff60:  3fff1484 0020002f 00ff1500 3fff1770
3fffff70:  00000000 3fff1790 3fff1770 402036a0  
3fffff80:  41c851ec 00000000 00000001 4010022c
3fffff90:  3fffdad0 00000000 3ffeecf4 3ffeed34  
3fffffa0:  3fffdad0 00000000 3ffeecf4 4020a33c
3fffffb0:  feefeffe feefeffe 3ffe872c 401011e9
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1392, room 16 
tail 0
chksum 0xd0
csum 0xd0
v3d128e5c

EDIT: I've dumped some variables on function get_lines from filter_exception_decoder.py that might be out of interest:

enc: utf-8
self.addr2line_path: C:\Users\cocus\.platformio\packages\toolchain-xtensa\bin\xtensa-lx106-elf-addr2line.exe

EDIT2: I've taken the mods from (google/adb-sync#37) and added them to the .py file. Not ideal but they do work!. However, I'm not sure if this is a fault only found on Windows or if it applies to other OSes as well.

Core 4.3.0b1,Home 3.1.1, platform git hash f81f371

ivankravets · 2020-03-16T12:52:36Z

@Tasssadar could you help @cocus?

Tasssadar · 2020-03-16T13:31:51Z

Definitely, will take a look in the evening. Man, encodings are fun.

Tasssadar · 2020-03-16T21:34:36Z

Fixed in #201 and platformio/platform-espressif32#303 respectively. I tested the fix on Linux and Windows with python 2.7 and 3.7 (on 3.8, subprocess eats bytes just fine, so the bug did not manifest).

ivankravets · 2020-03-16T21:44:55Z

@Tasssadar thank you so much!!! 😊

sblantipodi · 2020-04-23T23:55:28Z

How can I use this feature?
My serial console always show unreadable stack trace on my esp8266.

Please help

Pablo2048 · 2020-04-24T04:08:36Z

Did you try to use the documentation? https://docs.platformio.org/en/latest/core/userguide/device/cmd_monitor.html#filters

ali80 · 2020-05-21T06:30:38Z

Using PIO 4.3.3 when I Try
pio device monitor -f esp8266_exception_decoder
I get the normal monitor output and not the output from exception_decoder

it seems that PIO registers the filter esp8266_exception_decoder as a valid filter but it doesn't have any effect on its output

--- Available filters and text transformations: colorize, debug, default, direct, esp8266_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters

Please build project in debug configuration to get more details about an exception.
See https://docs.platformio.org/page/projectconf/build_configurations.html


--- Miniterm on COM22  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

Let's provoke the s/w wdt firing...

Soft WDT reset

>>>stack>>>

ctx: cont
sp: 3ffffdf0 end: 3fffffc0 offset: 01a0
3fffff90:  3fffdad0 00000000 3fff1d74 40204608
3fffffa0:  feefeffe feefeffe 3fff1e64 4020f48c
3fffffb0:  feefeffe feefeffe 3ffe8548 40100e75
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

Let's provoke the s/w wdt firing...

ivankravets · 2020-05-21T07:00:56Z

See note in log that is starting with “ Please build project...”

ali80 · 2020-05-21T08:55:41Z

if you mean using build_type = debug

same result, only hiding the mentioned note

--- Available filters and text transformations: colorize, debug, default, direct, esp8266_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters
--- Miniterm on COM22  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

Soft WDT reset

>>>stack>>>

ctx: cont
sp: 3ffffdf0 end: 3fffffc0 offset: 01a0
3fffff90:  3fffdad0 00000000 3ffe8548 40209062
3fffffa0:  feefeffe feefeffe feefeffe 40216f30
3fffffb0:  feefeffe feefeffe feefeffe 40101035
<<<stack<<<

last failed alloc call: 40209062(400000000)

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 3456, room 16
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

--- exit ---

used this code to generate the fault

Serial.begin(115200);
auto a = new int[100000000];
while (1);

Tasssadar · 2020-05-21T09:19:50Z

That log does not have this part:

Exception (9):
epc1=0x402193d4 epc2=0x00000000 epc3=0x402092bc excvaddr=0x04a00423 depc=0x00000000

The filter is looking for it and won't trigger unless it finds it. Can you please paste here the whole log output, with at least 2 crashes/restart in it so I can see all of the text esp8266 outputs in this case?

ali80 · 2020-05-21T09:27:52Z

It doesn't have that line!

--- Available filters and text transformations: colorize, debug, default, direct, esp8266_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters
--- Miniterm on COM22  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

Soft WDT reset

>>>stack>>>

ctx: cont
sp: 3ffffdf0 end: 3fffffc0 offset: 01a0
3fffff90:  3fffdad0 00000000 3ffe8548 40209062
3fffffa0:  feefeffe feefeffe feefeffe 40216f30
3fffffb0:  feefeffe feefeffe feefeffe 40101035
<<<stack<<<

last failed alloc call: 40209062(400000000)

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

Soft WDT reset

>>>stack>>>

ctx: cont
sp: 3ffffdf0 end: 3fffffc0 offset: 01a0
3fffff90:  3fffdad0 00000000 3ffe8548 40209062
3fffffa0:  feefeffe feefeffe feefeffe 40216f30
3fffffb0:  feefeffe feefeffe feefeffe 40101035
<<<stack<<<

last failed alloc call: 40209062(400000000)

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

Soft WDT reset

>>>stack>>>

ctx: cont
sp: 3ffffdf0 end: 3fffffc0 offset: 01a0
3fffff90:  3fffdad0 00000000 3ffe8548 40209062
3fffffa0:  feefeffe feefeffe feefeffe 40216f30
3fffffb0:  feefeffe feefeffe feefeffe 40101035
<<<stack<<<

last failed alloc call: 40209062(400000000)

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

--- exit ---

Tasssadar · 2020-05-21T09:29:38Z

Okay, thanks, I'll change it to also trigger on just the >>>stack>>> marker when I have some time (unless somebody else wants to do it :P).

Keep in mind that your crash is rather syntetic, it just triggers the Watchdog because it's stuck, so it should work for other "regular" crashes (but the filter should definitely handle this too).

ivankravets · 2020-05-21T09:31:10Z

@Tasssadar we would be thankful for your PR.

ali80 · 2020-05-21T09:33:06Z

Much appreciated,
This didn't work in my real use case, I tried to test this simplified case to see if that works

ali80 · 2020-05-21T10:04:25Z

@Tasssadar This is my actual trace output from real time usage, it does have the Exception ( yet the filter still won't trigger

Starting AP:
  SSID = ****
  Password = ****
  HostName = ****
  IP = 192.168.4.1
WIFI Init Ran
WIFI OTA Initialized

Exception (0):
epc1=0x4000dce5 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3ffffd80 end: 3fffffc0 offset: 0190
3fffff10:  00000000 4bc6a7f0 224dd2f1 3fff2c80
3fffff20:  00000003 02bf3a26 00000003 4020be58
3fffff30:  00000000 00000000 401005a6 00000441
3fffff40:  00000000 3fff3374 401005a6 00000441
3fffff50:  0004eb4f 15de7633 3fff2100 3fff2c80
3fffff60:  0004eb4e 15de75fb 3fff2a00 3fff2c80
3fffff70:  3fff2b4c 3fff2b4c 3fff2784 402138fd
3fffff80:  3fffdad0 00000000 000000e1 4020ba68
3fffff90:  3fffdad0 00000000 3fff4190 4020a5b4
3fffffa0:  3fffdad0 00000000 3ffe8548 40222c67
3fffffb0:  feefeffe feefeffe feefeffe 401011ad
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3456, room 16
tail 0
chksum 0x84
csum 0x84
va5432625
~ld

Tasssadar · 2020-05-21T18:39:22Z

CRLF vs LF strikes again! Thank god Github preserves the line endings, otherwise I'd have no idea what's happening.

#211 should fix both of your traces, are you able to test it @ali80?

ivankravets · 2020-05-21T18:52:40Z

Thanks, @Tasssadar!

@ali80 please re-test with upstream version https://docs.platformio.org/en/latest/platforms/espressif8266.html#upstream

ali80 · 2020-05-21T19:05:10Z

@Tasssadar OK, the initial example

Serial.begin(115200);
auto a = new int[100000000];
while (1);

works, but here is another case that still doesn't work.

Serial.begin(115200);
int a = 0;
int b = 10;
int c = b / a;

giving this output

--- Available filters and text transformations: colorize, debug, default, direct, esp8266_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at http://bit.ly/pio-monitor-filters
--- Miniterm on COM22  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Exception in thread rx:
Traceback (most recent call last):
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\ali\.platformio\penv\lib\site-packages\serial\tools\miniterm.py", line 452, in reader
    text = transformation.rx(text)
  File "C:\Users\Ali\.platformio\platforms\espressif8266\monitor\filter_exception_decoder.py", line 158, in rx
    if line[-1] == "\r":
IndexError: string index out of range

--- exit ---

Tasssadar · 2020-05-21T19:06:39Z

Yep, I made a mistake in the pull request, fix is already waiting at #212 , sorry :(

ivankravets · 2020-05-21T19:07:41Z

@ali80 please re-run pio update

ali80 · 2020-05-21T19:08:10Z

Could you please also fix it for ESP32, I think it had the same problem

ali80 · 2020-05-21T19:16:15Z

@ivankravets moved to upstream and everything is up-to-date,
@Tasssadar another interesting thing

Serial.begin(115200);
int a = 0;
int b = 10;
int c = b / a;

still doestn work but

Serial.begin(115200);
int a = 0;
int b = 10;
volatile int c = b / a;

works!

Tasssadar · 2020-05-21T19:20:27Z

That's because if you're not using the int c anywhere, compiler will just optimize it out and there is no crash, which is something it can't do when you make it volatile.

I take it the issues here are solved, thanks for testing!

As for Esp32, I checked its decoder, but it should already work with any line endings. If you have any trace it does not decode, please report it at https://github.com/platformio/platform-espressif32/issues

ali80 · 2020-05-21T19:28:23Z

DOH, silly me, many thanks, this was a much needed feature on PIO.

ali80 · 2020-05-21T19:32:45Z

@Tasssadar many of the faults are occasional and would not repeat that often, also most of the time there is no serial connection to the device
Is there a way to use your tool to parse from an already recorded file instead of Serial stream?

ivankravets changed the title ~~Add stack trace decoding~~ Add stack trace decoding [$35] Apr 30, 2017

ivankravets added the bounty label Apr 30, 2017

LaurentLouf mentioned this issue Nov 28, 2017

Stack Trace Decoding #58

Closed

ivankravets mentioned this issue Aug 3, 2018

Add EspExceptionDecoder platformio/platformio-vscode-ide#225

Closed

ivankravets mentioned this issue Feb 23, 2019

How I can use Esp Exception Decoder on PlatformIO? #126

Closed

ivankravets assigned valeros Jun 22, 2019

mcspr mentioned this issue Jul 17, 2019

Add a CLI version of ESPExceptionDecoder esp8266/Arduino#6310

Closed

jwyse mentioned this issue Oct 28, 2019

"AP mode failed" since commit 5197999 alf45tar/PedalinoMini#40

Closed

Tasssadar mentioned this issue Feb 15, 2020

Add ESP crash trace decoding to monitor platformio/platformio-core#3383

Merged

2 tasks

Tasssadar added a commit to RoboticsBrno/platformio-core that referenced this issue Feb 15, 2020

Implement mechanism for adding platform filters into miniterm

6d77765

Updates platformio/platform-espressif8266#31

Tasssadar added a commit to RoboticsBrno/platform-espressif8266 that referenced this issue Feb 21, 2020

Esp8266ExceptionDecoder: add monitor filter to decode exception traces

e1741c3

Add --filter=esp8266_exception_decoder to monitor_flags to use it. Fixes platformio#31

Tasssadar mentioned this issue Feb 21, 2020

Esp8266ExceptionDecoder: add monitor filter to decode exception traces #197

Merged

ivankravets closed this as completed in #197 Mar 11, 2020

ivankravets reopened this Mar 16, 2020

This was referenced Mar 16, 2020

Esp8266ExceptionDecoder: fix crash on Windows with Python <3;3.7> #201

Merged

Esp32ExceptionDecoder: fix crash on Windows with Python <3;3.7> platformio/platform-espressif32#303

Merged

ivankravets closed this as completed in #201 Mar 16, 2020

Tasssadar mentioned this issue May 21, 2020

Esp8266ExceptionDecoder compatibility fixes #211

Merged

stavultras mentioned this issue Jun 4, 2020

ESP8266_Exception_decoder filter does not work as expected #207

Open

Add stack trace decoding [$35] #31

Add stack trace decoding [$35] #31

Comments

diimdeep commented Apr 12, 2017 • edited Loading

psy0rz commented Apr 24, 2017

BBBSnowball commented Jul 22, 2017

osos commented Jan 21, 2018 • edited Loading

JarekParal commented Jun 27, 2018

JarekParal commented Jun 28, 2018

Misiu commented Aug 17, 2018

HenrikSte commented Nov 14, 2018

stritti commented Apr 2, 2019

edhedges commented Apr 25, 2019 • edited Loading

bzeeman commented May 8, 2019

HollisTech commented Dec 19, 2019

ivankravets commented Mar 13, 2020

cocus commented Mar 16, 2020 • edited Loading

ivankravets commented Mar 16, 2020

Tasssadar commented Mar 16, 2020

Tasssadar commented Mar 16, 2020

ivankravets commented Mar 16, 2020

sblantipodi commented Apr 23, 2020

Pablo2048 commented Apr 24, 2020

ali80 commented May 21, 2020

ivankravets commented May 21, 2020

ali80 commented May 21, 2020 • edited Loading

Tasssadar commented May 21, 2020

ali80 commented May 21, 2020

Tasssadar commented May 21, 2020 • edited Loading

ivankravets commented May 21, 2020

ali80 commented May 21, 2020

ali80 commented May 21, 2020

Tasssadar commented May 21, 2020

ivankravets commented May 21, 2020

ali80 commented May 21, 2020

Tasssadar commented May 21, 2020

ivankravets commented May 21, 2020

ali80 commented May 21, 2020

ali80 commented May 21, 2020

Tasssadar commented May 21, 2020

ali80 commented May 21, 2020

ali80 commented May 21, 2020 • edited Loading

diimdeep commented Apr 12, 2017 •

edited

Loading

osos commented Jan 21, 2018 •

edited

Loading

edhedges commented Apr 25, 2019 •

edited

Loading

cocus commented Mar 16, 2020 •

edited

Loading

ali80 commented May 21, 2020 •

edited

Loading

Tasssadar commented May 21, 2020 •

edited

Loading

ali80 commented May 21, 2020 •

edited

Loading