Problem with notifications. #468

tecufanujacu · 2017-02-25T15:57:38Z

I have a lot of problems with notifications:

if i disable the gms notifications I do not receive any notifications with WebSocket, why? what should I check to solve this problem?
using the gms notifications for the most time I do not receive them, how can i debug this?
i also noticed the issue Mobile unable to connect to the event server #403

From my server:

$ sudo zmdc.pl status zmeventnotification.pl
'zmeventnotification.pl' running since 17/02/25 12:58:14, pid = 3529

$ sudo tail -f /var/log/syslog | grep zmeventnotification
Feb 25 16:41:52 N40L zmeventnotification[3529]: INF [Reloading Monitors...]
Feb 25 16:41:52 N40L zmeventnotification[3529]: INF [Loading monitors]
Feb 25 16:46:57 N40L zmeventnotification[3529]: INF [Total event client connections: 1]
Feb 25 16:46:57 N40L zmeventnotification[3529]: INF [Reloading Monitors...]
Feb 25 16:46:57 N40L zmeventnotification[3529]: INF [Loading monitors]

and when I start zmNinja on my phone:

$ sudo tail -f /var/log/syslog | grep zmeventnotification

Feb 25 16:41:52 N40L zmeventnotification[3529]: INF [Reloading Monitors...]
Feb 25 16:41:52 N40L zmeventnotification[3529]: INF [Loading monitors]
Feb 25 16:46:57 N40L zmeventnotification[3529]: INF [Total event client connections: 1]
Feb 25 16:46:57 N40L zmeventnotification[3529]: INF [Reloading Monitors...]
Feb 25 16:46:57 N40L zmeventnotification[3529]: INF [Loading monitors]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [got a websocket connection from 192.168.1.1 (1) active connections]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [Websocket remotely disconnected from 192.168.1.1]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [got a websocket connection from 192.168.1.1 (1) active connections]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [Websockets: New Connection Handshake requested from 192.168.1.1:46667 state=pending auth]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [Correct authentication provided by 192.168.1.1]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [Duplicate token found, marking for deletion]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [Storing token ...SVSdQVaUfN,monlist:-1,intlist:-1,pushstate:enabled]
Feb 25 16:51:05 N40L zmeventnotification[3529]: INF [SaveTokens called with:monlist=-1, intlist=-1, platform=android, push=enabled]
Feb 25 16:51:06 N40L zmeventnotification[3529]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 16:51:06 N40L zmeventnotification[3529]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 16:51:07 N40L zmeventnotification[3529]: INF [Pushproxy registration success ]
Feb 25 16:51:07 N40L zmeventnotification[3529]: INF [Duplicate token found, marking for deletion]
Feb 25 16:51:07 N40L zmeventnotification[3529]: INF [Storing token ...SVSdQVaUfN,monlist:1,intlist:0,pushstate:enabled]
Feb 25 16:51:07 N40L zmeventnotification[3529]: INF [SaveTokens called with:monlist=1, intlist=0, platform=android, push=enabled]
Feb 25 16:51:07 N40L zmeventnotification[3529]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 16:51:07 N40L zmeventnotification[3529]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]

Is everything ok? Could you help me to solve my problems?

My server config:
OpenMediaVault 2.2.13 - based on Debian 7
ZoneMinder 1.30.2

My phone:
Android 6.0.1 with CyanogenMod13 without gapps and with microG GmsCore
zmNinja 1.2.38

On the router I also opened in TCP the port 9000 for the IP of my server.

pliablepixels · 2017-02-25T18:17:04Z

I can't say. I just tried on my phone and I don't see any problems.

What might help is this:
a) Stop zmeventserver
b) Edit /etc/private/tokens.txt and clear it (don't delete the file)
c) Start zmeventserver from command line (easier to debug)
e) Start a terminal and monitor logs

d) Start zmNinja
e) Go to Event server settings, make sure you have enabled everything and not put restrictions on minimum duration for alarms for any monitors
f) Press save a save times (2-3)

g) Make sure you see the event server registration in the logs
h) quit zmNinja (kill it, not exit)

i) Open up ZM, force an alarm. Make sure the monitor is NOT in record or monitor - it must be in mocord or modect - events are not registered otherwise

j) Make sure you see logs in zmeventserver that the event was received - if not, there is a problem (not sure what, but we can get there later)

Post logs.

tecufanujacu · 2017-02-25T20:55:18Z

Ok, just a question, for:
e) everything enabled in Event Server setting do you mean that I have also to enable the voice "use only websockets"? Anyway to take a log I also enabled this voice.

Here the log.

When I start zmeventserver:

sudo tail -f /var/log/syslog | grep zmeventnotification
Feb 25 22:07:37 N40L zmeventnotification[3922]: INF [Push enabled via PushProxy]
Feb 25 22:07:37 N40L zmeventnotification[3922]: INF [direct APNS disabled]
Feb 25 22:07:37 N40L zmeventnotification[3922]: INF [Event Notification daemon v 0.92 starting]
Feb 25 22:07:37 N40L zmeventnotification[3922]: INF [Total event client connections: 1]
Feb 25 22:07:37 N40L zmeventnotification[3922]: INF [Reloading Monitors...]
Feb 25 22:07:37 N40L zmeventnotification[3922]: INF [Loading monitors]
Feb 25 22:07:37 N40L zmeventnotification[3922]: INF [Checking https://185.124.74.36:8801 reachability...]
Feb 25 22:07:40 N40L zmeventnotification[3922]: INF [PushProxy https://185.124.74.36:8801 is reachable.]
Feb 25 22:07:40 N40L zmeventnotification[3922]: INF [About to start listening to socket]
Feb 25 22:07:40 N40L zmeventnotification[3922]: INF [Secure WS(WSS) is enabled...]
Feb 25 22:07:40 N40L zmeventnotification[3922]: INF [Web Socket Event Server listening on port 9000]

When I start zmNinja to save setting and then I kill it:

sudo tail -f /var/log/syslog | grep zmeventnotification
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [got a websocket connection from 192.168.1.1 (1) active connections]
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [Websocket remotely disconnected from 192.168.1.1]
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [got a websocket connection from 192.168.1.1 (1) active connections]
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [Websockets: New Connection Handshake requested from 192.168.1.1:46868 state=pending auth]
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [Correct authentication provided by 192.168.1.1]
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:-1,intlist:-1,pushstate:disabled]
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=-1, intlist=-1, platform=android, push=disabled]
Feb 25 22:09:34 N40L zmeventnotification[3922]: INF [APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN not found, creating new record with monlist=]
Feb 25 22:09:35 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:09:35 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:1,intlist:0,pushstate:disabled]
Feb 25 22:09:35 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=1, intlist=0, platform=android, push=disabled]
Feb 25 22:09:35 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: ]
Feb 25 22:09:35 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:09:36 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:09:50 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:-1,intlist:-1,pushstate:disabled]
Feb 25 22:09:50 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=-1, intlist=-1, platform=android, push=disabled]
Feb 25 22:09:50 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:09:50 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:09:52 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:09:52 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:1,intlist:0,pushstate:disabled]
Feb 25 22:09:52 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=1, intlist=0, platform=android, push=disabled]
Feb 25 22:09:52 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:09:52 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:09:53 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:09:57 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:-1,intlist:-1,pushstate:disabled]
Feb 25 22:09:57 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=-1, intlist=-1, platform=android, push=disabled]
Feb 25 22:09:57 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:09:57 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:09:58 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:09:58 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:1,intlist:0,pushstate:disabled]
Feb 25 22:09:58 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=1, intlist=0, platform=android, push=disabled]
Feb 25 22:09:58 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:09:58 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:10:00 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:10:04 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:-1,intlist:-1,pushstate:disabled]
Feb 25 22:10:04 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=-1, intlist=-1, platform=android, push=disabled]
Feb 25 22:10:04 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:10:04 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:10:05 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:10:05 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:1,intlist:0,pushstate:disabled]
Feb 25 22:10:05 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=1, intlist=0, platform=android, push=disabled]
Feb 25 22:10:05 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:10:05 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:10:06 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:10:11 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:-1,intlist:-1,pushstate:disabled]
Feb 25 22:10:11 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=-1, intlist=-1, platform=android, push=disabled]
Feb 25 22:10:11 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:10:11 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:10:12 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:10:12 N40L zmeventnotification[3922]: INF [Storing token ...SVSdQVaUfN,monlist:1,intlist:0,pushstate:disabled]
Feb 25 22:10:12 N40L zmeventnotification[3922]: INF [SaveTokens called with:monlist=1, intlist=0, platform=android, push=disabled]
Feb 25 22:10:12 N40L zmeventnotification[3922]: INF [token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN matched, previously stored monlist is: 1]
Feb 25 22:10:12 N40L zmeventnotification[3922]: INF [overwriting APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN monlist with:1]
Feb 25 22:10:13 N40L zmeventnotification[3922]: INF [Pushproxy registration success ]
Feb 25 22:10:44 N40L zmeventnotification[3922]: INF [Websocket remotely disconnected from 192.168.1.1]
Feb 25 22:10:44 N40L zmeventnotification[3922]: INF [NOT Marking 192.168.1.1 for deletion as token APA91bF9e0kf-Nm3UAbKvJMohT__fZPAESLPXYV5Ys7goBGD2Rz1p67bX4PIaYhI2rWOn8r52qMBaGYAG6jJDONZ_EgQORVTovl6wWada31pA5nVKHSqiI42QUXqfK4X1sSVSdQVaUfN active]

When there is a new event:

sudo tail -f /var/log/syslog | grep zmeventnotification
Feb 25 22:10:55 N40L zmeventnotification[3922]: INF [New event 1009 reported for IPCAM]
Feb 25 22:10:55 N40L zmeventnotification[3922]: INF [Broadcasting new events to all 2 websocket clients]
Feb 25 22:10:55 N40L zmeventnotification[3922]: INF [Checking alarm rules for (unknown)]
Feb 25 22:10:55 N40L zmeventnotification[3922]: INF [Monitor 1 event: last time not found, so sending]
Feb 25 22:10:55 N40L zmeventnotification[3922]: INF [Checking alarm rules for 0.0.0.0:0 token ending in:...SVSdQVaUfN]
Feb 25 22:10:56 N40L zmeventnotification[3922]: INF [Monitor 1 event: last time not found, so sending]
Feb 25 22:10:56 N40L zmeventnotification[3922]: INF [0.0.0.0-sending over websockets

but I haven't received a notification on my phone.
What do you think?

pliablepixels · 2017-02-25T21:26:38Z

can you force a push now? Also please let me know the last part of your IP address so I can trace it

pliablepixels · 2017-02-25T21:28:22Z

Ok, I just saw a push on my server

{"data": {"message": "Alarms: SALONE (1013) ", "soundname": "blop", "mid": "1"}, "registration_ids": ["APA91bF9e0kf-Nm3UAbKvJMohT__<deleted>"]}

Is this yours?

IF the above (SALONE) belongs to you, then everything is working on the server side. That means only one thing - push is not configured properly on your app. Since you compiled from source, its hard to know if you have every dependency installed and configured correctly for push. My suggestion is to recompile the app and making sure the push plugin is added using cordova plugin add phonegap-plugin-push --variable SENDER_ID=710936220256

tecufanujacu · 2017-02-25T22:00:59Z

Yes, SALONE is my ipcam and with gms push sometime i receive notifications and sometime not, just now I renabled in zmNinja the gms push notifications and at the moment I'm receiving them but when I took the logs them were disabled and i tried using only the websockets and with the websockets I never receive notifications.

Anyway tomorrow I'll recompile the app with your SENDER_IT but this should be unrelated with the websockets, right? and then why I do not receive notifications via WebSockets?

pliablepixels · 2017-02-25T22:06:07Z

web sockets will only work if zmninja is running.

tecufanujacu · 2017-02-26T16:08:44Z

I don't think that the problem could be related at the fact that I compiled the app by myself, if there has been some error I did not receive any notification; anyway I'm starting compile a new version but if sometime I receive the push notification I don't think that using a different SENDER_IT will do any difference.

P.S.: I just discovered that at building time I have automatically:
cordova plugin add phonegap-plugin-push --variable SENDER_ID="710936220256"
so my self compiled app is using the correct SENDER_ID, any other idea for the missing notifications?

pliablepixels · 2017-02-27T00:06:40Z

Based on what you've sent me so far, zmeventserver gets the event, it pushes it out, the push server gets the event and pushes it out. I can't think of anything besides thinking the problem is between GCM and the app you have. You're going to have to run adb logcat to see if the pushes arrive at all to the phone.

tecufanujacu · 2017-02-27T17:03:20Z

Ok, I'll investigate and I'll report here as soon there is some news.
A question, do you think is it possible to run the server for the push notifications on my server?

pliablepixels · 2017-02-27T20:14:16Z

You won't be able to run my server as-is without generating your own server side SSL certificates which are provided by Apple (and I can't offer them due to security reasons)

That being said:

You can opt to connect zmeventserver with a 3rd party push server. See here for how a user configured his system to use pushover instead of my server.
You can run my server as well, it is here but you are on your own with that - I won't be able to provide support, primarily due to time constraints. Its basically a hacked up version of an unsupported software called air notifier.

Note that in both cases, you will need to (as far as I know) sign up as an Apple Developer (costs $99 a year) so you can generate and download push certificates. As of late 2016, Apple also offers a token based system (like GCM) which is much better, but my server doesn't support it (because I forked another open source that did not). Its very likely 3rd party servers like pushover/pushwoosh etc will.

tecufanujacu · 2017-02-28T01:52:08Z

Thanks alot for your detailed explanation.
About the missing notification I saw that when I don not more receive them everything become normal If I restart the server, what could be the problem?

tecufanujacu · 2017-03-02T15:33:15Z

Ok I solved every problem with ZoneMinder but zmninja often fails to receive notifications.
I do not think that this is an issue related at the compilation, anyway I'd like to try the apk compiled by you, considering that I don't use the Gapps and its services as the PlayStore is there other way (as bitcoin or PayPal) to pay you and receive the apk?

pliablepixels · 2017-03-02T15:40:45Z

So when you say zmNinja fails to get events, do you see the events being caught by zmeventserver ?

tecufanujacu · 2017-03-02T15:47:29Z

Yes, for example just now I'm having this:

Mar 2 16:43:46 N40L zmeventnotification[1348]: INF [New event 341 reported for SALONE]
Mar 2 16:43:46 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:43:46 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:43:46 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 10 is >= interval of ]
Mar 2 16:44:01 N40L zmeventnotification[1348]: INF [New event 342 reported for SALONE]
Mar 2 16:44:01 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:44:01 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:44:01 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 15 is >= interval of ]
Mar 2 16:44:11 N40L zmeventnotification[1348]: INF [New event 343 reported for SALONE]
Mar 2 16:44:11 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:44:11 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:44:11 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 10 is >= interval of ]
Mar 2 16:44:31 N40L zmeventnotification[1348]: INF [New event 344 reported for SALONE]
Mar 2 16:44:31 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:44:31 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:44:31 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 20 is >= interval of ]
Mar 2 16:44:46 N40L zmeventnotification[1348]: INF [Total event client connections: 1]
Mar 2 16:44:46 N40L zmeventnotification[1348]: INF [Reloading Monitors...]
Mar 2 16:44:46 N40L zmeventnotification[1348]: INF [Loading monitors]
Mar 2 16:44:51 N40L zmeventnotification[1348]: INF [Checking https://185.124.74.36:8801 reachability...]
Mar 2 16:44:52 N40L zmeventnotification[1348]: INF [PushProxy https://185.124.74.36:8801 is reachable.]
Mar 2 16:44:52 N40L zmeventnotification[1348]: INF [New event 345 reported for SALONE]
Mar 2 16:44:52 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:44:52 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:44:52 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 21 is >= interval of ]
Mar 2 16:45:06 N40L zmeventnotification[1348]: INF [New event 346 reported for SALONE]
Mar 2 16:45:06 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:45:06 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:45:06 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 14 is >= interval of ]
Mar 2 16:45:56 N40L zmeventnotification[1348]: INF [New event 347 reported for SALONE]
Mar 2 16:45:56 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:45:56 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:45:56 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 50 is >= interval of ]
Mar 2 16:46:06 N40L zmeventnotification[1348]: INF [New event 348 reported for SALONE]
Mar 2 16:46:06 N40L zmeventnotification[1348]: INF [Broadcasting new events to all 1 websocket clients]
Mar 2 16:46:06 N40L zmeventnotification[1348]: INF [Checking alarm rules for (unknown)]
Mar 2 16:46:06 N40L zmeventnotification[1348]: INF [Monitor 1 event: sending this out as 10 is >= interval of ]

without any push notifications on two phones.

P.S.: my problems with ZoneMinder was caused by the modded version with ngnix support for OpenMediaVault that is now really old and with a lot of problem. Now I'm on Debian 8 with minimal install and ZoneMinder works without any problem.

About zmNinja I noticed that when I stop receive notifications if I enter in the app I start receive notifications for a while, same behavior on two phones. Further this I noticed that zmNinja has a lot of problems in saving settings, sometime it crashes at exit and also the notification icon is really strange, a blank square?

pliablepixels · 2017-03-02T15:59:41Z

Can you send me an email - pliablepixels@gmail

tecufanujacu · 2017-03-02T16:09:30Z

Just done, that strange address it's me.

pliablepixels · 2017-03-02T20:15:31Z

I have a strong feeling its related to the fact that you don't have a google play account. I don't know if GCM will route messages to a phone that it doesn't have a registration for. I actually think the only pushes you are receiving are over web sockets (not GCM) - that is when the app is running

Try this -

Connect your phone to your computer, and run
adb logcat | grep 'PushPlugin\|GCM'

Start zmNinja, make sure it registers with a push token and then kill zmNinja (from task bar). Keep logcat on.

Force a push

You should see logs like this on logcat. If not, I strongly suspect your lack of a play account being the key reason. Let me know if you see logs like this when you force a push

D/PushPlugin_GCMIntentService(15159): onMessage - from: 710936220256
D/PushPlugin_GCMIntentService(15159): normalize extras
D/PushPlugin_GCMIntentService(15159): key = soundname
D/PushPlugin_GCMIntentService(15159): replace key soundname with sound
D/PushPlugin_GCMIntentService(15159): key = google.sent_time
D/PushPlugin_GCMIntentService(15159): replace key google.sent_time with google.sent_time
D/PushPlugin_GCMIntentService(15159): key = mid
D/PushPlugin_GCMIntentService(15159): replace key mid with mid
D/PushPlugin_GCMIntentService(15159): key = google.message_id
D/PushPlugin_GCMIntentService(15159): replace key google.message_id with google.message_id
D/PushPlugin_GCMIntentService(15159): key = message
D/PushPlugin_GCMIntentService(15159): replace key message with message
D/PushPlugin_GCMIntentService(15159): no locale found for key = message, error Value Alarms of type java.lang.String cannot be converted to JSONObject
D/PushPlugin_GCMIntentService(15159): key = collapse_key
D/PushPlugin_GCMIntentService(15159): replace key collapse_key with collapse_key
D/PushPlugin_GCMIntentService(15159): background
D/PushPlugin_GCMIntentService(15159): message =[Alarms: Garage (72302) ]
D/PushPlugin_GCMIntentService(15159): title =[null]
D/PushPlugin_GCMIntentService(15159): contentAvailable =[null]
D/PushPlugin_GCMIntentService(15159): create notification
E/PushPlugin_GCMIntentService(15159): Number format exception - Error parsing notId: Invalid int: "null"
D/PushPlugin_GCMIntentService(15159): stored icon=ic_stat_notification
D/PushPlugin_GCMIntentService(15159): stored iconColor=null
D/PushPlugin_GCMIntentService(15159): stored sound=true
D/PushPlugin_GCMIntentService(15159): stored vibrate=true
D/PushPlugin_GCMIntentService(15159): using icon from plugin options
D/PushPlugin_GCMIntentService(15159): android.resource://com.pliablepixels.zmninja_pro/raw/blop
D/PushPlugin_GCMIntentService(15159): create actions: with in-line

tecufanujacu · 2017-03-02T23:52:19Z

I don't know if GCM will route messages to a phone that it doesn't have a registration for.

This is what microG GMS Core does, it really is a great project.
microG Project on github
microG Project on XDA

I have a strong feeling its related to the fact that you don't have a google play account.

Anyway I had already checked in the past and everything is working with microG:

D/GmsGcmMcsSvc(19389): Target: ResolveInfo{d92f1e2 com.pliablepixels.zmninja_pro/com.google.android.gms.gcm.GcmReceiver m=0x108000}
D/PushPlugin_GCMIntentService(19112): onMessage - from: 710936220256
D/PushPlugin_GCMIntentService(19112): normalize extras
D/PushPlugin_GCMIntentService(19112): key = soundname
D/PushPlugin_GCMIntentService(19112): replace key soundname with sound
D/PushPlugin_GCMIntentService(19112): key = mid
D/PushPlugin_GCMIntentService(19112): replace key mid with mid
D/PushPlugin_GCMIntentService(19112): key = message
D/PushPlugin_GCMIntentService(19112): replace key message with message
D/PushPlugin_GCMIntentService(19112): no locale found for key = message, error Value Alarms of type java.lang.String cannot be converted to JSONObject
D/PushPlugin_GCMIntentService(19112): key = collapse_key
D/PushPlugin_GCMIntentService(19112): replace key collapse_key with collapse_key
D/PushPlugin_GCMIntentService(19112): background
D/PushPlugin_GCMIntentService(19112): message =[Alarms: SALONE (518) ]
D/PushPlugin_GCMIntentService(19112): title =[null]
D/PushPlugin_GCMIntentService(19112): contentAvailable =[null]
D/PushPlugin_GCMIntentService(19112): create notification
E/PushPlugin_GCMIntentService(19112): Number format exception - Error parsing notId: Invalid int: "null"
D/PushPlugin_GCMIntentService(19112): stored icon=ic_stat_notification
D/PushPlugin_GCMIntentService(19112): stored iconColor=null
D/PushPlugin_GCMIntentService(19112): stored sound=true
D/PushPlugin_GCMIntentService(19112): stored vibrate=true
D/PushPlugin_GCMIntentService(19112): using icon from plugin options
D/PushPlugin_GCMIntentService(19112): android.resource://com.pliablepixels.zmninja_pro/raw/blop
D/PushPlugin_GCMIntentService(19112): create actions: with in-line

pliablepixels · 2017-03-03T00:12:14Z

If I had time at hand I'd like to help but I don't. I really won't have the time right now to help debug what microGMS does or does not do. However if you do figure out how to solve it in happy to accept PR.

tecufanujacu · 2017-03-03T00:48:03Z

No problem, I have other 3 phones and I can restore one of them at the factory conditions with the PlayServices to test zmNinja but I don't think that my problem is caused by microG, I use it from almost two years and I have never had problem with push notifications, for example I use without problems Signal and WhatsApp.

Anyway I'm really a lot confused, I restarted my server almost an hour ago and from then I don't have any problem. This is the second time that I noticed this and with two different configurations, some days ago with OpenMediaVault and ZoneMinder 1.29 and 1.30, and now with Debian 8 and ZoneMinder 1.29.

My server is on 24h/24 and 7d/7 and seems that if I reboot it when zmNinja fails to get notifications, then everything return to work without problems.

So when I start the server everything works without problems for a while but after some time I only have notifications when zmNinja is opened. In this case If I reboot the server I restart to receive notifications without doing nothing on zmNinja and also if the app isn't opened.

Could this be possible? ZoneMinder is properly configured and I don't have any warning or error in its logs.

At the moment on my server there only is a clean install of Debian 8 with only ZoneMinder 1.29.

Everything is really strange, I know, and at the moment I don't have any idea on how to debug this problem. For now I'll try to use a phone with Play Services instead of microG to be sure that the problem isn't it.

tecufanujacu · 2017-03-03T00:54:23Z

Confirmed, just now zmNinja stopped receiving push notifications, I only rebooted the server (without touching the phone, not entering or exiting from the app) and zmNinja restarted receiving the notifications.

What can I do? What could be the problem?

pliablepixels · 2017-03-03T01:04:28Z

Okay, lets see if we can be a bit more specific: (lets not use the term server - I get confused what that means)

ZM = ZoneMinder
ZMES = Event Server (zmeventserver.pl)

Questions:

When push does not work:
a) Did you make sure ZMES is still catching the event?
b) Did you do adb logcat | grep 'PushPlugin\|GCM' on the phone at_that_time and make sure the GCM layer of the phone got it?

pliablepixels · 2017-03-03T01:27:15Z

Also, if you are available tomorrow morning, please join irc.freenode.net #zoneminder - we can do some parallel debugging (I usually am av. in that channel after 9AM US ET)

tecufanujacu · 2017-03-03T01:28:35Z

Sure, excuse me.
When I told that I have restarted the server I meant the physical server (computer\nas), so Debian with ZM.

About the questions:

yes before rebooting ZMES was running
when push doesn't work:
2.a) ZMES is still catching the events (as the logs that I sent you today)
2.b) When zmNinja is closed and I don't receive push notifications I don't have nothing with adb logcat | grep 'PushPlugin|GCM'
2.c) When I'm not receiving push notifications and I open zmNinja then if it is opened I receive the notifications and in this case I have a normal adb logcat | grep 'PushPlugin|GCM'
2.d) When I'm not receiving notifications and I reboot Debian with ZoneMinder seems that everything restarts to work

The next time instead of rebooting Debian I'll try to reboot only ZoneMinder.

tecufanujacu · 2017-03-03T01:31:13Z

Tomorrow I'll try but I'm not sure, but this weekend I'm surely available at every hour.

pliablepixels · 2017-03-03T01:33:26Z

Okay, this is the core problem (please make sure you are typing adb logcat | grep 'PushPlugin\|GCM' NOT `adb logcat | grep 'PushPlugin|GCM' (slash - we are using grep OR). Assuming you are, that is the problem - that the phone's GCM layer is getting nothing.

IF its not, then that means its getting lost somewhere between my server to Google to your phone. Given you are using a non standard GCM layer, its impossible for me to know what is going on. And since I don't use what you are using, I can't debug it either. If GCM is not getting it, the app won't get it.

The fact that it works for a while and them stops - maybe google throttles the messages after a while if it detects something wrong with registration - hard to know

pliablepixels · 2017-03-03T01:44:40Z

Also, I'm recording logs to a file - please see if you can generate a situation where push does not work and then please let me know the IP address -I'll see what logs I see at the server for a failed push

tecufanujacu · 2017-03-03T01:45:41Z

Sure, the command is with the OR, I wrongly wrote in the previous post.
Anyway OK, I have to test on a phone with the PlayServices but then why if I restart Debian with ZoneMinder everything return to work without problem for a while? Isn't strange? What the OS reboot does?

Just a question, can I install zmNinja on Android 4.4.4, the stock rom of the phone that i would use for test with PlayServices?

Anyway thanks a lot for your patience and your time and sorry if it is hard to understand me for my bad english.

tecufanujacu · 2017-03-03T01:52:32Z

Unfortunately at the moment everything is working -.-" and I think that i have to go to sleep because here to me is really to late.
Anyway, as soon I'll have missing notifications, I'll get logs from zmeventserver.pl and from android with adb and I'll sign my ip.

tecufanujacu · 2017-03-03T14:39:03Z

Ok, there are news, I'm now testing everything with 3 phones, 2 with microg and one with google play services and they have the same behavior.

Further than this I noticed that is enough restarting ZoneMinder to solve the problem for a while.

Just few minutes ago my phones have stopped receive notifications, forcing an event from ZoneMinder I did not have any notifications and these this was the logs:


$ sudo tail -f /var/log/syslog | grep zmeventnotification
Mar  3 15:17:36 N40L zmeventnotification[1336]: INF [New event 795 reported for SALONE]
Mar  3 15:17:36 N40L zmeventnotification[1336]: INF [Broadcasting new events to all 1 websocket clients]
Mar  3 15:17:36 N40L zmeventnotification[1336]: INF [Checking alarm rules for (unknown)]
Mar  3 15:17:36 N40L zmeventnotification[1336]: INF [Monitor 1 event: sending this out as 95 is >= interval of ]

$ sudo zmdc.pl status zmeventnotification.pl
'zmeventnotification.pl' running since 17/03/03 01:51:38, pid = 1336

Then I simply restarted ZoneMinder:
$ sudo service zoneminder restart
and everything restart to works, and all the 3 phones are now receiving notifications, here the logs:

$ sudo tail -f /var/log/syslog | grep zmeventnotification
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [New event 807 reported for SALONE]
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [Broadcasting new events to all 5 websocket clients]
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [Checking alarm rules for  token ending in:...MVGzOvZ8iM]
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [Checking alarm rules for  token ending in:...YzOSiFP6Bd]
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [Monitor 1 event: sending this out as 40 is >= interval of 0]
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [Sending notification over PushProxy]
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [Pushproxy push message success ]
Mar  3 15:33:51 N40L zmeventnotification[4897]: INF [Checking alarm rules for (unknown)]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Monitor 1 event: sending this out as 41 is >= interval of ]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Checking alarm rules for  token ending in:...Ey6795SRvD]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Monitor 1 event: sending this out as 41 is >= interval of 0]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Sending notification over PushProxy]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Pushproxy push message success ]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Checking alarm rules for 192.168.1.1:42744 token ending in:...iINWms_vTW]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Monitor 1 event: sending this out as 40 is >= interval of 0]
Mar  3 15:33:52 N40L zmeventnotification[4897]: INF [Sending notification over PushProxy]
Mar  3 15:33:53 N40L zmeventnotification[4897]: INF [Pushproxy push message success ]
Mar  3 15:33:53 N40L zmeventnotification[4897]: INF [192.168.1.1-sending supplementary data over websockets]


$ sudo zmdc.pl status zmeventnotification.pl
'zmeventnotification.pl' running since 17/03/03 15:20:34, pid = 4897

My IP at the moment of these log was:
79.XX.XXX.200

So the problem seems to be the ZMES Event Server (zmeventserver.pl). The next time that I start to missing the push notifications I will only stop and restart it to be sure of this:
sudo zmdc.pl stop zmeventnotification.pl
sudo zmdc.pl start zmeventnotification.pl

pliablepixels · 2017-03-03T14:44:03Z

So it looks like zmeventserver is catching the event

Mar  3 15:17:36 N40L zmeventnotification[1336]: INF [New event 795 reported for SALONE]

But for whatever reason, it is not sending the push out to the push server (this should be 5 devices)

Mar  3 15:17:36 N40L zmeventnotification[1336]: INF [Broadcasting new events to all 1 websocket clients]

Correct?

tecufanujacu · 2017-03-03T14:54:10Z

Yes, seems that zmeventserver alwasy catchs the events but after a while it stops to sending the push notifications.

A the moment I'm using 3 devices:

a d802, with cm13 (android 6.0.1), with microg and with zmNinja compiled by me
a d802, with cm13 (android 6.0.1), with microg and with zmNinja compiled by you
an xt894 with stock rom motorola Android 4.1.2, with Google Play Services and with zmNinja compiled by me

and they all have the same behavior.

pliablepixels · 2017-03-03T15:04:30Z

Can you check if in /etc/private/token.txt what is the value of push - is it 'enabled' or 'disabled' when push stops working?

tecufanujacu · 2017-03-03T15:15:16Z

Yes, I'll do, but my tokex.txt is really strange, now everything is working and I have

:android:enabled

on only one row, anyway I just sent to you via email my actual tokens.txt.

pliablepixels · 2017-03-03T15:20:58Z

Okay, thanks - saw that file. Lets do this to reach a stable state.

kill all your zmninja apps (3 apps)
stop zmeventsever
edit tokens.txt and remove everything (don't delete file)
start zmeventserver
start zmNinja on 1 phone
Validate token is created in tokens.txt (you may need to save event server settings again, not sure)
repeat 5 & 6 for other 2 zmNinja apps (but please validate first tokens.txt for each)

When done, please email your tokens.txt

tecufanujacu · 2017-03-03T15:28:34Z

I already did this some days ago, anyway I'll retry and I'll send you the new tokens.txt via email in few minutes.

tecufanujacu · 2017-03-03T15:49:53Z

After a while that I sent you the last email I stopped receiving notifications, so I did:
sudo zmdc.pl stop zmeventnotification.pl
sudo zmdc.pl start zmeventnotification.pl

and everything restart working but after this I noticed that now the tokens.txt is a bit different, I'm sending it to you via email.

So the problem is surely zmeventserver.

pliablepixels · 2017-03-03T16:09:21Z

Can you stop zmeventserver, clear your tokens file and download the new version
https://raw.githubusercontent.com/pliablepixels/zmeventserver/master/zmeventnotification.pl

then start and let me know. I want to get to a state where your tokens.txt has 'enabled' for your tokens in all entries

tecufanujacu · 2017-03-03T16:24:26Z

Just done, no success, I have 'enabled' only on one token.
You have an email.

tecufanujacu · 2017-03-04T14:01:32Z

I sent you numerous email, if you want today I'm available also for irc.

pliablepixels · 2017-03-04T22:47:27Z

1) Please stop zmeventserver 2) Empty tokens.txt 3) Please download https://raw.githubusercontent.com/pliablepixels/zmeventserver/dev/zmeventnotification.pl 4) Please run this from command line (Not daemon) 5) It will print logs - (Directly in terminal, not syslog) - please let me know what it prints as you start adding phones

…

On Sat, Mar 4, 2017 at 9:01 AM, tecufanujacu ***@***.***> wrote: I sent you numerous email, if you want today I'm available also for irc. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/pliablepixels/zmNinja/issues/468#issuecomment-284153222>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD7QrtCoxtQuTloJNRy2HIuwvN8DKNaeks5riW69gaJpZM4MMDHF> .

tecufanujacu · 2017-03-05T17:31:59Z

Sorry, I saw this message only now, I'll do now.

pliablepixels · 2017-03-13T13:48:19Z

I am closing this - problem was resolved - there were bugs in zmeventserver that were fixed. Thanks @tecufanujacu

pliablepixels added source build user question labels Feb 25, 2017

pliablepixels removed the source build label Mar 2, 2017

pliablepixels closed this as completed Mar 13, 2017

pliablepixels added bug and removed user question labels Mar 13, 2017

Problem with notifications. #468

Problem with notifications. #468

Comments

tecufanujacu commented Feb 25, 2017

pliablepixels commented Feb 25, 2017

tecufanujacu commented Feb 25, 2017 • edited Loading

pliablepixels commented Feb 25, 2017

pliablepixels commented Feb 25, 2017 • edited Loading

tecufanujacu commented Feb 25, 2017

pliablepixels commented Feb 25, 2017

tecufanujacu commented Feb 26, 2017 • edited Loading

pliablepixels commented Feb 27, 2017

tecufanujacu commented Feb 27, 2017

pliablepixels commented Feb 27, 2017

tecufanujacu commented Feb 28, 2017

tecufanujacu commented Mar 2, 2017 • edited Loading

pliablepixels commented Mar 2, 2017

tecufanujacu commented Mar 2, 2017 • edited Loading

pliablepixels commented Mar 2, 2017

tecufanujacu commented Mar 2, 2017

pliablepixels commented Mar 2, 2017 • edited Loading

tecufanujacu commented Mar 2, 2017 • edited Loading

pliablepixels commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017 • edited Loading

tecufanujacu commented Mar 3, 2017

pliablepixels commented Mar 3, 2017 • edited Loading

pliablepixels commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017

pliablepixels commented Mar 3, 2017 • edited Loading

pliablepixels commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017 • edited Loading

pliablepixels commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017

pliablepixels commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017 • edited Loading

pliablepixels commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017 • edited Loading

pliablepixels commented Mar 3, 2017

tecufanujacu commented Mar 3, 2017 • edited Loading

tecufanujacu commented Mar 4, 2017

pliablepixels commented Mar 4, 2017 via email

tecufanujacu commented Mar 5, 2017

pliablepixels commented Mar 13, 2017

tecufanujacu commented Feb 25, 2017 •

edited

Loading

pliablepixels commented Feb 25, 2017 •

edited

Loading

tecufanujacu commented Feb 26, 2017 •

edited

Loading

tecufanujacu commented Mar 2, 2017 •

edited

Loading

tecufanujacu commented Mar 2, 2017 •

edited

Loading

pliablepixels commented Mar 2, 2017 •

edited

Loading

tecufanujacu commented Mar 2, 2017 •

edited

Loading

tecufanujacu commented Mar 3, 2017 •

edited

Loading

pliablepixels commented Mar 3, 2017 •

edited

Loading

pliablepixels commented Mar 3, 2017 •

edited

Loading

tecufanujacu commented Mar 3, 2017 •

edited

Loading

tecufanujacu commented Mar 3, 2017 •

edited

Loading

tecufanujacu commented Mar 3, 2017 •

edited

Loading

tecufanujacu commented Mar 3, 2017 •

edited

Loading