Merge pull request #96 from Tkd-Alex/analytics

Show analytics of your points mining

.gitignore

@@ -148,4 +148,5 @@ chromedriver*
 cookies/*
 logs/*
 screenshots/*
-htmls/*
+htmls/*
+analytics/*

README.md

@@ -34,10 +34,11 @@ Read more about channels point [here](https://help.twitch.tv/s/article/channel-p
         - [Bet strategy](#bet-strategy)
     - [FilterCondition](#filtercondition)
         - [Example](#example)
-6. 🍪 [Migrating from old repository (the original one)](#migrating-from-old-repository-the-original-one)
-7. 🪟 [Windows](#windows)
-8. 📱 [Termux](#termux)
-9. ⚠️ [Disclaimer](#disclaimer)
+6. 📈 [Analytics](#analytics)
+7. 🍪 [Migrating from old repository (the original one)](#migrating-from-old-repository-the-original-one)
+8. 🪟 [Windows](#windows)
+9. 📱 [Termux](#termux)
+10. ⚠️ [Disclaimer](#disclaimer)
 
 
 ## Community
@@ -61,6 +62,7 @@ If you have any issues or you want to contribute, you are welcome! But please be
 - Auto claim game drops from Twitch inventory [#21](https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues/21) Read more about game drops [here](https://help.twitch.tv/s/article/mission-based-drops)
 - Place the bet / make a prediction and win or lose (🍀) your channel points!
 No browser needed. [#41](https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues/41) ([@lay295](https://github.com/lay295))
+- Analytics chart
 
 ## Logs feature
 ### Full logs
@@ -259,7 +261,7 @@ If you follow so many streamers on Twitch, but you don't want to mine points for
 ```python
 from TwitchChannelPointsMiner import TwitchChannelPointsMiner
 twitch_miner = TwitchChannelPointsMiner("your-twitch-username")
-twitch_miner.mine(followers=True, blacklist=["user1", "user2"])  # Automatic use the followers list OR
+twitch_miner.mine(followers=True, blacklist=["user1", "user2"])  # Blacklist example
 ```
 4. Start mining! `python run.py`
 
@@ -389,6 +391,25 @@ Allowed values for `where` are: `GT, LT, GTE, LTE`
 - If you want to place the bet ONLY if the highest bet is lower than 2000
 `FilterCondition(by=OutcomeKeys.TOP_POINTS, where=Condition.LT, value=2000)`
 
+## Analytics
+We have recently introduced a little frontend where you can show with a chart you points trend. The script will spawn a Flask web-server on your machine where you can select binding address and port.
+The chart provides some annotation to handle the prediction and watch strike events. Usually annotation are used to notice big increase / decrease of points. If you want to can disable annotations.
+On each (x, y) points Its present a tooltip that show points, date time and reason of points gained / lost. This web page was just a funny idea, and it is not intended to use for a professional usage.
+If you want you can toggle the dark theme with the dedicated checkbox.
+
+| Light theme | Dark theme |
+| ----------- | ---------- |
+| ![Light theme](./assets/chart-analytics-light.png) | ![Dark theme](./assets/chart-analytics-dark.png) |
+
+For use this feature just call the `analytics` method before start mining. Read more at: [#96](https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues/96)
+The chart will be autofreshed each `refresh` minutes. If you want to connect from one to second machine that have that webpanel you have to use `0.0.0.0` instead of `127.0.0.1`.
+```python
+from TwitchChannelPointsMiner import TwitchChannelPointsMiner
+twitch_miner = TwitchChannelPointsMiner("your-twitch-username")
+twitch_miner.analytics(host="127.0.0.1", port=5000, refresh=5)   # Analytics web-server
+twitch_miner.mine(followers=True, blacklist=["user1", "user2"])
+```
+
 ## Migrating from an old repository (the original one):
 If you already have a `twitch-cookies.pkl` and you don't want to log in again, please create a `cookies/` folder in the current directory and then copy the .pkl file with a new name `your-twitch-username.pkl`
 ```

TwitchChannelPointsMiner/TwitchChannelPointsMiner.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 
-import copy
 import logging
+import os
 import random
 import signal
 import sys
@@ -10,7 +10,9 @@
 import uuid
 from collections import OrderedDict
 from datetime import datetime
+from pathlib import Path
 
+from TwitchChannelPointsMiner.classes.AnalyticsServer import AnalyticsServer
 from TwitchChannelPointsMiner.classes.entities.PubsubTopic import PubsubTopic
 from TwitchChannelPointsMiner.classes.entities.Streamer import (
     Streamer,
@@ -33,8 +35,10 @@
 #   - chardet.charsetprober - [feed]
 #   - chardet.charsetprober - [get_confidence]
 #   - requests - [Starting new HTTPS connection (1)]
+#   - Flask (werkzeug) logs
 logging.getLogger("chardet.charsetprober").setLevel(logging.ERROR)
 logging.getLogger("requests").setLevel(logging.ERROR)
+logging.getLogger("werkzeug").setLevel(logging.ERROR)
 
 logger = logging.getLogger(__name__)
 
@@ -62,12 +66,16 @@ def __init__(
         username: str,
         password: str = None,
         claim_drops_startup: bool = False,
+        # Settings for logging and selenium as you can see.
         priority: list = [Priority.STREAK, Priority.DROPS, Priority.ORDER],
         # This settings will be global shared trought Settings class
         logger_settings: LoggerSettings = LoggerSettings(),
         # Default values for all streamers
         streamer_settings: StreamerSettings = StreamerSettings(),
     ):
+        Settings.analytics_path = os.path.join(Path().absolute(), "analytics", username)
+        Path(Settings.analytics_path).mkdir(parents=True, exist_ok=True)
+
         self.username = username
 
         # Set as global config
@@ -100,6 +108,12 @@ def __init__(
         for sign in [signal.SIGINT, signal.SIGSEGV, signal.SIGTERM]:
             signal.signal(sign, self.end)
 
+    def analytics(self, host: str = "127.0.0.1", port: int = 5000, refresh: int = 5):
+        http_server = AnalyticsServer(host=host, port=port, refresh=refresh)
+        http_server.daemon = True
+        http_server.name = "Analytics Thread"
+        http_server.start()
+
     def mine(self, streamers: list = [], blacklist: list = [], followers=False):
         self.run(streamers=streamers, blacklist=blacklist, followers=followers)
 
@@ -185,7 +199,9 @@ def run(self, streamers: list = [], blacklist: list = [], followers=False):
                 if streamer.viewer_is_mod is True:
                     streamer.settings.make_predictions = False
 
-            self.original_streamers = copy.deepcopy(self.streamers)
+            self.original_streamers = [
+                streamer.channel_points for streamer in self.streamers
+            ]
 
             # If we have at least one streamer with settings = make_predictions True
             make_predictions = at_least_one_value_in_settings_is(
@@ -276,7 +292,13 @@ def end(self, signum, frame):
         if self.sync_campaigns_thread is not None:
             self.sync_campaigns_thread.join()
 
-        time.sleep(1)
+        # Check if all the mutex are unlocked.
+        # Prevent breaks of .json file
+        for streamer in self.streamers:
+            if streamer.mutex.locked():
+                streamer.mutex.acquire()
+                streamer.mutex.release()
+
         self.__print_report()
 
         sys.exit(0)
@@ -322,7 +344,7 @@ def __print_report(self):
             if self.streamers[streamer_index].history != {}:
                 gained = (
                     self.streamers[streamer_index].channel_points
-                    - self.original_streamers[streamer_index].channel_points
+                    - self.original_streamers[streamer_index]
                 )
                 logger.info(
                     f"{repr(self.streamers[streamer_index])}, Total Points Gained (after farming - before farming): {_millify(gained)}",

TwitchChannelPointsMiner/classes/AnalyticsServer.py

@@ -0,0 +1,62 @@
+import logging
+import os
+from pathlib import Path
+from threading import Thread
+
+from flask import Flask, Response, cli, render_template
+
+from TwitchChannelPointsMiner.classes.Settings import Settings
+
+cli.show_server_banner = lambda *_: None
+logger = logging.getLogger(__name__)
+
+
+def streamers_available():
+    path = Settings.analytics_path
+    return [
+        f
+        for f in os.listdir(path)
+        if os.path.isfile(os.path.join(path, f)) and f.endswith(".json")
+    ]
+
+
+def read_json(streamer):
+    path = Settings.analytics_path
+    streamer = streamer if streamer.endswith(".json") else f"{streamer}.json"
+    return Response(
+        open(os.path.join(path, streamer)) if streamer in streamers_available() else [],
+        status=200,
+        mimetype="application/json",
+    )
+
+
+def index(refresh=5):
+    return render_template(
+        "charts.html",
+        refresh=(refresh * 60 * 1000),
+        streamers=",".join(streamers_available()),
+    )
+
+
+class AnalyticsServer(Thread):
+    def __init__(self, host: str = "127.0.0.1", port: int = 5000, refresh: int = 5):
+        super(AnalyticsServer, self).__init__()
+
+        self.host = host
+        self.port = port
+        self.refresh = refresh
+
+        self.app = Flask(
+            __name__,
+            template_folder=os.path.join(Path().absolute(), "assets"),
+            static_folder=os.path.join(Path().absolute(), "assets"),
+        )
+        self.app.add_url_rule("/", "index", index, defaults={"refresh": refresh})
+        self.app.add_url_rule("/json/<string:streamer>", "json", read_json)
+
+    def run(self):
+        logger.info(
+            f"Analytics running on http://{self.host}:{self.port}/",
+            extra={"emoji": ":globe_with_meridians:"},
+        )
+        self.app.run(host=self.host, port=self.port, threaded=True)

TwitchChannelPointsMiner/classes/WebSocketsPool.py

@@ -176,11 +176,18 @@ def on_message(ws, message):
             if streamer_index != -1:
                 try:
                     if message.topic == "community-points-user-v1":
+                        if message.type in ["points-earned", "points-spent"]:
+                            balance = message.data["balance"]["balance"]
+                            ws.streamers[streamer_index].channel_points = balance
+                            ws.streamers[streamer_index].persistent_series(
+                                event_type=message.data["point_gain"]["reason_code"]
+                                if message.type == "points-earned"
+                                else "Spent"
+                            )
+
                         if message.type == "points-earned":
                             earned = message.data["point_gain"]["total_points"]
                             reason_code = message.data["point_gain"]["reason_code"]
-                            balance = message.data["balance"]["balance"]
-                            ws.streamers[streamer_index].channel_points = balance
                             logger.info(
                                 f"+{earned} → {ws.streamers[streamer_index]} - Reason: {reason_code}.",
                                 extra={
@@ -193,6 +200,9 @@ def on_message(ws, message):
                             ws.streamers[streamer_index].update_history(
                                 reason_code, earned
                             )
+                            ws.streamers[streamer_index].persistent_annotations(
+                                reason_code, f"+{earned} - {reason_code}"
+                            )
                         elif message.type == "claim-available":
                             ws.twitch.claim_bonus(
                                 ws.streamers[streamer_index],
@@ -333,8 +343,18 @@ def on_message(ws, message):
                                         -points["won"],
                                         counter=-1,
                                     )
+
+                                if event_prediction.result["type"] != "LOSE":
+                                    ws.streamers[streamer_index].persistent_annotations(
+                                        event_prediction.result["type"],
+                                        f"{ws.events_predictions[event_id].title}",
+                                    )
                             elif message.type == "prediction-made":
                                 event_prediction.bet_confirmed = True
+                                ws.streamers[streamer_index].persistent_annotations(
+                                    "PREDICTION_MADE",
+                                    f"Decision: {event_prediction.bet.decision['choice']} - {event_prediction.title}",
+                                )
                 except Exception:
                     logger.error(
                         f"Exception raised for topic: {message.topic} and message: {message}",

TwitchChannelPointsMiner/classes/entities/Message.py

@@ -58,7 +58,11 @@ def __get_channel_id(self):
                     else (
                         self.data["channel_id"]
                         if "channel_id" in self.data
-                        else self.topic_user
+                        else (
+                            self.data["balance"]["channel_id"]
+                            if "balance" in self.data
+                            else self.topic_user
+                        )
                     )
                 )
             )

TwitchChannelPointsMiner/classes/entities/Streamer.py

@@ -1,5 +1,9 @@
+import json
 import logging
+import os
 import time
+from datetime import datetime
+from threading import Lock
 
 from TwitchChannelPointsMiner.classes.entities.Bet import BetSettings
 from TwitchChannelPointsMiner.classes.entities.Stream import Stream
@@ -60,6 +64,7 @@ class Streamer(object):
         "raid",
         "history",
         "streamer_url",
+        "mutex",
     ]
 
     def __init__(self, username, settings=None):
@@ -81,6 +86,8 @@ def __init__(self, username, settings=None):
 
         self.streamer_url = f"{URL}/{self.username}"
 
+        self.mutex = Lock()
+
     def __repr__(self):
         return f"Streamer(username={self.username}, channel_id={self.channel_id}, channel_points={_millify(self.channel_points)})"
 
@@ -146,3 +153,44 @@ def drops_condition(self):
             and self.stream.drops_tags is True
             and self.stream.campaigns_ids != []
         )
+
+    # === ANALYTICS === #
+    def persistent_annotations(self, event_type, event_text):
+        event_type = event_type.upper()
+        if event_type in ["WATCH_STREAK", "WIN", "PREDICTION_MADE"]:
+            primary_color = (
+                "#45c1ff"
+                if event_type == "WATCH_STREAK"
+                else ("#ffe045" if event_type == "PREDICTION_MADE" else "#54ff45")
+            )
+            data = {
+                "borderColor": primary_color,
+                "label": {
+                    "style": {"color": "#000", "background": primary_color},
+                    "text": event_text,
+                },
+            }
+            self.__save_json("annotations", data)
+
+    def persistent_series(self, event_type="Watch"):
+        self.__save_json("series", event_type=event_type)
+
+    def __save_json(self, key, data={}, event_type="Watch"):
+        # https://stackoverflow.com/questions/4676195/why-do-i-need-to-multiply-unix-timestamps-by-1000-in-javascript
+        # data.update({"x": round(time.time() * 1000)})
+        now = datetime.now().replace(microsecond=0)
+        data.update({"x": round(datetime.timestamp(now) * 1000)})
+
+        if key == "series":
+            data.update({"y": self.channel_points})
+            if event_type is not None:
+                data.update({"z": event_type.replace("_", " ").title()})
+
+        fname = os.path.join(Settings.analytics_path, f"{self.username}.json")
+        with self.mutex:
+            json_data = json.load(open(fname, "r")) if os.path.isfile(fname) else {}
+            if key not in json_data:
+                json_data[key] = []
+
+            json_data[key].append(data)
+            json.dump(json_data, open(fname, "w"), indent=4)