Stabilize the connector and add monitoring #70

alimosaed · 2024-11-29T04:14:48Z

What is the purpose of this change?
This change covers a couple of issues and features:
1. Implement a monitoring system to periodically check the status of workflows.
2. Control log file sizes and archive older logs to prevent the AI Connector from breaking due to large log files.
3. Fix the issue where the connector aggressively logs errors in an infinite loop when the broker stops. This happens because the error log is written continuously without a delay until the broker reconnects.
4. Ensure the connector shuts down gracefully by addressing two main issues:
- The connector does not handle SIGINT and SIGTERM signals properly.
- Threads in sleep mode cannot be terminated by signals.
5. Remove dependencies on Langchain and Litellm from the connector’s core, as these libraries are not used directly by the connector’s core functionality.
6. Prevent logs from recording confidential information, such as keys and user payloads, to address security concerns.
7. Logging the connection/reconnection attempts. The PubSub API does not log attempts.
8. Getting the connection status using an API

How is this accomplished?

1. Monitoring:
1.1. Added a monitoring singleton class that is instantiated once the connector is created.
1.2. Updated the component base to run the monitoring per component and periodically get metrics of the component. Each metric is a dictionary that contains [flow name, flow index, component name, component index, time stamp, metric].
1.3. Each component defines the actual metrics. Herein, updated the broker base component to get the ACK metric. We do not need other component's metrics at this moment.
1.4. Updated the SolaceMessage and ComponentBase classes to manage the state of connection. Then, the componentBase pulls the connection status of each broker component. Finally, the monitoring module aggregates all brokers' status and result the final connection status of the AI connector. The AI Connector is connected when all broker modules are connected.
1.5 For logging connection/reconnection attempt, we added some threads to the connection/reconnection handler to log attempts.

2. Log files: Updated the logger in the log.py to support the below features:
2.1. Enable the log rotating. It gets the maximum size of logs and the number of archived files. It configures the logger to consider these parameters.
2.1. Set a pattern for the archived files (e.g., ..). Note that the main log should be the log file name rather than the pattern.

3. Aggressive log: Added the grow and reset sleep time methods. These methods exponentially increase the sleep time when they fail to connect to the broker and then reset the sleep time when it can connect.

4. Graceful shutdown: Handled SIGINT and SIGTERM in the main.py and replaced sleep.time methods with the stop_signal.wait, which breaks threads while they are in the sleep mode.

5. Langchain and Litellm dependencies: Removed these packages from the component-> init.py, requirements.txt and pyproject.toml files.

6. Log security: Removed payloads from logs.

Anything reviews should focus on/be aware of?

Langchain and Litellm dependencies: Do we use these libraries somewhere in the core Ai Connector?
Graceful shutdown: Is there any dangling thread or variable that needs to be closes/cleaned up?
Monitoring: Is there any edge case that needs to be covered?

gitstream-cm · 2024-11-29T04:15:45Z

Please mark whether you used Copilot to assist coding in this PR

Copilot Assisted

mgevantmakher · 2024-11-29T19:49:23Z

src/solace_ai_connector/components/component_base.py

@@ -65,18 +69,25 @@ def create_thread_and_run(self):
        return self.thread

    def run(self):
+        # Start the micro monitoring thread
+        monitoring_thread = threading.Thread(target=self.run_micro_monitoring)
+        monitoring_thread.start()


pls set all thread to daemon

enabled daemon for threads.

…78/add_monitoring

sonarqube-solacecloud · 2024-12-11T13:34:42Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
7 Code Smells

0.0% Coverage
0.0% Duplication

cyrus2281

Has the new changes been tested with cognitive-mesh?

src/solace_ai_connector/common/messaging/solace_messaging.py

alimosaed · 2025-01-17T14:20:45Z

Has the new changes been tested with cognitive-mesh?

Tested by a few agents. A comprehensive test may be required. Also, the CM configuration can be updated to take advantages of the new changes such as log rotation.

cyrus2281

Can you please ensure you can run cognitive-mesh locally smoothly with all your changes and test all the functionalities before merging the PR.

src/solace_ai_connector/common/messaging/solace_messaging.py

sonarqube-solacecloud · 2025-01-21T19:50:30Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

0.0% Coverage
0.0% Duplication

alimosaed · 2025-01-21T20:17:46Z

examples/llm/langchain_openai_with_history_chat.yaml

Goal: Remove dependency to Langchain.

alimosaed · 2025-01-21T20:18:35Z

examples/llm/litellm_chat.yaml

Goal: Add log rotation

alimosaed · 2025-01-21T20:19:25Z

examples/llm/openai_chroma_rag.yaml

Goal: Split a long payload to small chunks rather than getting an array of chunks.

alimosaed · 2025-01-21T20:20:20Z

pyproject.toml

Goal: Remove dependency to langchain and litellm. Upgrade the pubsub library.

alimosaed · 2025-01-21T20:21:07Z

src/solace_ai_connector/common/log.py

Goal: Support log rotation and backup

alimosaed · 2025-01-21T20:38:35Z

src/solace_ai_connector/components/inputs_outputs/broker_output.py

Goal: remove payload from logs due to security.

alimosaed · 2025-01-21T20:39:28Z

src/solace_ai_connector/components/inputs_outputs/broker_request_response.py

Goal: Add daemon to threads for terminating them gracefully.

alimosaed · 2025-01-21T20:40:11Z

src/solace_ai_connector/main.py

Goal: Gracefully terminate AI Connector.

alimosaed · 2025-01-21T20:40:56Z

src/solace_ai_connector/services/cache_service.py

Goal: Set daemon for threads to terminate them gracefully.

alimosaed · 2025-01-21T20:42:32Z

src/solace_ai_connector/solace_ai_connector.py

Goal:

Gracefully terminate the connector.

Configure log rotation.

cyrus2281

Tested! ✅

LGTM

feat: add monitring component

eddc391

alimosaed added 3 commits November 28, 2024 23:49

fix: resolve a bug

a955cfb

fix: add sleep time

2b271e3

fix: add sleep time

5075043

mgevantmakher reviewed Nov 29, 2024

View reviewed changes

alimosaed added 3 commits November 29, 2024 16:41

feat: add readiness and handle excessive logs

03d8b75

fix: handle sleep error

9965033

fix: handle sleep error

6c5362b

johnvincentcorpuz force-pushed the main branch from a2befd4 to 264fce2 Compare December 2, 2024 22:14

alimosaed added 18 commits December 2, 2024 23:05

feat: gracefully exit

3dbba60

feat: set the log back

a62d4cf

fix: rename log fields

116068a

fix: disabled monitoring

4c80e41

Merge branch 'main' of github.com:SolaceDev/solace-ai-connector

74f3d34

fix: resolve log naming

98b8699

fix: resolved logging issues

796fd67

fix: resolve log

cbbbd7c

fix: resolve log

37e3144

feat: remove dependency to Langchain

52b5233

feat: update monitoring

ad742f7

Merge branch 'main' of github.com:SolaceDev/solace-ai-connector

7d186bd

feat: drop error messages when the queue is full

d93e5de

feat: add a text splitter component

dfe4219

feat: updated docs

9648a94

Merge branch 'alireza/AI-278/add_vdb_text_splitter' into alireza/AI-2…

f3afadf

…78/add_monitoring

fix: resolve graceful termination issues

4eb4bef

fix: remove payloads from logs

488de44

Merge branch 'main' of github.com:SolaceDev/solace-ai-connector

a5de339

alimosaed added 10 commits January 9, 2025 17:52

feat: add reconnecting status

5f56c79

feat: add reconnecting log and handled signals

b30c99b

fix: update status

a64a48b

fix: update log

7467b8f

fix: fix bug

d49b4ef

fix: fix bug

ae69248

fix: resolve connection logs

8a9ba5c

fix: handle threads

1bff59e

fix: update connection state machine

89ecf1d

feat: add prefix to the broker logs

2e0cb5f

alimosaed requested a review from gregmeldrum January 14, 2025 20:35

cyrus2281 reviewed Jan 15, 2025

View reviewed changes

src/solace_ai_connector/common/messaging/solace_messaging.py Outdated Show resolved Hide resolved

alimosaed added 4 commits January 17, 2025 08:44

fix: synchronize logs with connection attempts

33e042f

fix: remove datadog dependency

d8cfd16

Merge branch 'main' of github.com:SolaceDev/solace-ai-connector

caab235

Merge branch 'main' into alireza/AI-278/add_monitoring

e4ad224

cyrus2281 reviewed Jan 17, 2025

View reviewed changes

src/solace_ai_connector/common/messaging/solace_messaging.py Outdated Show resolved Hide resolved

alimosaed added 7 commits January 17, 2025 15:44

fix: cover an exception

8d664d0

ref: upgrade to latest pubsub and replace a metric

7cac805

ref: capsulate some variables

7dfd113

ref: enable daemon for threads to close them safely

b982189

Merge branch 'main' of github.com:SolaceDev/solace-ai-connector

93ebd70

Merge branch 'main' into alireza/AI-278/add_monitoring

be7df51

ref: remove useless variable

9bfc6e6

alimosaed commented Jan 21, 2025

View reviewed changes

cyrus2281 approved these changes Jan 21, 2025

View reviewed changes

alimosaed merged commit 5fb33eb into main Jan 22, 2025
6 of 9 checks passed

alimosaed deleted the alireza/AI-278/add_monitoring branch January 22, 2025 13:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stabilize the connector and add monitoring #70

Stabilize the connector and add monitoring #70

alimosaed commented Nov 29, 2024 •

edited

Loading

gitstream-cm bot commented Nov 29, 2024

mgevantmakher Nov 29, 2024

alimosaed Jan 21, 2025

sonarqube-solacecloud bot commented Dec 11, 2024

cyrus2281 left a comment

alimosaed commented Jan 17, 2025

cyrus2281 left a comment

sonarqube-solacecloud bot commented Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

alimosaed Jan 21, 2025

cyrus2281 left a comment

Stabilize the connector and add monitoring #70

Stabilize the connector and add monitoring #70

Conversation

alimosaed commented Nov 29, 2024 • edited Loading

gitstream-cm bot commented Nov 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqube-solacecloud bot commented Dec 11, 2024

cyrus2281 left a comment

Choose a reason for hiding this comment

alimosaed commented Jan 17, 2025

cyrus2281 left a comment

Choose a reason for hiding this comment

sonarqube-solacecloud bot commented Jan 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyrus2281 left a comment

Choose a reason for hiding this comment

alimosaed commented Nov 29, 2024 •

edited

Loading