Feature n/lancedb #1122

akashAD98 · 2024-09-12T13:25:43Z

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Added support for lancedb as vectordb
Other information:
I have already created pr but there are a few things missing so I created a new pr

also, I tested it & it's working now with Lancedb

vercel · 2024-09-12T13:25:57Z

@akashAD98 is attempting to deploy a commit to the Arc53 Team on Vercel.

A member of the Team first needs to authorize it.

vercel · 2024-09-12T13:26:03Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-gpt	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 12, 2024 7:20pm
nextra-docsgpt	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Sep 12, 2024 7:20pm

akashAD98 · 2024-09-12T13:27:57Z

@dartpain i added this pr & checked all implementation in local its working fine

would be great if you can merge it . thank you

akashAD98 · 2024-09-16T09:46:15Z

@dartpain would be great if you could check & merge the pr .thank you

akashAD98 · 2024-09-18T13:19:18Z

@ManishMadan2882 can you please check this pr?

dartpain · 2024-09-23T22:18:35Z

Hi, I seem to have an issue running it.

I added "lancedb": LanceDBVectorStore, in application/vectorstore/vector_creator.py
also set VECTOR_STORE: str = "lancedb"
Maybe there is something wrong with my setup?
It does seem to be a good start for the solution imo

akashAD98 · 2024-09-25T06:41:24Z

same changes i .did & i runned & its giving me output

/DocsGPT$ ./run-with-docker-compose.sh

is their any other way to do it test it? what error your getting
logs:

frontend_1  |   VITE v5.3.5  ready in 191 ms
frontend_1  | 
frontend_1  |   ➜  Local:   http://localhost:5173/
frontend_1  |   ➜  Network: http://172.19.0.6:5173/
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.058+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53436","uuid":"472f9b51-b4e1-4281-acf2-4880015a8f53","connectionId":1,"connectionCount":1}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.059+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.19.0.4:53436","client":"conn1","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.061+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53442","uuid":"318b2a7b-a483-4f09-ad9d-91f76dd71274","connectionId":2,"connectionCount":2}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.061+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.19.0.4:53442","client":"conn2","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.062+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53448","uuid":"61a67a6c-e1c6-4148-8d46-65462455a803","connectionId":3,"connectionCount":3}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.062+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.19.0.4:53448","client":"conn3","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.064+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53456","uuid":"a753d070-5beb-4eb1-afbb-a3d107e3ae3d","connectionId":4,"connectionCount":4}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.064+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn4","msg":"client metadata","attr":{"remote":"172.19.0.4:53456","client":"conn4","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.078+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53472","uuid":"c649d152-ad89-43a0-8ab1-93bb3fee774b","connectionId":5,"connectionCount":5}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.078+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn5","msg":"client metadata","attr":{"remote":"172.19.0.4:53472","client":"conn5","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.080+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53478","uuid":"5d210022-e880-4790-8f86-cfe87caab474","connectionId":6,"connectionCount":6}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.080+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53488","uuid":"bd333a1f-e88e-4650-8903-4753f46f1f0f","connectionId":7,"connectionCount":7}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.080+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn6","msg":"client metadata","attr":{"remote":"172.19.0.4:53478","client":"conn6","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.081+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn7","msg":"client metadata","attr":{"remote":"172.19.0.4:53488","client":"conn7","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.088+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.4:53492","uuid":"530ac21b-f2c4-4578-936c-f563e225a851","connectionId":8,"connectionCount":8}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.088+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn8","msg":"client metadata","attr":{"remote":"172.19.0.4:53492","client":"conn8","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
worker_1    |  
worker_1    |  -------------- celery@dc6299e478d8 v5.3.6 (emerald-rush)
worker_1    | --- ***** ----- 
worker_1    | -- ******* ---- Linux-6.8.0-45-generic-x86_64-with-glibc2.39 2024-09-25 06:40:24
worker_1    | - *** --- * --- 
worker_1    | - ** ---------- [config]
worker_1    | - ** ---------- .> app:         application.celery_init:0x7fdbb3ae4b90
worker_1    | - ** ---------- .> transport:   redis://redis:6379/0
worker_1    | - ** ---------- .> results:     redis://redis:6379/1
worker_1    | - *** --- * --- .> concurrency: 16 (prefork)
worker_1    | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
worker_1    | --- ***** ----- 
worker_1    |  -------------- [queues]
worker_1    |                 .> celery           exchange=celery(direct) key=celery
worker_1    |                 
worker_1    | 
worker_1    | [tasks]
worker_1    |   . application.api.user.tasks.ingest
worker_1    |   . application.api.user.tasks.ingest_remote
worker_1    | 
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.157+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46672","uuid":"9473e058-63d7-4967-9430-8cc8147a1383","connectionId":9,"connectionCount":9}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.158+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn9","msg":"client metadata","attr":{"remote":"172.19.0.5:46672","client":"conn9","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.160+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46688","uuid":"446626f2-a5fb-4cdf-867a-f9987040cfdd","connectionId":10,"connectionCount":10}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.161+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn10","msg":"client metadata","attr":{"remote":"172.19.0.5:46688","client":"conn10","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.163+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46704","uuid":"2326ce0f-df9e-435b-999e-f7f6c8fca2fe","connectionId":11,"connectionCount":11}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.165+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn11","msg":"client metadata","attr":{"remote":"172.19.0.5:46704","client":"conn11","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.166+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46718","uuid":"8efdd73e-a2b6-4f81-8985-f08b83eb0345","connectionId":12,"connectionCount":12}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.166+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn12","msg":"client metadata","attr":{"remote":"172.19.0.5:46718","client":"conn12","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.177+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46734","uuid":"aef9ab2f-89a5-4dc1-bbb1-a3e8c2192b78","connectionId":13,"connectionCount":13}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.178+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn13","msg":"client metadata","attr":{"remote":"172.19.0.5:46734","client":"conn13","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.180+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46762","uuid":"2b9c3f69-e352-4464-a76f-b5b4538fef74","connectionId":14,"connectionCount":14}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.180+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46748","uuid":"cb4a4fb8-126d-4cd0-b9db-55eb6bee48d3","connectionId":15,"connectionCount":15}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.180+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn14","msg":"client metadata","attr":{"remote":"172.19.0.5:46762","client":"conn14","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.180+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn15","msg":"client metadata","attr":{"remote":"172.19.0.5:46748","client":"conn15","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.186+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46776","uuid":"ffa2ed9b-02c3-499b-8bcb-e332a459e49b","connectionId":16,"connectionCount":16}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.186+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn16","msg":"client metadata","attr":{"remote":"172.19.0.5:46776","client":"conn16","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.213+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46792","uuid":"31383af8-1d59-417a-84c8-ecac3cfd9cd2","connectionId":17,"connectionCount":17}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.213+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn17","msg":"client metadata","attr":{"remote":"172.19.0.5:46792","client":"conn17","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.214+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46800","uuid":"f5dbb87d-477e-46fe-899d-91729c54b861","connectionId":18,"connectionCount":18}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.215+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46812","uuid":"ae7cca76-5ce5-4085-afe7-afa0cd20cc6f","connectionId":19,"connectionCount":19}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.215+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn19","msg":"client metadata","attr":{"remote":"172.19.0.5:46812","client":"conn19","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.215+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn18","msg":"client metadata","attr":{"remote":"172.19.0.5:46800","client":"conn18","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.216+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46814","uuid":"4253face-8d0a-4438-ad0c-0936e9303542","connectionId":20,"connectionCount":20}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.217+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn20","msg":"client metadata","attr":{"remote":"172.19.0.5:46814","client":"conn20","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.226+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46828","uuid":"74e54ccc-3aef-4634-8782-cfcfb178ee45","connectionId":21,"connectionCount":21}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.226+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn21","msg":"client metadata","attr":{"remote":"172.19.0.5:46828","client":"conn21","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.227+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46834","uuid":"74742d1f-0ab3-4c7e-bbe6-d3d1cf75945e","connectionId":22,"connectionCount":22}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.227+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn22","msg":"client metadata","attr":{"remote":"172.19.0.5:46834","client":"conn22","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.227+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46836","uuid":"960b28a1-ae63-4a02-92fc-6d4491cf1d84","connectionId":23,"connectionCount":23}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.232+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn23","msg":"client metadata","attr":{"remote":"172.19.0.5:46836","client":"conn23","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.233+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:46844","uuid":"8b9a8427-5850-455d-8c27-e8b2d8805263","connectionId":24,"connectionCount":24}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:24.234+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn24","msg":"client metadata","attr":{"remote":"172.19.0.5:46844","client":"conn24","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
worker_1    | [2024-09-25 06:40:26,278] WARNING in warnings: /venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker_1    | whether broker connection retries are made during startup in Celery 6.0 and above.
worker_1    | If you wish to retain the existing behavior for retrying connections on startup,
worker_1    | you should set broker_connection_retry_on_startup to True.
worker_1    |   warnings.warn(
worker_1    | 
worker_1    | [2024-09-25 06:40:26,284] INFO in connection: Connected to redis://redis:6379/0
worker_1    | [2024-09-25 06:40:26,285] WARNING in warnings: /venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker_1    | whether broker connection retries are made during startup in Celery 6.0 and above.
worker_1    | If you wish to retain the existing behavior for retrying connections on startup,
worker_1    | you should set broker_connection_retry_on_startup to True.
worker_1    |   warnings.warn(
worker_1    | 
worker_1    | [2024-09-25 06:40:26,287] INFO in mingle: mingle: searching for neighbors
worker_1    | [2024-09-25 06:40:27,297] INFO in mingle: mingle: all alone
mongo_1     | {"t":{"$date":"2024-09-25T06:40:31.186+00:00"},"s":"I",  "c":"WTCHKPT",  "id":22430,   "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1727246431,"ts_usec":186058,"thread":"1:0x70c98b800640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 42, snapshot max: 42 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 1"}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:38.460+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:39410","uuid":"88060913-146c-4d82-87af-e045a72e9710","connectionId":25,"connectionCount":25}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:38.460+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:39408","uuid":"23317271-1fbf-48da-921c-f172111890c2","connectionId":26,"connectionCount":26}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:38.460+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn25","msg":"client metadata","attr":{"remote":"172.19.0.5:39410","client":"conn25","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:38.460+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn26","msg":"client metadata","attr":{"remote":"172.19.0.5:39408","client":"conn26","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
backend_1   | [2024-09-25 06:40:43,369] INFO in routes: /stream - request_data: {'question': 'What is DocsGPT?', 'active_docs': 'default/.project//huggingface_sentence-transformers/all-mpnet-base-v2/', 'history': '[{"prompt":"What is DocsGPT?"}]', 'conversation_id': None, 'prompt_id': 'default', 'chunks': '2', 'token_limit': 2000, 'isNoneDoc': False}, source: {'active_docs': 'default/.project//huggingface_sentence-transformers/all-mpnet-base-v2/'}
backend_1   | /venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
backend_1   |   warnings.warn(
Batches: 100%|██████████| 1/1 [00:00<00:00,  8.39it/s]
mongo_1     | {"t":{"$date":"2024-09-25T06:40:49.754+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:52592","uuid":"ad35a5b1-9d3c-4ec5-9d6a-93303bc62229","connectionId":27,"connectionCount":27}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:49.755+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn27","msg":"client metadata","attr":{"remote":"172.19.0.5:52592","client":"conn27","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:49.756+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"conn27","msg":"createCollection","attr":{"namespace":"docsgpt.token_usage","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"00cceaab-e9dd-4d18-8282-850558b2ebe3"}},"options":{}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:49.764+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"conn27","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"00cceaab-e9dd-4d18-8282-850558b2ebe3"}},"namespace":"docsgpt.token_usage","index":"_id_","ident":"index-8-6698800254010345513","collectionIdent":"collection-7-6698800254010345513","commitTimestamp":null}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:51.869+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.5:52594","uuid":"42529282-c5ff-4fc1-b1d0-b2508d11341d","connectionId":28,"connectionCount":28}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:51.870+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn28","msg":"client metadata","attr":{"remote":"172.19.0.5:52594","client":"conn28","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo","version":"4.8.0"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"6.8.0-45-generic"},"platform":"CPython 3.11.10.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:51.871+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"conn28","msg":"createCollection","attr":{"namespace":"docsgpt.conversations","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"b2bf513e-dcbc-467a-abea-e0557f71ef44"}},"options":{}}}
mongo_1     | {"t":{"$date":"2024-09-25T06:40:51.882+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"conn28","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"b2bf513e-dcbc-467a-abea-e0557f71ef44"}},"namespace":"docsgpt.conversations","index":"_id_","ident":"index-10-6698800254010345513","collectionIdent":"collection-9-6698800254010345513","commitTimestamp":null}}
backend_1   | [2024-09-25 06:40:51,896] INFO in routes: /api/answer - request_data: {'question': 'What is DocsGPT?', 'active_docs': 'default/.project//huggingface_sentence-transformers/all-mpnet-base-v2/', 'conversation_id': None, 'history': [{'prompt': 'What is DocsGPT?'}], 'chunks': '2', 'token_limit': 2000, 'isNoneDoc': False}, source: {'active_docs': 'default/.project//huggingface_sentence-transformers/all-mpnet-base-v2/'}
Batches: 100%|██████████| 1/1 [00:00<00:00, 10.15it/s]
mongo_1     | {"t":{"$date":"2024-09-25T06:41:31.190+00:00"},"s":"I",  "c":"WTCHKPT",  "id":22430,   "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1727246491,"ts_usec":189961,"thread":"1:0x70c98b800640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 61, snapshot max: 61 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 1"}}}

akashAD98 · 2024-09-25T06:46:30Z

also when i try to train custom pdf/docs it's stuck every time. I tried with other vectord db also. same behavior across all. (maybe im doing wrong something )

akashAD98 · 2024-10-05T03:40:56Z

@dartpain hi sir this pr is pending for 3 weeks.can you check it please?

dartpain · 2024-10-05T22:45:13Z

@akashAD98
I really appreciate you PR, honestly, dont let my lack of response discourage you.
I think there is are some issues with it. Sometimes I try to fix them all myself, but only if I have time...

The way you init it, in some other uses it might overwrite the table name open

    def __init__(self, path: str = settings.LANCEDB_PATH,
                 table_name: str = settings.LANCEDB_TABLE_NAME,
                 embeddings_key: str = "embeddings"):

check out this file https://github.com/arc53/DocsGPT/blob/main/application/parser/open_ai_func.py

we dont filter by source_id as we should
This db is launched in an in memory mode, it will not be compatible with docker, since they need to share the files (worker and api)
Lets lazy import the dependency please

Most issues should be easy to fix.
Hardest one is the 3rd one, here you may need to re-think connection to db and init.
If im misunderstanding something about it, please correct me.

AyushExel · 2024-10-08T15:12:11Z

Hi @dartpain , I'm from LanceDB team. Just came across this PR. Could you please elaborate on point 3, as to why Lance wont work with docker? As for in-memory mode, LanceDB is persisted in storage rather than memory, so it doesn't need to load/create vector indices in memory. Let me know if you're seeing any weird behaviours

akashAD98 · 2024-10-19T04:47:43Z

@dartpain any update on this

dartpain

LGTM, thank you so much, sorry for delays, had to test things

dartpain · 2024-10-21T19:25:40Z

@holopin-bot @akashAD98 Thank you!

dartpain · 2024-11-08T12:46:10Z

@holopin-bot @akashAD98 Thank you!

holopin-bot · 2024-11-08T12:46:13Z

Congratulations @akashAD98, the maintainer of this repository has issued you a badge! Here it is: https://holopin.io/claim/cm38qd81x73040clb2f38ca7t

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!

dartpain · 2024-11-11T22:25:58Z

Hey @akashAD98 please fill in this form if you are interested in a t-shirt
https://airtable.com/appikMaJwdHhC1SDP/pagoblCJ9W29wf6Hf/form

akashAD98 added 2 commits September 12, 2024 18:51

added support for lacedb as vectordb

7e75513

added lancedb in pacage

66ebfef

github-actions bot added the application Application label Sep 12, 2024

vercel bot deployed to Preview – nextra-docsgpt September 12, 2024 13:26 View deployment

vercel bot deployed to Preview – docs-gpt September 12, 2024 19:20 View deployment

akashmangoai added 2 commits October 11, 2024 21:53

removed cloud based parameter which are not needed

2438399

lazy import & fixed other issue

a9f6a06

vercel bot deployed to Preview – nextra-docsgpt October 11, 2024 17:29 View deployment

dartpain approved these changes Oct 19, 2024

View reviewed changes

Merge branch 'main' into feature_n/lancedb

fd8e277

dartpain merged commit 3f7c8bd into arc53:main Oct 19, 2024
3 of 5 checks passed

vercel bot deployed to Preview – nextra-docsgpt October 19, 2024 15:23 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature n/lancedb #1122

Feature n/lancedb #1122

akashAD98 commented Sep 12, 2024

vercel bot commented Sep 12, 2024

vercel bot commented Sep 12, 2024 •

edited

Loading

akashAD98 commented Sep 12, 2024

akashAD98 commented Sep 16, 2024

akashAD98 commented Sep 18, 2024

dartpain commented Sep 23, 2024

akashAD98 commented Sep 25, 2024 •

edited

Loading

akashAD98 commented Sep 25, 2024

akashAD98 commented Oct 5, 2024

dartpain commented Oct 5, 2024

AyushExel commented Oct 8, 2024

akashAD98 commented Oct 19, 2024

dartpain left a comment

dartpain commented Oct 21, 2024

dartpain commented Nov 8, 2024

holopin-bot bot commented Nov 8, 2024

dartpain commented Nov 11, 2024

Feature n/lancedb #1122

Feature n/lancedb #1122

Conversation

akashAD98 commented Sep 12, 2024

vercel bot commented Sep 12, 2024

vercel bot commented Sep 12, 2024 • edited Loading

akashAD98 commented Sep 12, 2024

akashAD98 commented Sep 16, 2024

akashAD98 commented Sep 18, 2024

dartpain commented Sep 23, 2024

akashAD98 commented Sep 25, 2024 • edited Loading

akashAD98 commented Sep 25, 2024

akashAD98 commented Oct 5, 2024

dartpain commented Oct 5, 2024

AyushExel commented Oct 8, 2024

akashAD98 commented Oct 19, 2024

dartpain left a comment

Choose a reason for hiding this comment

dartpain commented Oct 21, 2024

dartpain commented Nov 8, 2024

holopin-bot bot commented Nov 8, 2024

dartpain commented Nov 11, 2024

vercel bot commented Sep 12, 2024 •

edited

Loading

akashAD98 commented Sep 25, 2024 •

edited

Loading