diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 484ebd1f38313..d90322e897799 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -3,43 +3,4 @@ Hi there! Thank you for even being interested in contributing to LangChain. As an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes. -To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/) - -## 🗺️ Guidelines - -### 👩‍💻 Ways to contribute - -There are many ways to contribute to LangChain. Here are some common ways people contribute: - -- [**Documentation**](https://python.langchain.com/docs/contributing/documentation): Help improve our docs, including this one! -- [**Code**](https://python.langchain.com/docs/contributing/code): Help us write code, fix bugs, or improve our infrastructure. -- [**Integrations**](https://python.langchain.com/docs/contributing/integrations): Help us integrate with your favorite vendors and tools. - -### 🚩GitHub Issues - -Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests. - -There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help organize issues. - -If you start working on an issue, please assign it to yourself. - -If you are adding an issue, please try to keep it focused on a single, modular bug/improvement/feature. -If two issues are related, or blocking, please link them rather than combining them. - -We will try to keep these issues as up-to-date as possible, though -with the rapid rate of development in this field some may get out of date. -If you notice this happening, please let us know. - -### 🙋Getting Help - -Our goal is to have the simplest developer setup possible. Should you experience any difficulty getting setup, please -contact a maintainer! Not only do we want to help get you unblocked, but we also want to make sure that the process is -smooth for future contributors. - -In a similar vein, we do enforce certain linting, formatting, and documentation standards in the codebase. -If you are finding these difficult (or even just annoying) to work with, feel free to contact a maintainer for help - -we do not want these to get in the way of getting good code into the codebase. - -### Contributor Documentation - -To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/) +To learn how to contribute to LangChain, please follow the [contribution guide here](https://python.langchain.com/docs/contributing/). \ No newline at end of file diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index aee9d9f05b6fd..8d776064019a7 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,19 +1,24 @@ Thank you for contributing to LangChain! -Checklist: - -- [ ] PR title: Please title your PR "package: description", where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..." for purely docs changes, "templates: ..." for template changes, "infra: ..." for CI changes. +- [ ] **PR title**: "package: description" + - Where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..." for purely docs changes, "templates: ..." for template changes, "infra: ..." for CI changes. - Example: "community: add foobar LLM" -- [ ] PR message: **Delete this entire template message** and replace it with the following bulleted list + + +- [ ] **PR message**: ***Delete this entire checklist*** and replace with - **Description:** a description of the change - **Issue:** the issue # it fixes, if applicable - **Dependencies:** any dependencies required for this change - **Twitter handle:** if your PR gets announced, and you'd like a mention, we'll gladly shout you out! -- [ ] Pass lint and test: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified to check that you're passing lint and testing. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ -- [ ] Add tests and docs: If you're adding a new integration, please include + + +- [ ] **Add tests and docs**: If you're adding a new integration, please include 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. + +- [ ] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ + Additional guidelines: - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. @@ -21,4 +26,4 @@ Additional guidelines: - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. -If no one reviews your PR within a few days, please @-mention one of @baskaryan, @efriis, @eyurtsev, @hwchase17. \ No newline at end of file +If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17. diff --git a/.github/actions/people/Dockerfile b/.github/actions/people/Dockerfile new file mode 100644 index 0000000000000..bf214eea7dcc3 --- /dev/null +++ b/.github/actions/people/Dockerfile @@ -0,0 +1,7 @@ +FROM python:3.9 + +RUN pip install httpx PyGithub "pydantic==2.0.2" pydantic-settings "pyyaml>=5.3.1,<6.0.0" + +COPY ./app /app + +CMD ["python", "/app/main.py"] \ No newline at end of file diff --git a/.github/actions/people/action.yml b/.github/actions/people/action.yml new file mode 100644 index 0000000000000..15a00c108b472 --- /dev/null +++ b/.github/actions/people/action.yml @@ -0,0 +1,11 @@ +# Adapted from https://github.com/tiangolo/fastapi/blob/master/.github/actions/people/action.yml +name: "Generate LangChain People" +description: "Generate the data for the LangChain People page" +author: "Jacob Lee " +inputs: + token: + description: 'User token, to read the GitHub API. Can be passed in using {{ secrets.LANGCHAIN_PEOPLE_GITHUB_TOKEN }}' + required: true +runs: + using: 'docker' + image: 'Dockerfile' \ No newline at end of file diff --git a/.github/actions/people/app/main.py b/.github/actions/people/app/main.py new file mode 100644 index 0000000000000..ef714450aed80 --- /dev/null +++ b/.github/actions/people/app/main.py @@ -0,0 +1,641 @@ +# Adapted from https://github.com/tiangolo/fastapi/blob/master/.github/actions/people/app/main.py + +import logging +import subprocess +import sys +from collections import Counter +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any, Container, Dict, List, Set, Union + +import httpx +import yaml +from github import Github +from pydantic import BaseModel, SecretStr +from pydantic_settings import BaseSettings + +github_graphql_url = "https://api.github.com/graphql" +questions_category_id = "DIC_kwDOIPDwls4CS6Ve" + +# discussions_query = """ +# query Q($after: String, $category_id: ID) { +# repository(name: "langchain", owner: "langchain-ai") { +# discussions(first: 100, after: $after, categoryId: $category_id) { +# edges { +# cursor +# node { +# number +# author { +# login +# avatarUrl +# url +# } +# title +# createdAt +# comments(first: 100) { +# nodes { +# createdAt +# author { +# login +# avatarUrl +# url +# } +# isAnswer +# replies(first: 10) { +# nodes { +# createdAt +# author { +# login +# avatarUrl +# url +# } +# } +# } +# } +# } +# } +# } +# } +# } +# } +# """ + +# issues_query = """ +# query Q($after: String) { +# repository(name: "langchain", owner: "langchain-ai") { +# issues(first: 100, after: $after) { +# edges { +# cursor +# node { +# number +# author { +# login +# avatarUrl +# url +# } +# title +# createdAt +# state +# comments(first: 100) { +# nodes { +# createdAt +# author { +# login +# avatarUrl +# url +# } +# } +# } +# } +# } +# } +# } +# } +# """ + +prs_query = """ +query Q($after: String) { + repository(name: "langchain", owner: "langchain-ai") { + pullRequests(first: 100, after: $after, states: MERGED) { + edges { + cursor + node { + changedFiles + additions + deletions + number + labels(first: 100) { + nodes { + name + } + } + author { + login + avatarUrl + url + ... on User { + twitterUsername + } + } + title + createdAt + state + reviews(first:100) { + nodes { + author { + login + avatarUrl + url + ... on User { + twitterUsername + } + } + state + } + } + } + } + } + } +} +""" + + +class Author(BaseModel): + login: str + avatarUrl: str + url: str + twitterUsername: Union[str, None] = None + + +# Issues and Discussions + + +class CommentsNode(BaseModel): + createdAt: datetime + author: Union[Author, None] = None + + +class Replies(BaseModel): + nodes: List[CommentsNode] + + +class DiscussionsCommentsNode(CommentsNode): + replies: Replies + + +class Comments(BaseModel): + nodes: List[CommentsNode] + + +class DiscussionsComments(BaseModel): + nodes: List[DiscussionsCommentsNode] + + +class IssuesNode(BaseModel): + number: int + author: Union[Author, None] = None + title: str + createdAt: datetime + state: str + comments: Comments + + +class DiscussionsNode(BaseModel): + number: int + author: Union[Author, None] = None + title: str + createdAt: datetime + comments: DiscussionsComments + + +class IssuesEdge(BaseModel): + cursor: str + node: IssuesNode + + +class DiscussionsEdge(BaseModel): + cursor: str + node: DiscussionsNode + + +class Issues(BaseModel): + edges: List[IssuesEdge] + + +class Discussions(BaseModel): + edges: List[DiscussionsEdge] + + +class IssuesRepository(BaseModel): + issues: Issues + + +class DiscussionsRepository(BaseModel): + discussions: Discussions + + +class IssuesResponseData(BaseModel): + repository: IssuesRepository + + +class DiscussionsResponseData(BaseModel): + repository: DiscussionsRepository + + +class IssuesResponse(BaseModel): + data: IssuesResponseData + + +class DiscussionsResponse(BaseModel): + data: DiscussionsResponseData + + +# PRs + + +class LabelNode(BaseModel): + name: str + + +class Labels(BaseModel): + nodes: List[LabelNode] + + +class ReviewNode(BaseModel): + author: Union[Author, None] = None + state: str + + +class Reviews(BaseModel): + nodes: List[ReviewNode] + + +class PullRequestNode(BaseModel): + number: int + labels: Labels + author: Union[Author, None] = None + changedFiles: int + additions: int + deletions: int + title: str + createdAt: datetime + state: str + reviews: Reviews + # comments: Comments + + +class PullRequestEdge(BaseModel): + cursor: str + node: PullRequestNode + + +class PullRequests(BaseModel): + edges: List[PullRequestEdge] + + +class PRsRepository(BaseModel): + pullRequests: PullRequests + + +class PRsResponseData(BaseModel): + repository: PRsRepository + + +class PRsResponse(BaseModel): + data: PRsResponseData + + +class Settings(BaseSettings): + input_token: SecretStr + github_repository: str + httpx_timeout: int = 30 + + +def get_graphql_response( + *, + settings: Settings, + query: str, + after: Union[str, None] = None, + category_id: Union[str, None] = None, +) -> Dict[str, Any]: + headers = {"Authorization": f"token {settings.input_token.get_secret_value()}"} + # category_id is only used by one query, but GraphQL allows unused variables, so + # keep it here for simplicity + variables = {"after": after, "category_id": category_id} + response = httpx.post( + github_graphql_url, + headers=headers, + timeout=settings.httpx_timeout, + json={"query": query, "variables": variables, "operationName": "Q"}, + ) + if response.status_code != 200: + logging.error( + f"Response was not 200, after: {after}, category_id: {category_id}" + ) + logging.error(response.text) + raise RuntimeError(response.text) + data = response.json() + if "errors" in data: + logging.error(f"Errors in response, after: {after}, category_id: {category_id}") + logging.error(data["errors"]) + logging.error(response.text) + raise RuntimeError(response.text) + return data + + +# def get_graphql_issue_edges(*, settings: Settings, after: Union[str, None] = None): +# data = get_graphql_response(settings=settings, query=issues_query, after=after) +# graphql_response = IssuesResponse.model_validate(data) +# return graphql_response.data.repository.issues.edges + + +# def get_graphql_question_discussion_edges( +# *, +# settings: Settings, +# after: Union[str, None] = None, +# ): +# data = get_graphql_response( +# settings=settings, +# query=discussions_query, +# after=after, +# category_id=questions_category_id, +# ) +# graphql_response = DiscussionsResponse.model_validate(data) +# return graphql_response.data.repository.discussions.edges + + +def get_graphql_pr_edges(*, settings: Settings, after: Union[str, None] = None): + if after is None: + print("Querying PRs...") + else: + print(f"Querying PRs with cursor {after}...") + data = get_graphql_response( + settings=settings, + query=prs_query, + after=after + ) + graphql_response = PRsResponse.model_validate(data) + return graphql_response.data.repository.pullRequests.edges + + +# def get_issues_experts(settings: Settings): +# issue_nodes: List[IssuesNode] = [] +# issue_edges = get_graphql_issue_edges(settings=settings) + +# while issue_edges: +# for edge in issue_edges: +# issue_nodes.append(edge.node) +# last_edge = issue_edges[-1] +# issue_edges = get_graphql_issue_edges(settings=settings, after=last_edge.cursor) + +# commentors = Counter() +# last_month_commentors = Counter() +# authors: Dict[str, Author] = {} + +# now = datetime.now(tz=timezone.utc) +# one_month_ago = now - timedelta(days=30) + +# for issue in issue_nodes: +# issue_author_name = None +# if issue.author: +# authors[issue.author.login] = issue.author +# issue_author_name = issue.author.login +# issue_commentors = set() +# for comment in issue.comments.nodes: +# if comment.author: +# authors[comment.author.login] = comment.author +# if comment.author.login != issue_author_name: +# issue_commentors.add(comment.author.login) +# for author_name in issue_commentors: +# commentors[author_name] += 1 +# if issue.createdAt > one_month_ago: +# last_month_commentors[author_name] += 1 + +# return commentors, last_month_commentors, authors + + +# def get_discussions_experts(settings: Settings): +# discussion_nodes: List[DiscussionsNode] = [] +# discussion_edges = get_graphql_question_discussion_edges(settings=settings) + +# while discussion_edges: +# for discussion_edge in discussion_edges: +# discussion_nodes.append(discussion_edge.node) +# last_edge = discussion_edges[-1] +# discussion_edges = get_graphql_question_discussion_edges( +# settings=settings, after=last_edge.cursor +# ) + +# commentors = Counter() +# last_month_commentors = Counter() +# authors: Dict[str, Author] = {} + +# now = datetime.now(tz=timezone.utc) +# one_month_ago = now - timedelta(days=30) + +# for discussion in discussion_nodes: +# discussion_author_name = None +# if discussion.author: +# authors[discussion.author.login] = discussion.author +# discussion_author_name = discussion.author.login +# discussion_commentors = set() +# for comment in discussion.comments.nodes: +# if comment.author: +# authors[comment.author.login] = comment.author +# if comment.author.login != discussion_author_name: +# discussion_commentors.add(comment.author.login) +# for reply in comment.replies.nodes: +# if reply.author: +# authors[reply.author.login] = reply.author +# if reply.author.login != discussion_author_name: +# discussion_commentors.add(reply.author.login) +# for author_name in discussion_commentors: +# commentors[author_name] += 1 +# if discussion.createdAt > one_month_ago: +# last_month_commentors[author_name] += 1 +# return commentors, last_month_commentors, authors + + +# def get_experts(settings: Settings): +# ( +# discussions_commentors, +# discussions_last_month_commentors, +# discussions_authors, +# ) = get_discussions_experts(settings=settings) +# commentors = discussions_commentors +# last_month_commentors = discussions_last_month_commentors +# authors = {**discussions_authors} +# return commentors, last_month_commentors, authors + + +def _logistic(x, k): + return x / (x + k) + + +def get_contributors(settings: Settings): + pr_nodes: List[PullRequestNode] = [] + pr_edges = get_graphql_pr_edges(settings=settings) + + while pr_edges: + for edge in pr_edges: + pr_nodes.append(edge.node) + last_edge = pr_edges[-1] + pr_edges = get_graphql_pr_edges(settings=settings, after=last_edge.cursor) + + contributors = Counter() + contributor_scores = Counter() + recent_contributor_scores = Counter() + reviewers = Counter() + authors: Dict[str, Author] = {} + + for pr in pr_nodes: + pr_reviewers: Set[str] = set() + for review in pr.reviews.nodes: + if review.author: + authors[review.author.login] = review.author + pr_reviewers.add(review.author.login) + for reviewer in pr_reviewers: + reviewers[reviewer] += 1 + if pr.author: + authors[pr.author.login] = pr.author + contributors[pr.author.login] += 1 + files_changed = pr.changedFiles + lines_changed = pr.additions + pr.deletions + score = _logistic(files_changed, 20) + _logistic(lines_changed, 100) + contributor_scores[pr.author.login] += score + three_months_ago = (datetime.now(timezone.utc) - timedelta(days=3*30)) + if pr.createdAt > three_months_ago: + recent_contributor_scores[pr.author.login] += score + return contributors, contributor_scores, recent_contributor_scores, reviewers, authors + + +def get_top_users( + *, + counter: Counter, + min_count: int, + authors: Dict[str, Author], + skip_users: Container[str], +): + users = [] + for commentor, count in counter.most_common(): + if commentor in skip_users: + continue + if count >= min_count: + author = authors[commentor] + users.append( + { + "login": commentor, + "count": count, + "avatarUrl": author.avatarUrl, + "twitterUsername": author.twitterUsername, + "url": author.url, + } + ) + return users + + +if __name__ == "__main__": + logging.basicConfig(level=logging.INFO) + settings = Settings() + logging.info(f"Using config: {settings.model_dump_json()}") + g = Github(settings.input_token.get_secret_value()) + repo = g.get_repo(settings.github_repository) + # question_commentors, question_last_month_commentors, question_authors = get_experts( + # settings=settings + # ) + contributors, contributor_scores, recent_contributor_scores, reviewers, pr_authors = get_contributors( + settings=settings + ) + # authors = {**question_authors, **pr_authors} + authors = {**pr_authors} + maintainers_logins = { + "hwchase17", + "agola11", + "baskaryan", + "hinthornw", + "nfcampos", + "efriis", + "eyurtsev", + "rlancemartin" + } + hidden_logins = { + "dev2049", + "vowelparrot", + "obi1kenobi", + "langchain-infra", + "jacoblee93", + "dqbd", + "bracesproul", + "akira", + } + bot_names = {"dosubot", "github-actions", "CodiumAI-Agent"} + maintainers = [] + for login in maintainers_logins: + user = authors[login] + maintainers.append( + { + "login": login, + "count": contributors[login], #+ question_commentors[login], + "avatarUrl": user.avatarUrl, + "twitterUsername": user.twitterUsername, + "url": user.url, + } + ) + + # min_count_expert = 10 + # min_count_last_month = 3 + min_score_contributor = 1 + min_count_reviewer = 5 + skip_users = maintainers_logins | bot_names | hidden_logins + # experts = get_top_users( + # counter=question_commentors, + # min_count=min_count_expert, + # authors=authors, + # skip_users=skip_users, + # ) + # last_month_active = get_top_users( + # counter=question_last_month_commentors, + # min_count=min_count_last_month, + # authors=authors, + # skip_users=skip_users, + # ) + top_recent_contributors = get_top_users( + counter=recent_contributor_scores, + min_count=min_score_contributor, + authors=authors, + skip_users=skip_users, + ) + top_contributors = get_top_users( + counter=contributor_scores, + min_count=min_score_contributor, + authors=authors, + skip_users=skip_users, + ) + top_reviewers = get_top_users( + counter=reviewers, + min_count=min_count_reviewer, + authors=authors, + skip_users=skip_users, + ) + + people = { + "maintainers": maintainers, + # "experts": experts, + # "last_month_active": last_month_active, + "top_recent_contributors": top_recent_contributors, + "top_contributors": top_contributors, + "top_reviewers": top_reviewers, + } + people_path = Path("./docs/data/people.yml") + people_old_content = people_path.read_text(encoding="utf-8") + new_people_content = yaml.dump( + people, sort_keys=False, width=200, allow_unicode=True + ) + if ( + people_old_content == new_people_content + ): + logging.info("The LangChain People data hasn't changed, finishing.") + sys.exit(0) + people_path.write_text(new_people_content, encoding="utf-8") + logging.info("Setting up GitHub Actions git user") + subprocess.run(["git", "config", "user.name", "github-actions"], check=True) + subprocess.run( + ["git", "config", "user.email", "github-actions@github.com"], check=True + ) + branch_name = "langchain/langchain-people" + logging.info(f"Creating a new branch {branch_name}") + subprocess.run(["git", "checkout", "-B", branch_name], check=True) + logging.info("Adding updated file") + subprocess.run( + ["git", "add", str(people_path)], check=True + ) + logging.info("Committing updated file") + message = "👥 Update LangChain people data" + result = subprocess.run(["git", "commit", "-m", message], check=True) + logging.info("Pushing branch") + subprocess.run(["git", "push", "origin", branch_name, "-f"], check=True) + logging.info("Creating PR") + pr = repo.create_pull(title=message, body=message, base="master", head=branch_name) + logging.info(f"Created PR: {pr.number}") + logging.info("Finished") \ No newline at end of file diff --git a/.github/scripts/check_diff.py b/.github/scripts/check_diff.py index 4cbe1ec8b2f19..f02d59acad96d 100644 --- a/.github/scripts/check_diff.py +++ b/.github/scripts/check_diff.py @@ -1,17 +1,23 @@ import json import sys import os +from typing import Dict -LANGCHAIN_DIRS = { +LANGCHAIN_DIRS = [ "libs/core", "libs/langchain", "libs/experimental", "libs/community", -} +] if __name__ == "__main__": files = sys.argv[1:] - dirs_to_run = set() + + dirs_to_run: Dict[str, set] = { + "lint": set(), + "test": set(), + "extended-test": set(), + } if len(files) == 300: # max diff length is 300 files - there are likely files missing @@ -24,27 +30,42 @@ ".github/workflows", ".github/tools", ".github/actions", - "libs/core", ".github/scripts/check_diff.py", ) ): - dirs_to_run.update(LANGCHAIN_DIRS) - elif "libs/community" in file: - dirs_to_run.update( - ("libs/community", "libs/langchain", "libs/experimental") - ) - elif "libs/partners" in file: + # add all LANGCHAIN_DIRS for infra changes + dirs_to_run["extended-test"].update(LANGCHAIN_DIRS) + dirs_to_run["lint"].add(".") + + if any(file.startswith(dir_) for dir_ in LANGCHAIN_DIRS): + # add that dir and all dirs after in LANGCHAIN_DIRS + # for extended testing + found = False + for dir_ in LANGCHAIN_DIRS: + if file.startswith(dir_): + found = True + if found: + dirs_to_run["extended-test"].add(dir_) + elif file.startswith("libs/partners"): partner_dir = file.split("/")[2] if os.path.isdir(f"libs/partners/{partner_dir}"): - dirs_to_run.add(f"libs/partners/{partner_dir}") + dirs_to_run["test"].add(f"libs/partners/{partner_dir}") # Skip if the directory was deleted - elif "libs/langchain" in file: - dirs_to_run.update(("libs/langchain", "libs/experimental")) - elif "libs/experimental" in file: - dirs_to_run.add("libs/experimental") elif file.startswith("libs/"): - dirs_to_run.update(LANGCHAIN_DIRS) - else: - pass - json_output = json.dumps(list(dirs_to_run)) - print(f"dirs-to-run={json_output}") # noqa: T201 + raise ValueError( + f"Unknown lib: {file}. check_diff.py likely needs " + "an update for this new library!" + ) + elif any(file.startswith(p) for p in ["docs/", "templates/", "cookbook/"]): + dirs_to_run["lint"].add(".") + + outputs = { + "dirs-to-lint": list( + dirs_to_run["lint"] | dirs_to_run["test"] | dirs_to_run["extended-test"] + ), + "dirs-to-test": list(dirs_to_run["test"] | dirs_to_run["extended-test"]), + "dirs-to-extended-test": list(dirs_to_run["extended-test"]), + } + for key, value in outputs.items(): + json_output = json.dumps(value) + print(f"{key}={json_output}") # noqa: T201 diff --git a/.github/workflows/_all_ci.yml b/.github/workflows/_all_ci.yml deleted file mode 100644 index f6b27f029d1a1..0000000000000 --- a/.github/workflows/_all_ci.yml +++ /dev/null @@ -1,110 +0,0 @@ ---- -name: langchain CI - -on: - workflow_call: - inputs: - working-directory: - required: true - type: string - description: "From which folder this pipeline executes" - workflow_dispatch: - inputs: - working-directory: - required: true - type: choice - default: 'libs/langchain' - options: - - libs/langchain - - libs/core - - libs/experimental - - libs/community - - -# If another push to the same PR or branch happens while this workflow is still running, -# cancel the earlier run in favor of the next run. -# -# There's no point in testing an outdated version of the code. GitHub only allows -# a limited number of job runners to be active at the same time, so it's better to cancel -# pointless jobs early so that more useful jobs can run sooner. -concurrency: - group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.working-directory }} - cancel-in-progress: true - -env: - POETRY_VERSION: "1.7.1" - -jobs: - lint: - name: "-" - uses: ./.github/workflows/_lint.yml - with: - working-directory: ${{ inputs.working-directory }} - secrets: inherit - - test: - name: "-" - uses: ./.github/workflows/_test.yml - with: - working-directory: ${{ inputs.working-directory }} - secrets: inherit - - compile-integration-tests: - name: "-" - uses: ./.github/workflows/_compile_integration_test.yml - with: - working-directory: ${{ inputs.working-directory }} - secrets: inherit - - dependencies: - name: "-" - uses: ./.github/workflows/_dependencies.yml - with: - working-directory: ${{ inputs.working-directory }} - secrets: inherit - - extended-tests: - name: "make extended_tests #${{ matrix.python-version }}" - runs-on: ubuntu-latest - strategy: - matrix: - python-version: - - "3.8" - - "3.9" - - "3.10" - - "3.11" - defaults: - run: - working-directory: ${{ inputs.working-directory }} - if: ${{ ! startsWith(inputs.working-directory, 'libs/partners/') }} - steps: - - uses: actions/checkout@v4 - - - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }} - uses: "./.github/actions/poetry_setup" - with: - python-version: ${{ matrix.python-version }} - poetry-version: ${{ env.POETRY_VERSION }} - working-directory: ${{ inputs.working-directory }} - cache-key: extended - - - name: Install dependencies - shell: bash - run: | - echo "Running extended tests, installing dependencies with poetry..." - poetry install -E extended_testing --with test - - - name: Run extended tests - run: make extended_tests - - - name: Ensure the tests did not create any additional files - shell: bash - run: | - set -eu - - STATUS="$(git status)" - echo "$STATUS" - - # grep will exit non-zero if the target message isn't found, - # and `set -e` above will cause the step to fail. - echo "$STATUS" | grep 'nothing to commit, working tree clean' diff --git a/.github/workflows/_integration_test.yml b/.github/workflows/_integration_test.yml index 7e4afe70e2bfa..1189907e96695 100644 --- a/.github/workflows/_integration_test.yml +++ b/.github/workflows/_integration_test.yml @@ -52,6 +52,7 @@ jobs: - name: Run integration tests shell: bash env: + AI21_API_KEY: ${{ secrets.AI21_API_KEY }} GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }} ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} @@ -62,8 +63,13 @@ jobs: GOOGLE_CSE_ID: ${{ secrets.GOOGLE_CSE_ID }} EXA_API_KEY: ${{ secrets.EXA_API_KEY }} NOMIC_API_KEY: ${{ secrets.NOMIC_API_KEY }} + WATSONX_APIKEY: ${{ secrets.WATSONX_APIKEY }} + WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }} PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }} PINECONE_ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }} + ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }} + ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }} + ASTRA_DB_KEYSPACE: ${{ secrets.ASTRA_DB_KEYSPACE }} run: | make integration_tests diff --git a/.github/workflows/_release.yml b/.github/workflows/_release.yml index a0b31c866497b..fbd053ec9c06e 100644 --- a/.github/workflows/_release.yml +++ b/.github/workflows/_release.yml @@ -166,18 +166,31 @@ jobs: - name: Run integration tests if: ${{ startsWith(inputs.working-directory, 'libs/partners/') }} env: + AI21_API_KEY: ${{ secrets.AI21_API_KEY }} GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }} ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + AZURE_OPENAI_API_VERSION: ${{ secrets.AZURE_OPENAI_API_VERSION }} + AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }} + AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }} + AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }} + AZURE_OPENAI_LLM_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_LLM_DEPLOYMENT_NAME }} + AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME }} NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }} GOOGLE_SEARCH_API_KEY: ${{ secrets.GOOGLE_SEARCH_API_KEY }} GOOGLE_CSE_ID: ${{ secrets.GOOGLE_CSE_ID }} + GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }} EXA_API_KEY: ${{ secrets.EXA_API_KEY }} NOMIC_API_KEY: ${{ secrets.NOMIC_API_KEY }} + WATSONX_APIKEY: ${{ secrets.WATSONX_APIKEY }} + WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }} PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }} PINECONE_ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }} + ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }} + ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }} + ASTRA_DB_KEYSPACE: ${{ secrets.ASTRA_DB_KEYSPACE }} run: make integration_tests working-directory: ${{ inputs.working-directory }} diff --git a/.github/workflows/api_doc_build.yml b/.github/workflows/api_doc_build.yml new file mode 100644 index 0000000000000..7f88768c56a62 --- /dev/null +++ b/.github/workflows/api_doc_build.yml @@ -0,0 +1,52 @@ +name: API docs build + +on: + workflow_dispatch: + schedule: + - cron: '0 13 * * *' +env: + POETRY_VERSION: "1.7.1" + PYTHON_VERSION: "3.10" + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + ref: bagatur/api_docs_build + + - name: Set Git config + run: | + git config --local user.email "actions@github.com" + git config --local user.name "Github Actions" + + - name: Merge master + run: | + git fetch origin master + git merge origin/master -m "Merge master" --allow-unrelated-histories -X theirs + + - name: Set up Python ${{ env.PYTHON_VERSION }} + Poetry ${{ env.POETRY_VERSION }} + uses: "./.github/actions/poetry_setup" + with: + python-version: ${{ env.PYTHON_VERSION }} + poetry-version: ${{ env.POETRY_VERSION }} + cache-key: api-docs + + - name: Install dependencies + run: | + poetry run python -m pip install --upgrade --no-cache-dir pip setuptools + poetry run python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext + poetry run python -m pip install ./libs/partners/* + poetry run python -m pip install --exists-action=w --no-cache-dir -r docs/api_reference/requirements.txt + + - name: Build docs + run: | + poetry run python -m pip install --upgrade --no-cache-dir pip setuptools + poetry run python docs/api_reference/create_api_rst.py + poetry run python -m sphinx -T -E -b html -d _build/doctrees -c docs/api_reference docs/api_reference api_reference_build/html -j auto + + # https://github.com/marketplace/actions/add-commit + - uses: EndBug/add-and-commit@v9 + with: + message: 'Update API docs build' diff --git a/.github/workflows/check_diffs.yml b/.github/workflows/check_diffs.yml index 9a2e11dd3a0e4..c4bd8b448b826 100644 --- a/.github/workflows/check_diffs.yml +++ b/.github/workflows/check_diffs.yml @@ -16,6 +16,9 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true +env: + POETRY_VERSION: "1.7.1" + jobs: build: runs-on: ubuntu-latest @@ -30,15 +33,119 @@ jobs: run: | python .github/scripts/check_diff.py ${{ steps.files.outputs.all }} >> $GITHUB_OUTPUT outputs: - dirs-to-run: ${{ steps.set-matrix.outputs.dirs-to-run }} - ci: + dirs-to-lint: ${{ steps.set-matrix.outputs.dirs-to-lint }} + dirs-to-test: ${{ steps.set-matrix.outputs.dirs-to-test }} + dirs-to-extended-test: ${{ steps.set-matrix.outputs.dirs-to-extended-test }} + lint: + name: cd ${{ matrix.working-directory }} + needs: [ build ] + if: ${{ needs.build.outputs.dirs-to-lint != '[]' }} + strategy: + matrix: + working-directory: ${{ fromJson(needs.build.outputs.dirs-to-lint) }} + uses: ./.github/workflows/_lint.yml + with: + working-directory: ${{ matrix.working-directory }} + secrets: inherit + + test: name: cd ${{ matrix.working-directory }} needs: [ build ] + if: ${{ needs.build.outputs.dirs-to-test != '[]' }} strategy: matrix: - working-directory: ${{ fromJson(needs.build.outputs.dirs-to-run) }} - uses: ./.github/workflows/_all_ci.yml + working-directory: ${{ fromJson(needs.build.outputs.dirs-to-test) }} + uses: ./.github/workflows/_test.yml with: working-directory: ${{ matrix.working-directory }} + secrets: inherit + compile-integration-tests: + name: cd ${{ matrix.working-directory }} + needs: [ build ] + if: ${{ needs.build.outputs.dirs-to-test != '[]' }} + strategy: + matrix: + working-directory: ${{ fromJson(needs.build.outputs.dirs-to-test) }} + uses: ./.github/workflows/_compile_integration_test.yml + with: + working-directory: ${{ matrix.working-directory }} + secrets: inherit + dependencies: + name: cd ${{ matrix.working-directory }} + needs: [ build ] + if: ${{ needs.build.outputs.dirs-to-test != '[]' }} + strategy: + matrix: + working-directory: ${{ fromJson(needs.build.outputs.dirs-to-test) }} + uses: ./.github/workflows/_dependencies.yml + with: + working-directory: ${{ matrix.working-directory }} + secrets: inherit + + extended-tests: + name: "cd ${{ matrix.working-directory }} / make extended_tests #${{ matrix.python-version }}" + needs: [ build ] + if: ${{ needs.build.outputs.dirs-to-extended-test != '[]' }} + strategy: + matrix: + # note different variable for extended test dirs + working-directory: ${{ fromJson(needs.build.outputs.dirs-to-extended-test) }} + python-version: + - "3.8" + - "3.9" + - "3.10" + - "3.11" + runs-on: ubuntu-latest + defaults: + run: + working-directory: ${{ matrix.working-directory }} + steps: + - uses: actions/checkout@v4 + + - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }} + uses: "./.github/actions/poetry_setup" + with: + python-version: ${{ matrix.python-version }} + poetry-version: ${{ env.POETRY_VERSION }} + working-directory: ${{ matrix.working-directory }} + cache-key: extended + + - name: Install dependencies + shell: bash + run: | + echo "Running extended tests, installing dependencies with poetry..." + poetry install -E extended_testing --with test + + - name: Run extended tests + run: make extended_tests + + - name: Ensure the tests did not create any additional files + shell: bash + run: | + set -eu + + STATUS="$(git status)" + echo "$STATUS" + + # grep will exit non-zero if the target message isn't found, + # and `set -e` above will cause the step to fail. + echo "$STATUS" | grep 'nothing to commit, working tree clean' + ci_success: + name: "CI Success" + needs: [build, lint, test, compile-integration-tests, dependencies, extended-tests] + if: | + always() + runs-on: ubuntu-latest + env: + JOBS_JSON: ${{ toJSON(needs) }} + RESULTS_JSON: ${{ toJSON(needs.*.result) }} + EXIT_CODE: ${{!contains(needs.*.result, 'failure') && !contains(needs.*.result, 'cancelled') && '0' || '1'}} + steps: + - name: "CI Success" + run: | + echo $JOBS_JSON + echo $RESULTS_JSON + echo "Exiting with $EXIT_CODE" + exit $EXIT_CODE diff --git a/.github/workflows/codespell.yml b/.github/workflows/codespell.yml index a5a63b996de86..28ede12c3e4a7 100644 --- a/.github/workflows/codespell.yml +++ b/.github/workflows/codespell.yml @@ -32,5 +32,6 @@ jobs: - name: Codespell uses: codespell-project/actions-codespell@v2 with: - skip: guide_imports.json + skip: guide_imports.json,*.ambr ignore_words_list: ${{ steps.extract_ignore_words.outputs.ignore_words_list }} + exclude_file: libs/community/langchain_community/llms/yuan2.py diff --git a/.github/workflows/doc_lint.yml b/.github/workflows/doc_lint.yml deleted file mode 100644 index 298e1509b0101..0000000000000 --- a/.github/workflows/doc_lint.yml +++ /dev/null @@ -1,37 +0,0 @@ ---- -name: CI / cd . - -on: - push: - branches: [ master ] - pull_request: - paths: - - 'docs/**' - - 'templates/**' - - 'cookbook/**' - - '.github/workflows/_lint.yml' - - '.github/workflows/doc_lint.yml' - workflow_dispatch: - -jobs: - check: - name: Check for "from langchain import x" imports - runs-on: ubuntu-latest - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Run import check - run: | - # We should not encourage imports directly from main init file - # Expect for hub - git grep 'from langchain import' {docs/docs,templates,cookbook} | grep -vE 'from langchain import (hub)' && exit 1 || exit 0 - - lint: - name: "-" - uses: - ./.github/workflows/_lint.yml - with: - working-directory: "." - secrets: inherit \ No newline at end of file diff --git a/.github/workflows/people.yml b/.github/workflows/people.yml new file mode 100644 index 0000000000000..e176dd691717e --- /dev/null +++ b/.github/workflows/people.yml @@ -0,0 +1,36 @@ +name: LangChain People + +on: + schedule: + - cron: "0 14 1 * *" + push: + branches: [jacob/people] + workflow_dispatch: + inputs: + debug_enabled: + description: 'Run the build with tmate debugging enabled (https://github.com/marketplace/actions/debugging-with-tmate)' + required: false + default: 'false' + +jobs: + langchain-people: + if: github.repository_owner == 'langchain-ai' + runs-on: ubuntu-latest + steps: + - name: Dump GitHub context + env: + GITHUB_CONTEXT: ${{ toJson(github) }} + run: echo "$GITHUB_CONTEXT" + - uses: actions/checkout@v4 + # Ref: https://github.com/actions/runner/issues/2033 + - name: Fix git safe.directory in container + run: mkdir -p /home/runner/work/_temp/_github_home && printf "[safe]\n\tdirectory = /github/workspace" > /home/runner/work/_temp/_github_home/.gitconfig + # Allow debugging with tmate + - name: Setup tmate session + uses: mxschmitt/action-tmate@v3 + if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled == 'true' }} + with: + limit-access-to-actor: true + - uses: ./.github/actions/people + with: + token: ${{ secrets.LANGCHAIN_PEOPLE_GITHUB_TOKEN }} \ No newline at end of file diff --git a/.gitignore b/.gitignore index 407a65571adcd..db21b911098a8 100644 --- a/.gitignore +++ b/.gitignore @@ -177,4 +177,6 @@ docs/docs/build docs/docs/node_modules docs/docs/yarn.lock _dist -docs/docs/templates \ No newline at end of file +docs/docs/templates + +prof diff --git a/.readthedocs.yaml b/.readthedocs.yaml index aad4d18ea4c0e..ca809231d9361 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -13,15 +13,8 @@ build: tools: python: "3.11" commands: - - python -m virtualenv $READTHEDOCS_VIRTUALENV_PATH - - python -m pip install --upgrade --no-cache-dir pip setuptools - - python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext - - python -m pip install ./libs/partners/* - - python -m pip install --exists-action=w --no-cache-dir -r docs/api_reference/requirements.txt - - python docs/api_reference/create_api_rst.py - - cat docs/api_reference/conf.py - - python -m sphinx -T -E -b html -d _build/doctrees -c docs/api_reference docs/api_reference $READTHEDOCS_OUTPUT/html -j auto - + - mkdir -p $READTHEDOCS_OUTPUT + - cp -r api_reference_build/* $READTHEDOCS_OUTPUT # Build documentation in the docs/ directory with Sphinx sphinx: configuration: docs/api_reference/conf.py diff --git a/Makefile b/Makefile index c21d230ab474b..5e66e6c07fb85 100644 --- a/Makefile +++ b/Makefile @@ -15,7 +15,12 @@ docs_build: docs/.local_build.sh docs_clean: - rm -r _dist + @if [ -d _dist ]; then \ + rm -r _dist; \ + echo "Directory _dist has been cleaned."; \ + else \ + echo "Nothing to clean."; \ + fi docs_linkcheck: poetry run linkchecker _dist/docs/ --ignore-url node_modules @@ -45,11 +50,13 @@ lint lint_package lint_tests: poetry run ruff docs templates cookbook poetry run ruff format docs templates cookbook --diff poetry run ruff --select I docs templates cookbook + git grep 'from langchain import' {docs/docs,templates,cookbook} | grep -vE 'from langchain import (hub)' && exit 1 || exit 0 format format_diff: poetry run ruff format docs templates cookbook poetry run ruff --select I --fix docs templates cookbook + ###################### # HELP ###################### diff --git a/README.md b/README.md index a9490035ac828..65985434439d5 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Looking for the JS/TS library? Check out [LangChain.js](https://github.com/langc To help you ship LangChain apps to production faster, check out [LangSmith](https://smith.langchain.com). [LangSmith](https://smith.langchain.com) is a unified developer platform for building, testing, and monitoring LLM applications. -Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) to get off the waitlist or speak with our sales team. +Fill out [this form](https://www.langchain.com/contact-sales) to speak with our sales team. ## Quick Install diff --git a/cookbook/advanced_rag_eval.ipynb b/cookbook/advanced_rag_eval.ipynb index 45d424b452d4c..02e86817b29ce 100644 --- a/cookbook/advanced_rag_eval.ipynb +++ b/cookbook/advanced_rag_eval.ipynb @@ -520,7 +520,7 @@ "source": [ "import re\n", "\n", - "from langchain.schema import Document\n", + "from langchain_core.documents import Document\n", "from langchain_core.runnables import RunnableLambda\n", "\n", "\n", diff --git a/cookbook/amazon_personalize_how_to.ipynb b/cookbook/amazon_personalize_how_to.ipynb new file mode 100644 index 0000000000000..7555e39d89494 --- /dev/null +++ b/cookbook/amazon_personalize_how_to.ipynb @@ -0,0 +1,284 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Amazon Personalize\n", + "\n", + "[Amazon Personalize](https://docs.aws.amazon.com/personalize/latest/dg/what-is-personalize.html) is a fully managed machine learning service that uses your data to generate item recommendations for your users. It can also generate user segments based on the users' affinity for certain items or item metadata.\n", + "\n", + "This notebook goes through how to use Amazon Personalize Chain. You need a Amazon Personalize campaign_arn or a recommender_arn before you get started with the below notebook.\n", + "\n", + "Following is a [tutorial](https://github.com/aws-samples/retail-demo-store/blob/master/workshop/1-Personalization/Lab-1-Introduction-and-data-preparation.ipynb) to setup a campaign_arn/recommender_arn on Amazon Personalize. Once the campaign_arn/recommender_arn is setup, you can use it in the langchain ecosystem. \n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Install Dependencies" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "!pip install boto3" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Sample Use-cases" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.1 [Use-case-1] Setup Amazon Personalize Client and retrieve recommendations" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_experimental.recommenders import AmazonPersonalize\n", + "\n", + "recommender_arn = \"\"\n", + "\n", + "client = AmazonPersonalize(\n", + " credentials_profile_name=\"default\",\n", + " region_name=\"us-west-2\",\n", + " recommender_arn=recommender_arn,\n", + ")\n", + "client.get_recommendations(user_id=\"1\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "jupyter": { + "outputs_hidden": false + } + }, + "source": [ + "### 2.2 [Use-case-2] Invoke Personalize Chain for summarizing results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "jupyter": { + "outputs_hidden": false + } + }, + "outputs": [], + "source": [ + "from langchain.llms.bedrock import Bedrock\n", + "from langchain_experimental.recommenders import AmazonPersonalizeChain\n", + "\n", + "bedrock_llm = Bedrock(model_id=\"anthropic.claude-v2\", region_name=\"us-west-2\")\n", + "\n", + "# Create personalize chain\n", + "# Use return_direct=True if you do not want summary\n", + "chain = AmazonPersonalizeChain.from_llm(\n", + " llm=bedrock_llm, client=client, return_direct=False\n", + ")\n", + "response = chain({\"user_id\": \"1\"})\n", + "print(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.3 [Use-Case-3] Invoke Amazon Personalize Chain using your own prompt" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.prompts.prompt import PromptTemplate\n", + "\n", + "RANDOM_PROMPT_QUERY = \"\"\"\n", + "You are a skilled publicist. Write a high-converting marketing email advertising several movies available in a video-on-demand streaming platform next week, \n", + " given the movie and user information below. Your email will leverage the power of storytelling and persuasive language. \n", + " The movies to recommend and their information is contained in the tag. \n", + " All movies in the tag must be recommended. Give a summary of the movies and why the human should watch them. \n", + " Put the email between tags.\n", + "\n", + " \n", + " {result} \n", + " \n", + "\n", + " Assistant:\n", + " \"\"\"\n", + "\n", + "RANDOM_PROMPT = PromptTemplate(input_variables=[\"result\"], template=RANDOM_PROMPT_QUERY)\n", + "\n", + "chain = AmazonPersonalizeChain.from_llm(\n", + " llm=bedrock_llm, client=client, return_direct=False, prompt_template=RANDOM_PROMPT\n", + ")\n", + "chain.run({\"user_id\": \"1\", \"item_id\": \"234\"})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.4 [Use-case-4] Invoke Amazon Personalize in a Sequential Chain " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.chains import LLMChain, SequentialChain\n", + "\n", + "RANDOM_PROMPT_QUERY_2 = \"\"\"\n", + "You are a skilled publicist. Write a high-converting marketing email advertising several movies available in a video-on-demand streaming platform next week, \n", + " given the movie and user information below. Your email will leverage the power of storytelling and persuasive language. \n", + " You want the email to impress the user, so make it appealing to them.\n", + " The movies to recommend and their information is contained in the tag. \n", + " All movies in the tag must be recommended. Give a summary of the movies and why the human should watch them. \n", + " Put the email between tags.\n", + "\n", + " \n", + " {result}\n", + " \n", + "\n", + " Assistant:\n", + " \"\"\"\n", + "\n", + "RANDOM_PROMPT_2 = PromptTemplate(\n", + " input_variables=[\"result\"], template=RANDOM_PROMPT_QUERY_2\n", + ")\n", + "personalize_chain_instance = AmazonPersonalizeChain.from_llm(\n", + " llm=bedrock_llm, client=client, return_direct=True\n", + ")\n", + "random_chain_instance = LLMChain(llm=bedrock_llm, prompt=RANDOM_PROMPT_2)\n", + "overall_chain = SequentialChain(\n", + " chains=[personalize_chain_instance, random_chain_instance],\n", + " input_variables=[\"user_id\"],\n", + " verbose=True,\n", + ")\n", + "overall_chain.run({\"user_id\": \"1\", \"item_id\": \"234\"})" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "jupyter": { + "outputs_hidden": false + } + }, + "source": [ + "### 2.5 [Use-case-5] Invoke Amazon Personalize and retrieve metadata " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "jupyter": { + "outputs_hidden": false + } + }, + "outputs": [], + "source": [ + "recommender_arn = \"\"\n", + "metadata_column_names = [\n", + " \"\",\n", + " \"\",\n", + "]\n", + "metadataMap = {\"ITEMS\": metadata_column_names}\n", + "\n", + "client = AmazonPersonalize(\n", + " credentials_profile_name=\"default\",\n", + " region_name=\"us-west-2\",\n", + " recommender_arn=recommender_arn,\n", + ")\n", + "client.get_recommendations(user_id=\"1\", metadataColumns=metadataMap)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "jupyter": { + "outputs_hidden": false + } + }, + "source": [ + "### 2.6 [Use-Case 6] Invoke Personalize Chain with returned metadata for summarizing results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "jupyter": { + "outputs_hidden": false + } + }, + "outputs": [], + "source": [ + "bedrock_llm = Bedrock(model_id=\"anthropic.claude-v2\", region_name=\"us-west-2\")\n", + "\n", + "# Create personalize chain\n", + "# Use return_direct=True if you do not want summary\n", + "chain = AmazonPersonalizeChain.from_llm(\n", + " llm=bedrock_llm, client=client, return_direct=False\n", + ")\n", + "response = chain({\"user_id\": \"1\", \"metadata_columns\": metadataMap})\n", + "print(response)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + }, + "vscode": { + "interpreter": { + "hash": "15e58ce194949b77a891bd4339ce3d86a9bd138e905926019517993f97db9e6c" + } + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/cookbook/apache_kafka_message_handling.ipynb b/cookbook/apache_kafka_message_handling.ipynb new file mode 100644 index 0000000000000..36a0c07e965bd --- /dev/null +++ b/cookbook/apache_kafka_message_handling.ipynb @@ -0,0 +1,922 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "rT1cmV4qCa2X" + }, + "source": [ + "# Using Apache Kafka to route messages\n", + "\n", + "---\n", + "\n", + "\n", + "\n", + "This notebook shows you how to use LangChain's standard chat features while passing the chat messages back and forth via Apache Kafka.\n", + "\n", + "This goal is to simulate an architecture where the chat front end and the LLM are running as separate services that need to communicate with one another over an internal nework.\n", + "\n", + "It's an alternative to typical pattern of requesting a reponse from the model via a REST API (there's more info on why you would want to do this at the end of the notebook)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UPYtfAR_9YxZ" + }, + "source": [ + "### 1. Install the main dependencies\n", + "\n", + "Dependencies include:\n", + "\n", + "- The Quix Streams library for managing interactions with Apache Kafka (or Kafka-like tools such as Redpanda) in a \"Pandas-like\" way.\n", + "- The LangChain library for managing interactions with Llama-2 and storing conversation state." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ZX5tfKiy9cN-" + }, + "outputs": [], + "source": [ + "!pip install quixstreams==2.1.2a langchain==0.0.340 huggingface_hub==0.19.4 langchain-experimental==0.0.42 python-dotenv" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "losTSdTB9d9O" + }, + "source": [ + "### 2. Build and install the llama-cpp-python library (with CUDA enabled so that we can advantage of Google Colab GPU\n", + "\n", + "The `llama-cpp-python` library is a Python wrapper around the `llama-cpp` library which enables you to efficiently leverage just a CPU to run quantized LLMs.\n", + "\n", + "When you use the standard `pip install llama-cpp-python` command, you do not get GPU support by default. Generation can be very slow if you rely on just the CPU in Google Colab, so the following command adds an extra option to build and install\n", + "`llama-cpp-python` with GPU support (make sure you have a GPU-enabled runtime selected in Google Colab)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-JCQdl1G9tbl" + }, + "outputs": [], + "source": [ + "!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install llama-cpp-python" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5_vjVIAh9rLl" + }, + "source": [ + "### 3. Download and setup Kafka and Zookeeper instances\n", + "\n", + "Download the Kafka binaries from the Apache website and start the servers as daemons. We'll use the default configurations (provided by Apache Kafka) for spinning up the instances." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "zFz7czGRW5Wr" + }, + "outputs": [], + "source": [ + "!curl -sSOL https://dlcdn.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz\n", + "!tar -xzf kafka_2.13-3.6.1.tgz" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Uf7NR_UZ9wye" + }, + "outputs": [], + "source": [ + "!./kafka_2.13-3.6.1/bin/zookeeper-server-start.sh -daemon ./kafka_2.13-3.6.1/config/zookeeper.properties\n", + "!./kafka_2.13-3.6.1/bin/kafka-server-start.sh -daemon ./kafka_2.13-3.6.1/config/server.properties\n", + "!echo \"Waiting for 10 secs until kafka and zookeeper services are up and running\"\n", + "!sleep 10" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "H3SafFuS94p1" + }, + "source": [ + "### 4. Check that the Kafka Daemons are running\n", + "\n", + "Show the running processes and filter it for Java processes (you should see two—one for each server)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CZDC2lQP99yp" + }, + "outputs": [], + "source": [ + "!ps aux | grep -E '[j]ava'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Snoxmjb5-V37" + }, + "source": [ + "### 5. Import the required dependencies and initialize required variables\n", + "\n", + "Import the Quix Streams library for interacting with Kafka, and the necessary LangChain components for running a `ConversationChain`." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "plR9e_MF-XL5" + }, + "outputs": [], + "source": [ + "# Import utility libraries\n", + "import json\n", + "import random\n", + "import re\n", + "import time\n", + "import uuid\n", + "from os import environ\n", + "from pathlib import Path\n", + "from random import choice, randint, random\n", + "\n", + "from dotenv import load_dotenv\n", + "\n", + "# Import a Hugging Face utility to download models directly from Hugging Face hub:\n", + "from huggingface_hub import hf_hub_download\n", + "from langchain.chains import ConversationChain\n", + "\n", + "# Import Langchain modules for managing prompts and conversation chains:\n", + "from langchain.llms import LlamaCpp\n", + "from langchain.memory import ConversationTokenBufferMemory\n", + "from langchain.prompts import PromptTemplate, load_prompt\n", + "from langchain_core.messages import SystemMessage\n", + "from langchain_experimental.chat_models import Llama2Chat\n", + "from quixstreams import Application, State, message_key\n", + "\n", + "# Import Quix dependencies\n", + "from quixstreams.kafka import Producer\n", + "\n", + "# Initialize global variables.\n", + "AGENT_ROLE = \"AI\"\n", + "chat_id = \"\"\n", + "\n", + "# Set the current role to the role constant and initialize variables for supplementary customer metadata:\n", + "role = AGENT_ROLE" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HgJjJ9aZ-liy" + }, + "source": [ + "### 6. Download the \"llama-2-7b-chat.Q4_K_M.gguf\" model\n", + "\n", + "Download the quantized LLama-2 7B model from Hugging Face which we will use as a local LLM (rather than relying on REST API calls to an external service)." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 67, + "referenced_widgets": [ + "969343cdbe604a26926679bbf8bd2dda", + "d8b8370c9b514715be7618bfe6832844", + "0def954cca89466b8408fadaf3b82e64", + "462482accc664729980562e208ceb179", + "80d842f73c564dc7b7cc316c763e2633", + "fa055d9f2a9d4a789e9cf3c89e0214e5", + "30ecca964a394109ac2ad757e3aec6c0", + "fb6478ce2dac489bb633b23ba0953c5c", + "734b0f5da9fc4307a95bab48cdbb5d89", + "b32f3a86a74741348511f4e136744ac8", + "e409071bff5a4e2d9bf0e9f5cc42231b" + ] + }, + "id": "Qwu4YoSA-503", + "outputId": "f956976c-7485-415b-ac93-4336ade31964" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The model path does not exist in state. Downloading model...\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "969343cdbe604a26926679bbf8bd2dda", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "llama-2-7b-chat.Q4_K_M.gguf: 0%| | 0.00/4.08G [00:00 \u001b[0m\u001b[32;49m24.0\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n", + "Found existing installation: langchain-fireworks 0.0.1\n", + "Uninstalling langchain-fireworks-0.0.1:\n", + " Successfully uninstalled langchain-fireworks-0.0.1\n", + "Note: you may need to restart the kernel to use updated packages.\n", + "Obtaining file:///mnt/disks/data/langchain/libs/partners/fireworks\n", + " Installing build dependencies ... \u001b[?25ldone\n", + "\u001b[?25h Checking if build backend supports build_editable ... \u001b[?25ldone\n", + "\u001b[?25h Getting requirements to build editable ... \u001b[?25ldone\n", + "\u001b[?25h Preparing editable metadata (pyproject.toml) ... \u001b[?25ldone\n", + "\u001b[?25hRequirement already satisfied: aiohttp<4.0.0,>=3.9.1 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-fireworks==0.0.1) (3.9.3)\n", + "Requirement already satisfied: fireworks-ai<0.13.0,>=0.12.0 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-fireworks==0.0.1) (0.12.0)\n", + "Requirement already satisfied: langchain-core<0.2,>=0.1 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-fireworks==0.0.1) (0.1.23)\n", + "Requirement already satisfied: requests<3,>=2 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-fireworks==0.0.1) (2.31.0)\n", + "Requirement already satisfied: aiosignal>=1.1.2 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.9.1->langchain-fireworks==0.0.1) (1.3.1)\n", + "Requirement already satisfied: attrs>=17.3.0 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.9.1->langchain-fireworks==0.0.1) (23.1.0)\n", + "Requirement already satisfied: frozenlist>=1.1.1 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.9.1->langchain-fireworks==0.0.1) (1.4.0)\n", + "Requirement already satisfied: multidict<7.0,>=4.5 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.9.1->langchain-fireworks==0.0.1) (6.0.4)\n", + "Requirement already satisfied: yarl<2.0,>=1.0 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.9.1->langchain-fireworks==0.0.1) (1.9.2)\n", + "Requirement already satisfied: async-timeout<5.0,>=4.0 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.9.1->langchain-fireworks==0.0.1) (4.0.3)\n", + "Requirement already satisfied: httpx in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (0.26.0)\n", + "Requirement already satisfied: httpx-sse in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (0.4.0)\n", + "Requirement already satisfied: pydantic in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (2.4.2)\n", + "Requirement already satisfied: Pillow in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (10.2.0)\n", + "Requirement already satisfied: PyYAML>=5.3 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (6.0.1)\n", + "Requirement already satisfied: anyio<5,>=3 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (3.7.1)\n", + "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (1.33)\n", + "Requirement already satisfied: langsmith<0.2.0,>=0.1.0 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (0.1.5)\n", + "Requirement already satisfied: packaging<24.0,>=23.2 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (23.2)\n", + "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (8.2.3)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from requests<3,>=2->langchain-fireworks==0.0.1) (3.3.0)\n", + "Requirement already satisfied: idna<4,>=2.5 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from requests<3,>=2->langchain-fireworks==0.0.1) (3.4)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from requests<3,>=2->langchain-fireworks==0.0.1) (2.0.6)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from requests<3,>=2->langchain-fireworks==0.0.1) (2023.7.22)\n", + "Requirement already satisfied: sniffio>=1.1 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (1.3.0)\n", + "Requirement already satisfied: exceptiongroup in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (1.1.3)\n", + "Requirement already satisfied: jsonpointer>=1.9 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.2,>=0.1->langchain-fireworks==0.0.1) (2.4)\n", + "Requirement already satisfied: annotated-types>=0.4.0 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from pydantic->fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (0.5.0)\n", + "Requirement already satisfied: pydantic-core==2.10.1 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from pydantic->fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (2.10.1)\n", + "Requirement already satisfied: typing-extensions>=4.6.1 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from pydantic->fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (4.8.0)\n", + "Requirement already satisfied: httpcore==1.* in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from httpx->fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (1.0.2)\n", + "Requirement already satisfied: h11<0.15,>=0.13 in /mnt/disks/data/langchain/.venv/lib/python3.9/site-packages (from httpcore==1.*->httpx->fireworks-ai<0.13.0,>=0.12.0->langchain-fireworks==0.0.1) (0.14.0)\n", + "Building wheels for collected packages: langchain-fireworks\n", + " Building editable for langchain-fireworks (pyproject.toml) ... \u001b[?25ldone\n", + "\u001b[?25h Created wheel for langchain-fireworks: filename=langchain_fireworks-0.0.1-py3-none-any.whl size=2228 sha256=564071b120b09ec31f2dc737733448a33bbb26e40b49fcde0c129ad26045259d\n", + " Stored in directory: /tmp/pip-ephem-wheel-cache-oz368vdk/wheels/e0/ad/31/d7e76dd73d61905ff7f369f5b0d21a4b5e7af4d3cb7487aece\n", + "Successfully built langchain-fireworks\n", + "Installing collected packages: langchain-fireworks\n", + "Successfully installed langchain-fireworks-0.0.1\n", + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install --quiet pypdf chromadb tiktoken openai \n", + "%pip uninstall -y langchain-fireworks\n", + "%pip install --editable /mnt/disks/data/langchain/libs/partners/fireworks" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "cf719376", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "import fireworks\n", + "\n", + "print(fireworks)\n", + "import fireworks.client" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9ab49327-0532-4480-804c-d066c302a322", + "metadata": {}, + "outputs": [], + "source": [ + "# Load\n", + "import requests\n", + "from langchain_community.document_loaders import PyPDFLoader\n", + "\n", + "# Download the PDF from a URL and save it to a temporary location\n", + "url = \"https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf\"\n", + "response = requests.get(url, stream=True)\n", + "file_name = \"temp_file.pdf\"\n", + "with open(file_name, \"wb\") as pdf:\n", + " pdf.write(response.content)\n", + "\n", + "loader = PyPDFLoader(file_name)\n", + "data = loader.load()\n", + "\n", + "# Split\n", + "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", + "\n", + "text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=0)\n", + "all_splits = text_splitter.split_documents(data)\n", + "\n", + "# Add to vectorDB\n", + "from langchain_community.vectorstores import Chroma\n", + "from langchain_fireworks.embeddings import FireworksEmbeddings\n", + "\n", + "vectorstore = Chroma.from_documents(\n", + " documents=all_splits,\n", + " collection_name=\"rag-chroma\",\n", + " embedding=FireworksEmbeddings(),\n", + ")\n", + "\n", + "retriever = vectorstore.as_retriever()" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "4efaddd9-3dbb-455c-ba54-0ad7f2d2ce0f", + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_core.output_parsers import StrOutputParser\n", + "from langchain_core.prompts import ChatPromptTemplate\n", + "from langchain_core.pydantic_v1 import BaseModel\n", + "from langchain_core.runnables import RunnableParallel, RunnablePassthrough\n", + "\n", + "# RAG prompt\n", + "template = \"\"\"Answer the question based only on the following context:\n", + "{context}\n", + "\n", + "Question: {question}\n", + "\"\"\"\n", + "prompt = ChatPromptTemplate.from_template(template)\n", + "\n", + "# LLM\n", + "from langchain_together import Together\n", + "\n", + "llm = Together(\n", + " model=\"mistralai/Mixtral-8x7B-Instruct-v0.1\",\n", + " temperature=0.0,\n", + " max_tokens=2000,\n", + " top_k=1,\n", + ")\n", + "\n", + "# RAG chain\n", + "chain = (\n", + " RunnableParallel({\"context\": retriever, \"question\": RunnablePassthrough()})\n", + " | prompt\n", + " | llm\n", + " | StrOutputParser()\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "88b1ee51-1b0f-4ebf-bb32-e50e843f0eeb", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'\\nAnswer: The architectural details of Mixtral are as follows:\\n- Dimension (dim): 4096\\n- Number of layers (n\\\\_layers): 32\\n- Dimension of each head (head\\\\_dim): 128\\n- Hidden dimension (hidden\\\\_dim): 14336\\n- Number of heads (n\\\\_heads): 32\\n- Number of kv heads (n\\\\_kv\\\\_heads): 8\\n- Context length (context\\\\_len): 32768\\n- Vocabulary size (vocab\\\\_size): 32000\\n- Number of experts (num\\\\_experts): 8\\n- Number of top k experts (top\\\\_k\\\\_experts): 2\\n\\nMixtral is based on a transformer architecture and uses the same modifications as described in [18], with the notable exceptions that Mixtral supports a fully dense context length of 32k tokens, and the feedforward block picks from a set of 8 distinct groups of parameters. At every layer, for every token, a router network chooses two of these groups (the “experts”) to process the token and combine their output additively. This technique increases the number of parameters of a model while controlling cost and latency, as the model only uses a fraction of the total set of parameters per token. Mixtral is pretrained with multilingual data using a context size of 32k tokens. It either matches or exceeds the performance of Llama 2 70B and GPT-3.5, over several benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.'" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "chain.invoke(\"What are the Architectural details of Mixtral?\")" + ] + }, + { + "cell_type": "markdown", + "id": "755cf871-26b7-4e30-8b91-9ffd698470f4", + "metadata": {}, + "source": [ + "Trace: \n", + "\n", + "https://smith.langchain.com/public/935fd642-06a6-4b42-98e3-6074f93115cd/r" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/cookbook/forward_looking_retrieval_augmented_generation.ipynb b/cookbook/forward_looking_retrieval_augmented_generation.ipynb index 0abfe0bfeff60..4406c1812db08 100644 --- a/cookbook/forward_looking_retrieval_augmented_generation.ipynb +++ b/cookbook/forward_looking_retrieval_augmented_generation.ipynb @@ -73,8 +73,9 @@ " AsyncCallbackManagerForRetrieverRun,\n", " CallbackManagerForRetrieverRun,\n", ")\n", - "from langchain.schema import BaseRetriever, Document\n", "from langchain_community.utilities import GoogleSerperAPIWrapper\n", + "from langchain_core.documents import Document\n", + "from langchain_core.retrievers import BaseRetriever\n", "from langchain_openai import ChatOpenAI, OpenAI" ] }, diff --git a/cookbook/openai_functions_retrieval_qa.ipynb b/cookbook/openai_functions_retrieval_qa.ipynb index 648b28b5e2c17..621e997088e32 100644 --- a/cookbook/openai_functions_retrieval_qa.ipynb +++ b/cookbook/openai_functions_retrieval_qa.ipynb @@ -358,7 +358,7 @@ "\n", "from langchain.chains.openai_functions import create_qa_with_structure_chain\n", "from langchain.prompts.chat import ChatPromptTemplate, HumanMessagePromptTemplate\n", - "from langchain.schema import HumanMessage, SystemMessage\n", + "from langchain_core.messages import HumanMessage, SystemMessage\n", "from pydantic import BaseModel, Field" ] }, diff --git a/cookbook/rag_fusion.ipynb b/cookbook/rag_fusion.ipynb index 976e8cfab41cb..5cac01e9076cd 100644 --- a/cookbook/rag_fusion.ipynb +++ b/cookbook/rag_fusion.ipynb @@ -19,7 +19,9 @@ "source": [ "## Setup\n", "\n", - "For this example, we will use Pinecone and some fake data" + "For this example, we will use Pinecone and some fake data. To configure Pinecone, set the following environment variable:\n", + "\n", + "- `PINECONE_API_KEY`: Your Pinecone API key" ] }, { @@ -29,11 +31,8 @@ "metadata": {}, "outputs": [], "source": [ - "import pinecone\n", - "from langchain_community.vectorstores import Pinecone\n", "from langchain_openai import OpenAIEmbeddings\n", - "\n", - "pinecone.init(api_key=\"...\", environment=\"...\")" + "from langchain_pinecone import PineconeVectorStore" ] }, { @@ -64,7 +63,7 @@ "metadata": {}, "outputs": [], "source": [ - "vectorstore = Pinecone.from_texts(\n", + "vectorstore = PineconeVectorStore.from_texts(\n", " list(all_documents.values()), OpenAIEmbeddings(), index_name=\"rag-fusion\"\n", ")" ] @@ -162,7 +161,7 @@ "metadata": {}, "outputs": [], "source": [ - "vectorstore = Pinecone.from_existing_index(\"rag-fusion\", OpenAIEmbeddings())\n", + "vectorstore = PineconeVectorStore.from_existing_index(\"rag-fusion\", OpenAIEmbeddings())\n", "retriever = vectorstore.as_retriever()" ] }, diff --git a/cookbook/rag_with_quantized_embeddings.ipynb b/cookbook/rag_with_quantized_embeddings.ipynb new file mode 100644 index 0000000000000..a0113ff6ce8f7 --- /dev/null +++ b/cookbook/rag_with_quantized_embeddings.ipynb @@ -0,0 +1,591 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "6195da33-34c3-4ca2-943a-050b6dcbacbc", + "metadata": {}, + "source": [ + "# Embedding Documents using Optimized and Quantized Embedders\n", + "\n", + "In this tutorial, we will demo how to build a RAG pipeline, with the embedding for all documents done using Quantized Embedders.\n", + "\n", + "We will use a pipeline that will:\n", + "\n", + "* Create a document collection.\n", + "* Embed all documents using Quantized Embedders.\n", + "* Fetch relevant documents for our question.\n", + "* Run an LLM answer the question.\n", + "\n", + "For more information about optimized models, we refer to [optimum-intel](https://github.com/huggingface/optimum-intel.git) and [IPEX](https://github.com/intel/intel-extension-for-pytorch).\n", + "\n", + "This tutorial is based on the [Langchain RAG tutorial here](https://towardsai.net/p/machine-learning/dense-x-retrieval-technique-in-langchain-and-llamaindex)." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "26db2da5-3733-4a90-909e-6c11508ea140", + "metadata": {}, + "outputs": [], + "source": [ + "import uuid\n", + "from pathlib import Path\n", + "\n", + "import langchain\n", + "import torch\n", + "from bs4 import BeautifulSoup as Soup\n", + "from langchain.retrievers.multi_vector import MultiVectorRetriever\n", + "from langchain.storage import InMemoryByteStore, LocalFileStore\n", + "\n", + "# For our example, we'll load docs from the web\n", + "from langchain.text_splitter import RecursiveCharacterTextSplitter # noqa\n", + "from langchain_community.document_loaders.recursive_url_loader import (\n", + " RecursiveUrlLoader,\n", + ")\n", + "\n", + "# noqa\n", + "from langchain_community.vectorstores import Chroma\n", + "\n", + "DOCSTORE_DIR = \".\"\n", + "DOCSTORE_ID_KEY = \"doc_id\"" + ] + }, + { + "cell_type": "markdown", + "id": "f5ccda4e-7af5-4355-b9c4-25547edf33f9", + "metadata": {}, + "source": [ + "Lets first load up this paper, and split into text chunks of size 1000." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "5f4d8888-53a6-49f5-a198-da5c92419ca4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 1 documents\n", + "Split into 73 documents\n" + ] + } + ], + "source": [ + "# Could add more parsing here, as it's very raw.\n", + "loader = RecursiveUrlLoader(\n", + " \"https://ar5iv.labs.arxiv.org/html/1706.03762\",\n", + " max_depth=2,\n", + " extractor=lambda x: Soup(x, \"html.parser\").text,\n", + ")\n", + "data = loader.load()\n", + "print(f\"Loaded {len(data)} documents\")\n", + "\n", + "# Split\n", + "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n", + "all_splits = text_splitter.split_documents(data)\n", + "print(f\"Split into {len(all_splits)} documents\")" + ] + }, + { + "cell_type": "markdown", + "id": "73e90632-2ac2-49eb-80da-ffe9ac4a278d", + "metadata": {}, + "source": [ + "In order to embed our documents, we can use the ```QuantizedBiEncoderEmbeddings```, for efficient and fast embedding. " + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "9a68a6f6-332d-481e-bbea-ad763155ea36", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "89af89b48c55409b9999b8e0387fab5b", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "config.json: 0%| | 0.00/747 [00:00 2:chain:RunnableParallel] Entering Chain run with input:\n", + "\u001b[0m{\n", + " \"input\": \"What is the first transduction model relying entirely on self-attention?\"\n", + "}\n", + "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RunnableSequence > 2:chain:RunnableParallel > 4:chain:RunnablePassthrough] Entering Chain run with input:\n", + "\u001b[0m{\n", + " \"input\": \"What is the first transduction model relying entirely on self-attention?\"\n", + "}\n", + "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RunnableSequence > 2:chain:RunnableParallel > 4:chain:RunnablePassthrough] [1ms] Exiting Chain run with output:\n", + "\u001b[0m{\n", + " \"output\": \"What is the first transduction model relying entirely on self-attention?\"\n", + "}\n", + "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RunnableSequence > 2:chain:RunnableParallel] [66ms] Exiting Chain run with output:\n", + "\u001b[0m[outputs]\n", + "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RunnableSequence > 5:prompt:ChatPromptTemplate] Entering Prompt run with input:\n", + "\u001b[0m[inputs]\n", + "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RunnableSequence > 5:prompt:ChatPromptTemplate] [1ms] Exiting Prompt run with output:\n", + "\u001b[0m{\n", + " \"lc\": 1,\n", + " \"type\": \"constructor\",\n", + " \"id\": [\n", + " \"langchain\",\n", + " \"prompts\",\n", + " \"chat\",\n", + " \"ChatPromptValue\"\n", + " ],\n", + " \"kwargs\": {\n", + " \"messages\": [\n", + " {\n", + " \"lc\": 1,\n", + " \"type\": \"constructor\",\n", + " \"id\": [\n", + " \"langchain\",\n", + " \"schema\",\n", + " \"messages\",\n", + " \"HumanMessage\"\n", + " ],\n", + " \"kwargs\": {\n", + " \"content\": \"You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\\nQuestion: What is the first transduction model relying entirely on self-attention? \\nContext: [Document(page_content='To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.\\\\nIn the following sections, we will describe the Transformer, motivate self-attention and discuss its advantages over models such as (neural_gpu, ; NalBytenet2017, ) and (JonasFaceNet2017, ).\\\\n\\\\n\\\\n\\\\n\\\\n3 Model Architecture\\\\n\\\\nFigure 1: The Transformer - model architecture.', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'}), Document(page_content='In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention.\\\\n\\\\n\\\\nFor translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles. \\\\n\\\\n\\\\nWe are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video.\\\\nMaking generation less sequential is another research goals of ours.', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'}), Document(page_content='Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences (bahdanau2014neural, ; structuredAttentionNetworks, ). In all but a few cases (decomposableAttnModel, ), however, such attention mechanisms are used in conjunction with a recurrent network.\\\\n\\\\n\\\\nIn this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n2 Background', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'}), Document(page_content='The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'})] \\nAnswer:\",\n", + " \"additional_kwargs\": {}\n", + " }\n", + " }\n", + " ]\n", + " }\n", + "}\n", + "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:chain:RunnableSequence > 6:llm:HuggingFacePipeline] Entering LLM run with input:\n", + "\u001b[0m{\n", + " \"prompts\": [\n", + " \"Human: You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\\nQuestion: What is the first transduction model relying entirely on self-attention? \\nContext: [Document(page_content='To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.\\\\nIn the following sections, we will describe the Transformer, motivate self-attention and discuss its advantages over models such as (neural_gpu, ; NalBytenet2017, ) and (JonasFaceNet2017, ).\\\\n\\\\n\\\\n\\\\n\\\\n3 Model Architecture\\\\n\\\\nFigure 1: The Transformer - model architecture.', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'}), Document(page_content='In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention.\\\\n\\\\n\\\\nFor translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles. \\\\n\\\\n\\\\nWe are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video.\\\\nMaking generation less sequential is another research goals of ours.', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'}), Document(page_content='Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences (bahdanau2014neural, ; structuredAttentionNetworks, ). In all but a few cases (decomposableAttnModel, ), however, such attention mechanisms are used in conjunction with a recurrent network.\\\\n\\\\n\\\\nIn this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n2 Background', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'}), Document(page_content='The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the', metadata={'source': 'https://ar5iv.labs.arxiv.org/html/1706.03762', 'title': '[1706.03762] Attention Is All You Need', 'language': 'en'})] \\nAnswer:\"\n", + " ]\n", + "}\n", + "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:chain:RunnableSequence > 6:llm:HuggingFacePipeline] [4.34s] Exiting LLM run with output:\n", + "\u001b[0m{\n", + " \"generations\": [\n", + " [\n", + " {\n", + " \"text\": \" The first transduction model relying entirely on self-attention is the Transformer.\",\n", + " \"generation_info\": null,\n", + " \"type\": \"Generation\"\n", + " }\n", + " ]\n", + " ],\n", + " \"llm_output\": null,\n", + " \"run\": null\n", + "}\n", + "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RunnableSequence] [4.41s] Exiting Chain run with output:\n", + "\u001b[0m{\n", + " \"output\": \" The first transduction model relying entirely on self-attention is the Transformer.\"\n", + "}\n" + ] + } + ], + "source": [ + "langchain.verbose = True\n", + "langchain.debug = True\n", + "\n", + "llm_res = rag_chain.invoke(\n", + " \"What is the first transduction model relying entirely on self-attention?\",\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "023404a1-401a-46e1-8ab5-cafbc8593b04", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "' The first transduction model relying entirely on self-attention is the Transformer.'" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "llm_res" + ] + }, + { + "cell_type": "markdown", + "id": "0eaefd01-254a-445d-a95f-37889c126e0e", + "metadata": {}, + "source": [ + "Based on the retrieved documents, the answer is indeed correct :)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.18" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/cookbook/sales_agent_with_context.ipynb b/cookbook/sales_agent_with_context.ipynb index 158329a5f09e6..e125046af9228 100644 --- a/cookbook/sales_agent_with_context.ipynb +++ b/cookbook/sales_agent_with_context.ipynb @@ -51,10 +51,10 @@ "from langchain.chains.base import Chain\n", "from langchain.prompts import PromptTemplate\n", "from langchain.prompts.base import StringPromptTemplate\n", - "from langchain.schema import AgentAction, AgentFinish\n", "from langchain.text_splitter import CharacterTextSplitter\n", "from langchain_community.llms import BaseLLM\n", "from langchain_community.vectorstores import Chroma\n", + "from langchain_core.agents import AgentAction, AgentFinish\n", "from langchain_openai import ChatOpenAI, OpenAI, OpenAIEmbeddings\n", "from pydantic import BaseModel, Field" ] diff --git a/cookbook/wikibase_agent.ipynb b/cookbook/wikibase_agent.ipynb index 692193b0229df..13c4063cf7852 100644 --- a/cookbook/wikibase_agent.ipynb +++ b/cookbook/wikibase_agent.ipynb @@ -401,7 +401,7 @@ ")\n", "from langchain.chains import LLMChain\n", "from langchain.prompts import StringPromptTemplate\n", - "from langchain.schema import AgentAction, AgentFinish" + "from langchain_core.agents import AgentAction, AgentFinish" ] }, { diff --git a/docker/Makefile b/docker/Makefile new file mode 100644 index 0000000000000..d578580c32396 --- /dev/null +++ b/docker/Makefile @@ -0,0 +1,12 @@ +# Makefile + +build_graphdb: + docker build --tag graphdb ./graphdb + +start_graphdb: + docker-compose up -d graphdb + +down: + docker-compose down -v --remove-orphans + +.PHONY: build_graphdb start_graphdb down diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index ce680ccafda51..33c873e60e0af 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -1,5 +1,10 @@ # docker-compose to make it easier to spin up integration tests. # Services should use NON standard ports to avoid collision with +# any existing services that might be used for development. +# ATTENTION: When adding a service below use a non-standard port +# increment by one from the preceding port. +# For credentials always use `langchain` and `langchain` for the +# username and password. version: "3" name: langchain-tests @@ -15,3 +20,15 @@ services: - "6020:6379" volumes: - ./redis-volume:/data + graphdb: + image: graphdb + ports: + - "6021:7200" + mongo: + image: mongo:latest + container_name: mongo_container + ports: + - "6022:27017" + environment: + MONGO_INITDB_ROOT_USERNAME: langchain + MONGO_INITDB_ROOT_PASSWORD: langchain diff --git a/docker/graphdb/Dockerfile b/docker/graphdb/Dockerfile new file mode 100644 index 0000000000000..dfcbe7e622d7a --- /dev/null +++ b/docker/graphdb/Dockerfile @@ -0,0 +1,5 @@ +FROM ontotext/graphdb:10.5.1 +RUN mkdir -p /opt/graphdb/dist/data/repositories/langchain +COPY config.ttl /opt/graphdb/dist/data/repositories/langchain/ +COPY graphdb_create.sh /run.sh +ENTRYPOINT bash /run.sh diff --git a/docker/graphdb/config.ttl b/docker/graphdb/config.ttl new file mode 100644 index 0000000000000..dcbdeeebe1283 --- /dev/null +++ b/docker/graphdb/config.ttl @@ -0,0 +1,46 @@ +@prefix rdfs: . +@prefix rep: . +@prefix sr: . +@prefix sail: . +@prefix graphdb: . + +[] a rep:Repository ; + rep:repositoryID "langchain" ; + rdfs:label "" ; + rep:repositoryImpl [ + rep:repositoryType "graphdb:SailRepository" ; + sr:sailImpl [ + sail:sailType "graphdb:Sail" ; + + graphdb:read-only "false" ; + + # Inference and Validation + graphdb:ruleset "empty" ; + graphdb:disable-sameAs "true" ; + graphdb:check-for-inconsistencies "false" ; + + # Indexing + graphdb:entity-id-size "32" ; + graphdb:enable-context-index "false" ; + graphdb:enablePredicateList "true" ; + graphdb:enable-fts-index "false" ; + graphdb:fts-indexes ("default" "iri") ; + graphdb:fts-string-literals-index "default" ; + graphdb:fts-iris-index "none" ; + + # Queries and Updates + graphdb:query-timeout "0" ; + graphdb:throw-QueryEvaluationException-on-timeout "false" ; + graphdb:query-limit-results "0" ; + + # Settable in the file but otherwise hidden in the UI and in the RDF4J console + graphdb:base-URL "http://example.org/owlim#" ; + graphdb:defaultNS "" ; + graphdb:imports "" ; + graphdb:repository-type "file-repository" ; + graphdb:storage-folder "storage" ; + graphdb:entity-index-size "10000000" ; + graphdb:in-memory-literal-properties "true" ; + graphdb:enable-literal-index "true" ; + ] + ]. diff --git a/docker/graphdb/graphdb_create.sh b/docker/graphdb/graphdb_create.sh new file mode 100644 index 0000000000000..52ffe8ad74a06 --- /dev/null +++ b/docker/graphdb/graphdb_create.sh @@ -0,0 +1,28 @@ +#! /bin/bash +REPOSITORY_ID="langchain" +GRAPHDB_URI="http://localhost:7200/" + +echo -e "\nUsing GraphDB: ${GRAPHDB_URI}" + +function startGraphDB { + echo -e "\nStarting GraphDB..." + exec /opt/graphdb/dist/bin/graphdb +} + +function waitGraphDBStart { + echo -e "\nWaiting GraphDB to start..." + for _ in $(seq 1 5); do + CHECK_RES=$(curl --silent --write-out '%{http_code}' --output /dev/null ${GRAPHDB_URI}/rest/repositories) + if [ "${CHECK_RES}" = '200' ]; then + echo -e "\nUp and running" + break + fi + sleep 30s + echo "CHECK_RES: ${CHECK_RES}" + done +} + + +startGraphDB & +waitGraphDBStart +wait diff --git a/docs/api_reference/conf.py b/docs/api_reference/conf.py index e993048fc57f4..6ba5c06d97db6 100644 --- a/docs/api_reference/conf.py +++ b/docs/api_reference/conf.py @@ -49,7 +49,7 @@ def run(self): class_or_func_name = self.arguments[0] links = imported_classes.get(class_or_func_name, {}) list_node = nodes.bullet_list() - for doc_name, link in links.items(): + for doc_name, link in sorted(links.items()): item_node = nodes.list_item() para_node = nodes.paragraph() link_node = nodes.reference() @@ -114,8 +114,8 @@ def setup(app): autodoc_member_order = "groupwise" autoclass_content = "both" autodoc_typehints_format = "short" +autodoc_typehints = "both" -# autodoc_typehints = "description" # Add any paths that contain templates here, relative to this directory. templates_path = ["templates"] diff --git a/docs/api_reference/create_api_rst.py b/docs/api_reference/create_api_rst.py index 9413d90423e2f..88e453821441d 100644 --- a/docs/api_reference/create_api_rst.py +++ b/docs/api_reference/create_api_rst.py @@ -14,7 +14,6 @@ ROOT_DIR = Path(__file__).parents[2].absolute() HERE = Path(__file__).parent - ClassKind = Literal["TypedDict", "Regular", "Pydantic", "enum"] @@ -218,8 +217,8 @@ def _construct_doc( for module in namespaces: _members = members_by_namespace[module] - classes = _members["classes_"] - functions = _members["functions"] + classes = [el for el in _members["classes_"] if el["is_public"]] + functions = [el for el in _members["functions"] if el["is_public"]] if not (classes or functions): continue section = f":mod:`{package_namespace}.{module}`" @@ -245,9 +244,6 @@ def _construct_doc( """ for class_ in sorted(classes, key=lambda c: c["qualified_name"]): - if not class_["is_public"]: - continue - if class_["kind"] == "TypedDict": template = "typeddict.rst" elif class_["kind"] == "enum": @@ -265,7 +261,7 @@ def _construct_doc( """ if functions: - _functions = [f["qualified_name"] for f in functions if f["is_public"]] + _functions = [f["qualified_name"] for f in functions] fstring = "\n ".join(sorted(_functions)) full_doc += f"""\ Functions @@ -323,30 +319,52 @@ def _package_dir(package_name: str = "langchain") -> Path: def _get_package_version(package_dir: Path) -> str: - with open(package_dir.parent / "pyproject.toml", "r") as f: - pyproject = toml.load(f) + """Return the version of the package.""" + try: + with open(package_dir.parent / "pyproject.toml", "r") as f: + pyproject = toml.load(f) + except FileNotFoundError as e: + print( + f"pyproject.toml not found in {package_dir.parent}.\n" + "You are either attempting to build a directory which is not a package or " + "the package is missing a pyproject.toml file which should be added." + "Aborting the build." + ) + exit(1) return pyproject["tool"]["poetry"]["version"] -def _out_file_path(package_name: str = "langchain") -> Path: +def _out_file_path(package_name: str) -> Path: """Return the path to the file containing the documentation.""" return HERE / f"{package_name.replace('-', '_')}_api_reference.rst" -def _doc_first_line(package_name: str = "langchain") -> str: +def _doc_first_line(package_name: str) -> str: """Return the path to the file containing the documentation.""" return f".. {package_name.replace('-', '_')}_api_reference:\n\n" def main() -> None: """Generate the api_reference.rst file for each package.""" + print("Starting to build API reference files.") for dir in os.listdir(ROOT_DIR / "libs"): + # Skip any hidden directories + # Some of these could be present by mistake in the code base + # e.g., .pytest_cache from running tests from the wrong location. + if dir.startswith("."): + print("Skipping dir:", dir) + continue + if dir in ("cli", "partners"): continue else: + print("Building package:", dir) _build_rst_file(package_name=dir) - for dir in os.listdir(ROOT_DIR / "libs" / "partners"): + partner_packages = os.listdir(ROOT_DIR / "libs" / "partners") + print("Building partner packages:", partner_packages) + for dir in partner_packages: _build_rst_file(package_name=dir) + print("API reference files built.") if __name__ == "__main__": diff --git a/docs/api_reference/themes/scikit-learn-modern/search.html b/docs/api_reference/themes/scikit-learn-modern/search.html index a1ededafb5240..2c9f40f15a6cc 100644 --- a/docs/api_reference/themes/scikit-learn-modern/search.html +++ b/docs/api_reference/themes/scikit-learn-modern/search.html @@ -5,7 +5,7 @@ - +