Skip to content

Latest commit

 

History

History
108 lines (78 loc) · 2.66 KB

README.md

File metadata and controls

108 lines (78 loc) · 2.66 KB

Data Transformation Assistant

This project is a data transformation assistant that uses natural language input to generate and apply structured transformations on a Pandas DataFrame. It leverages OpenAI's GPT-based models to interpret instructions and execute transformations like filtering, selecting columns, and adding calculated columns.

Features

  • Accepts natural language instructions to transform data.
  • Automatically generates transformation commands in a structured JSON format.
  • Supports the following operations:
    • Filter: Filters rows based on a condition.
    • Select: Selects specific columns.
    • Add Column: Adds new calculated columns using expressions.
  • Modular and extensible design for easy maintenance and scalability.

Installation

Prerequisites

  • Python 3.12+
  • A valid OpenAI API key

Steps

  1. Clone this repository:

    git clone https://github.com/NuperSu/dataspell_ai_chat.git
cd your_project

  2. Set up a virtual environment:

    python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

  3. Install dependencies:

    pip install -r requirements.txt

  4. Rename .env.example' file to .env` in the project root and replace placeholder with your OpenAI API key

  5. Run the application:

    python main.py

Example

Input:

Filter rows where salary > 50000 and select the name and salary columns.

Output:

Original DataFrame:
      name  age  salary
0    Alice   25   50000
1      Bob   35   60000
2  Charlie   45   70000
3    David   28   52000

Generated Transformations:
[
    Command(command='filter', parameters={'predicate': 'salary > 50000'}),
    Command(command='select', parameters={'columns': ['name', 'salary']})
]

Transformed DataFrame:
      name  salary
1      Bob   60000
2  Charlie   70000
3    David   52000

Modules

ai_chat/chat_interface.py

  • Description: Manages interaction with the OpenAI API.
  • Key Features:
    • Generates transformation commands based on natural language input.
    • Uses Pydantic to validate LLM output against a predefined schema.

ai_chat/prompt_template.py

  • Description: Contains the LLM prompt template for generating transformations.

transformations/commands.py

  • Description: Implements individual transformation functions.
  • Functions:
    • filter_data: Filters rows based on a condition.
    • select_columns: Selects specific columns.
    • add_column: Adds calculated columns.

transformations/apply_transformations.py

  • Description: Applies a sequence of transformations to a Pandas DataFrame.