The RadioGPT project is an educational project that demonstrates how to build and train language models from scratch. Through a series of progressive notebooks, you'll learn how to create increasingly sophisticated language models, from a basic character-level model to a more advanced chat-capable LLM.

The project consists of three main notebooks, each building upon the previous one:

1. RadioGPT_1_Generateur_de_Moliere.ipynb

   • Introduction to basic language model concepts
   • Character-level tokenization
   • Training a simple model on Molière's works
   • Understanding fundamental LLM components
   • 

2. RadioGPT_2_Larger_LLM_Chat.ipynb

   • Playing with a larger model (83.1M parameters)
   • Working with more sophisticated architectures
   • Improved text generation capabilities
   • Chat-oriented model training
   • 

3. RadioGPT_3_Finally_RadioGPT.ipynb

   • Fine-tuning on radio station data
   • Advanced model architecture
   • Real-world application with radio content
   • Multiple radio station dataset options
   • 

• GPTlite: A lightweight GPT-style transformer
• Modular attention mechanism
• Scalable architecture with configurable parameters
• Support for both training and generation

• Multiple dataset classes:
  • TinyShakespeare: Character-level dataset
  • AlpacaDataset: Chat-oriented dataset

• TinyShakespeare style:
  • petit molière
• ALpaca style:
  • Acquiesce_data_110k_instructions

• Training and evaluation functions
• Model saving and loading
• Text generation utilities
• Performance monitoring

pip install torch datasets tqdm transformers

• GPU recommended on Google Colab (T4 or better)

1. Clone the repository:

git clone https://github.com/yourusername/RadioGPT.git

2. Install dependencies:

pip install torch datasets tqdm transformers

3. Open the notebooks in order:
   • Start with RadioGPT_1_Generateur_de_Moliere.ipynb
   • Progress to RadioGPT_2_Larger_LLM_Chat.ipynb
   • Finally, explore RadioGPT_3_Finally_RadioGPT.ipynb

Contributions are welcome! Please feel free to submit pull requests or open issues for improvements and bug fixes. This has been developed in less than 6 days, so there is a lot of room for improvement. 😉

• Developped by Chloé Lavrat for Radio France
• All rights reserved by Radio France and Chloé Lavrat

• Thanks to Marc Yefimchuk and Jade Moillic for the help in processing the data and improving the notebooks
• Thanks to the "attention is all you need" authors and the GPT-2 authors for the inspiration 🥰
• Thanks to the PyTorch team for the excellent deep learning framework
• Special thanks to contributors and the community