Controllable Text Generation for Large Language Models: A Survey

Xun Liang^1*, Hanyu Wang^1*, Yezhaohui Wang^2*,
Shichao Song¹, Jiawei Yang¹, Simin Niu¹, Jie Hu³, Dan Liu³, Shunyu Yao³, Feiyu Xiong², Zhiyu Li^2†

¹Renmin University of China
²Institute for Advanced Algorithms Research, Shanghai
³China Telecom Research Institute

^*Equal contribution.

^†Corresponding author: Zhiyu Li ([email protected]).

If you find our work helpful, please consider staring our GitHub to stay updated with the latest in Controllable Text Generation!

📰 News

• [2024/08/23] Our paper is published on the arXiv platform: https://arxiv.org/abs/2408.12599.
• [2024/08/23] Our paper secured the second position on Hugging Face's Daily Papers module: https://huggingface.co/papers/2408.12599.
• [2024/08/26] We have updated our paper list, which can now be accessed on our GitHub page.

🔗 Introduction

Welcome to the GitHub repository for our survey paper titled "Controllable Text Generation for Large Language Models: A Survey." This repository includes all the resources, code, and references related to the paper. Our objective is to provide a thorough overview of the techniques and methodologies used to control text generation in large language models (LLMs), with an emphasis on both theoretical underpinnings and practical implementations.

Our survey explores the following key areas:

🎯 Demands of Controllable Text Generation

Controllable Text Generation (CTG) must meet two main requirements:

1. Meeting Predefined Control Conditions: Ensuring that the generated text adheres to specified criteria, such as thematic consistency, safety, and stylistic adherence.

2. Maintaining Text Quality: Ensuring that the text produced is fluent, helpful, and diverse while balancing control with overall quality.

📜 Formal Definition of Controllable Text Generation

We define CTG as follows:

1. Relationship with LLM Capabilities: CTG is an ability dimension that is orthogonal to the objective knowledge capabilities of LLMs, focusing on how information is presented to meet specific needs, such as style or sentiment.

2. Injection of Control Conditions: Control conditions can be integrated into the text generation process at various stages using resources like text corpora, graphs, or databases.

3. Quality of CTG: High-quality CTG strikes a balance between adherence to control conditions and maintaining fluency, coherence, and helpfulness in the generated text.

🗂️ Classification of Controllable Text Generation Tasks

CTG tasks are categorized into two main types:

1. Content Control (Linguistic Control/Hard Control): Focuses on managing content structure, such as format and vocabulary.

2. Attribute Control (Semantic Control/Soft Control): Focuses on managing attributes like sentiment, style, and safety.

🔧 Controllable Text Generation Method Classification

CTG methods are systematically categorized into two stages:

1. Training-Stage Methods: Techniques such as model retraining, fine-tuning, and reinforcement learning that occur during the training phase.

2. Inference-Stage Methods: Techniques such as prompt engineering, latent space manipulation, and decoding-time intervention applied during inference.

📊 Evaluation Methods and Applications

We review the evaluation methods and their applications in CTG:

1. Evaluation Methods: We introduce a range of automatic and human-based evaluation metrics, along with benchmarks that assess the effectiveness of CTG techniques, focusing on how well they balance control and text quality.

2. Applications: We explore CTG applications across both specialized vertical domains and general tasks.

🚀 Challenges and Future Directions

This survey addresses key challenges in CTG research and suggests future directions:

1. Key Challenges: Issues such as achieving precise control, maintaining fluency and coherence, and handling multi-attribute control in complex scenarios.

2. Proposed Appeals: We advocate for a greater focus on real-world applications and the development of robust evaluation frameworks to advance CTG techniques.

This paper aims to provide valuable insights and guidance for researchers and developers working in the field of Controllable Text Generation. All references, along with a Chinese version of this survey, are open-sourced and available at https://github.com/IAAR-Shanghai/CTGSurvey.

🧩 Project Structure

• figures/: Contains all the figures used in the repository.
• latex/: Includes the LaTeX source files for the survey paper.
• CTG_Survey_Chinese.pdf: The Chinese version of the survey paper.
• README.md: This file, providing an overview of the repository.

📚 Paper List

We’ve compiled a comprehensive spreadsheet of all the papers we reviewed, accessible here. A more user-friendly table format is in progress.

Below, you'll find a categorized list of papers from 2023 and 2024, organized by Type, Phase, and Classification.

Type: Method

Training Stage

Retraining

• Fine-Grained Sentiment-Controlled Text Generation Approach Based on Pre-Trained Language Model
  Zhejiang University of Technology, Appl. Sci., 2023 [Paper]
• Lexical Complexity Controlled Sentence Generation for Language Learning
  BLCU, CCL'23, 2023 [Paper]
• Semantic Space Grounded Weighted Decoding for Multi-Attribute Controllable Dialogue Generation
  Shanghai Jiao Tong University, EMNLP'23, 2023 [Paper]
• SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish
  KTH Royal Institute of Technology, arxiv'23, 2023 [Paper]

Fine-Tuning

• Language Detoxification with Attribute-Discriminative Latent Space
  KAIST, ACL'23, 2023 [Paper]
• Controlled Text Generation with Hidden Representation Transformations
  UCLA, ACL'23_findings, 2023 [Paper]
• CLICK: Controllable Text Generation with Sequence Likelihood Contrastive Learning
  THU, ACL'24_findings, 2023 [Paper]
• Seen to Unseen: Exploring Compositional Generalization of Multi-Attribute Controllable Dialogue Generation
  BUPT, ACL'23, 2023 [Paper]
• DeepPress: guided press release topic-aware text generation using ensemble transformers
  Universite´ de Moncton,, Neural Computing and Applications, 2023 [Paper]
• DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generation
  The University of British Columbia, ACL'23, 2023 [Paper]
• Controlled text generation with natural language instructions
  ETH Zürich, ICML'23, 2023 [Paper]
• Controlling keywords and their positions in text generation
  Hitachi, Ltd. Research and Development Group, INLG'23, 2023 [Paper]
• Toward Unified Controllable Text Generation via Regular Expression Instruction
  ISCAS, IJCNLP-AACL'23, 2023 [Paper]
• Controllable Text Generation with Residual Memory Transformer
  BIT, arxiv'23, 2023 [Paper]
• Continuous Language Model Interpolation for Dynamic and Controllable Text Generation
  Harvard University, arxiv'24, 2024 [Paper]
• CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP
  UMD, arxiv'24, 2024 [Paper]
• Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models
  SAP, arxiv'24, 2024 [Paper]
• CTGGAN: Controllable Text Generation with Generative Adversarial Network
  JIUTIAN Team, China Mobile Research, Appl. Sci., 2024 [Paper]
• ECCRG: A Emotion- and Content-Controllable Response Generation Model
  TJU, Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 2024 [Paper]
• LiFi: Lightweight Controlled Text Generation with Fine-Grained Control Codes
  THU, arxiv'24, 2024 [Paper]

Reinforcement Learning

• STEER: Unified Style Transfer with Expert Reinforcement
  University of Washington, EMNLP'23_findings, 2023 [Paper]
• Prompt-Based Length Controlled Generation with Multiple Control Types
  NWPU, ACL'24_findings, 2024 [Paper]
• Reinforcement Learning with Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation
  University of Minnesota, arxiv'24, 2024 [Paper]
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
  PKU, ICLR'24_spotlight, 2024 [Paper]
• Token-level Direct Preference Optimization
  IACAS, arxiv'24, 2024 [Paper]
• Reinforcement Learning with Token-level Feedback for Controllable Text Generation
  HUST, NAACL'24, 2024 [Paper]

Inference Stage

Prompt Engineering

• Controllable Generation of Dialogue Acts for Dialogue Systems via Few-Shot Response Generation and Ranking
  University of California Santa Cruz, SIGDIAL'23, 2023 [Paper]
• PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation
  Johns Hopkins University, SEM'23, 2023 [Paper]
• Harnessing the Plug-and-Play Controller by Prompting
  BUAA, GEM'23, 2023 [Paper]
• An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation
  THU&Meituan, ACL'23, 2023 [Paper]
• Tailor: A Soft-Prompt-Based Approach to Attribute-Based Controlled Text Generation
  Alibaba, ACL'23, 2023 [Paper]
• InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models
  CNU, ACL'24_findings, 2024 [Paper]
• Topic-Oriented Controlled Text Generation for Social Networks
  WHU, Journal of Signal Processing Systems, 2024 [Paper]
• Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation
  University of Toronto, AAAI'24_workshop, 2024 [Paper]
• TrustAgent: Towards Safe and Trustworthy LLM-based Agents through Agent Constitution
  UCSB, arxiv'24, 2024 [Paper]

Latent Space Manipulation

• Activation Addition: Steering Language Models Without Optimization
  UC Berkeley, arxiv'23, 2023 [Paper]
• Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models
  PKU, arxiv'23, 2023 [Paper]
• In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering
  Stanford University, arxiv'23, 2023 [Paper]
• MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space
  ICT CAS, EMNLP'23_findings, 2023 [Paper]
• Miracle: Towards Personalized Dialogue Generation with Latent-Space Multiple Personal Attribute Control
  HUST, EMNLP'23_findings, 2023 [Paper]
• Controllable Text Generation via Probability Density Estimation in the Latent Space
  HIT, EMNLP'23, 2023 [Paper]
• Self-Detoxifying Language Models via Toxification Reversal
  The Hong Kong Polytechnic University, EMNLP'23, 2023 [Paper]
• DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion
  Tongji University, arxiv'24, 2024 [Paper]
• FreeCtrl: Constructing Control Centers with Feedforward Layers for Learning-Free Controllable Text Generation
  NTU, ACL'24, 2024 [Paper]
• InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance
  FuDan, arxiv'24, 2024 [Paper]
• Multi-Aspect Controllable Text Generation with Disentangled Counterfactual Augmentation
  NJU, ACL'24, 2024 [Paper]
• Style Vectors for Steering Generative Large Language Models
  German Aerospace Center (DLR), EACL'24_findings, 2024 [Paper]

Decoding-time Intervention

• Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation
  USTC, EMNLP'23, 2023 [Paper]
• A Block Metropolis-Hastings Sampler for Controllable Energy-based Text Generation
  UCSD, CoNLL'23, 2023 [Paper]
• BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases
  University of Michigan, ACL'23_short, 2023 [Paper]
• Controlled Decoding from Language Models
  Google, NeurIPS_SoLaR'23, 2023 [Paper]
• Focused Prefix Tuning for Controllable Text Generation
  Tokyo Institute of Technology, ACL'23_short, 2023 [Paper]
• Focused Prefix Tuning for Controllable Text Generation
  Tokyo Tech, ACL'23_short, 2023 [Paper]
• Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models
  Cohere For AI, EMNLP'23_findings, 2023 [Paper]
• GRACE: Gradient-guided Controllable Retrieval for Augmenting Attribute-based Text Generation
  National University of Defense Technology, ACL'23_findings, 2023 [Paper]
• An Invariant Learning Characterization of Controlled Text Generation
  Columbia University, ACL'23, 2023 [Paper]
• Style Locality for Controllable Generation with kNN Language Models
  University of Marburg, SIGDIAL'23_TamingLLM workshop, 2023 [Paper]
• Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts
  University of Washington&CMU, ACL'23_short, 2023 [Paper]
• MIL-Decoding: Detoxifying Language Models at Token-Level via Multiple Instance Learning
  PKU, ACL'23, 2023 [Paper]
• Controllable Story Generation Based on Perplexity Minimization
  Vyatka State University, AIST 2023, 2023 [Paper]
• PREADD: Prefix-Adaptive Decoding for Controlled Text Generation
  UC Berkeley, ACL'23_findings, 2023 [Paper]
• Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
  UNC-Chapel Hill, EMNLP'23_short, 2023 [Paper]
• Controlled Text Generation for Black-box Language Models via Score-based Progressive Editor
  Seoul National University, ACL'24, 2023 [Paper]
• Successor Features for Efficient Multisubject Controlled Text Generation
  Microsoft, arxiv'23, 2023 [Paper]
• Controlled Text Generation via Language Model Arithmetic
  ETH Zurich, ICLR'24_spotlight, 2024 [Paper]
• COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability
  UIUC, arxiv'24, 2024 [Paper]
• Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs
  RUC, arxiv'24, 2024 [Paper]
• DECIDER: A Rule-Controllable Decoding Strategy for Language Generation by Imitating Dual-System Cognitive Theory
  BIT, TKDE_submitted, 2024 [Paper]
• Word Embeddings Are Steers for Language Models
  UIUC, ACL'24, 2024 [Paper]
• RAIN: Your Language Models Can Align Themselves without Finetuning
  PKU, ICLR'24, 2024 [Paper]
• ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding
  WHU, arxiv'24, 2024 [Paper]
• Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models
  Rutgers, arxiv'24, 2024 [Paper]

Type: Benchmark

• Causal ATE Mitigates Unintended Bias in Controlled Text Generation
  IISc, Bangalore, arxiv'23, 2023 [Paper]
• Evaluating Large Language Models on Controlled Generation Tasks
  University of Southern California, EMNLP'23, 2023 [Paper]
• Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  USTC, AAAI'24, 2024 [Paper]
• Controllable Text Generation in the Instruction-Tuning Era
  CMU, arxiv'24, 2024 [Paper]
• FOFO: A Benchmark to Evaluate LLMs’ Format-Following Capability
  Salesforce Research, arxiv'24, 2024 [Paper]
• Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization
  Yale University, NAACL'24, 2024 [Paper]

Type: Survey

• How to Control Sentiment in Text Generation: A Survey of the State-of-the-Art in Sentiment-Control Techniques
  DCU, nan, 2023 [Paper]
• A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models
  BIT, nan, 2023 [Paper]
• A recent survey on controllable text generation: A causal perspective
  Tongji, nan, 2024 [Paper]