Flag Targeting Using Stratification

[kentakozuka]

Objective

We aim to create a flag targeting feature that supports accurate and reliable A/B testing by utilizing stratification to group users into homogeneous segments. This feature leverages the following two types of data for flag assignment and evaluation:

1. User Flag Assignment Information (Evaluation Events)
   Data recorded during flag evaluation:

   • Details:
     Information on when, for which user, which flag was assigned, and in what variation.
   • Purpose:
     To track flag assignment status and enable analysis based on accurate data.

2. User Activity (Goal Events)
   Data recorded when users perform specific actions:

   • Details:
     Information on when, which user performed which activity (e.g., purchases, clicks, page views).
   • Purpose:
     To measure key metrics (e.g., conversions) for A/B testing goals and assess flag effectiveness.

Below is an example of the proposed targeting interface.

To implement this system, it is essential to efficiently integrate evaluation data (Evaluation Events) and activity data (Goal Events) while performing stratified flag evaluations. Two approaches are under consideration:

Comparison of Approaches

Approach 1: Server-Side Evaluation

A method where flag evaluations are centrally managed on the server.

Mechanism

• The client sends a request containing user attributes and contextual information to the server.
• The server integrates real-time user attributes, activity data, and historical flag evaluation data.
• The server applies stratification algorithms to determine the appropriate variation and returns it to the client.

Advantages

• Efficient Data Management:
  Enables centralized management of large-scale data (e.g., activity data for 100 million users).
• Consistency:
  Centralized evaluation logic ensures consistent results.
• Dynamic Data Integration:
  Allows for flexible evaluation by integrating real-time attributes and activity data.
• Scalability:
  Handles high traffic with distributed architecture and caching.

Disadvantages

• Network Dependency:
  Evaluation speed may be affected by network latency.
• Cost:
  Operating and scaling server infrastructure increases costs.
• Security Risks:
  Data transmitted from the client may pose security risks.

Approach 2: Client-Side Evaluation

A method where flag evaluations are conducted using a client SDK.

Mechanism

• The client SDK loads user attributes, flag definitions, and stratification rules.
• The SDK applies stratification logic locally to evaluate flag variations.

Advantages

• Low Latency:
  Immediate evaluation without network communication.
• Offline Capability:
  Enables flag evaluation even without network connectivity.
• Reduced Server Load:
  Offloads processing from the server.

Disadvantages

• High Memory Usage:
  Storing large-scale data (e.g., 1.6GB of user information) on the client is impractical.
• Data Synchronization Challenges:
  Keeping activity data and flag definitions up-to-date is difficult.
• Security Risks:
  Data stored on the client is prone to tampering and leaks.

Comparison Table

Aspect	Server-Side Evaluation	Client-Side Evaluation
Latency	Requires network communication (potential delay)	Immediate evaluation (low latency)
Data Management	Centralized management (suitable for large-scale data)	High memory usage required
Scalability	Distributed architecture for high traffic	Increased resource burden on the client
Offline Capability	Not supported	Supported
Consistency	Centralized management ensures consistency	Data synchronization is difficult
Security	Secure data management on the server	Higher risk of tampering and leaks

Current Challenges

Data Volume Constraints

For client-side evaluation SDKs, implementing stratification requires storing massive amounts of user ID and segment data in memory. For example:

• Example:
  • Storing 100 million user UUIDs (16 bytes each) would require approximately 1.6GB of data.
  • Additional activity data and historical flag evaluation data would further increase memory requirements.

User Activity Data Constraints

Activity data, often aggregated via daily batch processing or on the server, cannot be easily provided to clients in real-time.

Current SDK Design

Server-side and client-side evaluation SDKs have distinct roles. For example:

• Client-side evaluation SDKs (e.g., Go, Node.js) are often used in scenarios requiring immediacy and offline functionality. Currently, there is no clear method to introduce stratification logic in these SDKs.

Implementations in Other Products

• https://docs.statsig.com/experiments-plus/stratified-sampling/