[BUG] Unable to query across mutliple rollup indexes when rollup has been modified

[sharebear]

What is the bug?
Back in 2023 we started using using rollup jobs to aggregate some data into year based indeces, so the data ends up in the indexes rollup-data-2023, rollup-data-2024 and rollup-data-2025. When visualising data from these indeces we have an index pattern rollup-data-*.

Towards the end of 2024 we have added a field to the rollup job leading to the following scenario:

• rollup-data-2023 : contains only data without the new field
• rollup-data-2024 : contains some data without the new field and some with
• rollup-data-2025 : contains only data with the new field

When performing a query for the new field against the individual indices we get the following (expected) behavour:

• rollup-data-2023 : illegal_argument_exception : Could not find a rollup job that can answer this query because [missing field myNewField.keyword]
• rollup-data-2024 : SUCCESS
• rollup-data-2025 : SUCCESS

This is all understandable so far, the problem comes when trying to execute the same query against the index pattern that matches all three indices.

GET rollup-api-stats-*/_search
{
  "aggs": {
    "2": {
      "terms": {
        "field": "myNewField.keyword",
        "order": {
          "1": "desc"
        },
        "size": 5
      },
      "aggs": {
        "1": {
          "sum": {
            "field": "numberOfCalls"
          }
        }
      }
    }
  },
  "size": 0
}

When doing this all 3 shards fail.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Could not find a rollup job that can answer this query because [missing field myNewField.keyword]"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "Could not find a rollup job that can answer this query because [missing field myNewField.keyword]"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "Could not find a rollup job that can answer this query because [missing field myNewField.keyword]"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
...

How can one reproduce the bug?
See description for now, I might try and create a smaller reproduction if more explanation is needed.

What is the expected behavior?
I would expect only one shard to fail, allowing us to create visualisations for the index pattern.

What is your host/environment?

• OS: Linux (Aiven)
• Version: 2.17.1

[sharebear]

I found some time to work on a "minimal" reproduction, I might be able to reduce the code further but in this format it also demonstrates the original workflow (although rollups are managed though ISM policies).

Posting the below into the Dev console and running each expression in sequnce, the error on the last query is the one demonstrating the issue.

## Prep some testdata and perform rollups

POST sharej-data-2023-07-25/_doc
{
  "timestamp": "2023-07-25T14:10:43.1",
  "city": "Oslo",
  "numberOfCalls": 1
}

PUT _plugins/_rollup/jobs/sharej-data-2023
{
  "rollup": {
    "source_index": "sharej-data-2023-07-25",
    "target_index": "rollup-sharej-data-2023",
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes"
      }
    },
    "description": "Rollup data for 2023",
    "enabled": true,
    "page_size": 200,
    "delay": 0,
    "continuous": false,
    "dimensions": [
      {
        "date_histogram": {
          "source_field": "timestamp",
          "fixed_interval": "1h",
          "timezone": "Europe/Oslo"
        }
      },
      {
        "terms": {
          "source_field": "city.keyword",
          "target_field": "city.keyword"
        }
      }
    ],
    "metrics": [
      {
        "source_field": "numberOfCalls",
        "metrics": [
          {
            "sum": {}
          },
          {
            "value_count": {}
          }
        ]
      }
    ]
  }
}

POST sharej-data-2024-07-25/_doc
{
  "timestamp": "2024-07-25T14:10:43.1",
  "city": "Amsterdam",
  "numberOfCalls": 1
}

PUT _plugins/_rollup/jobs/sharej-data-2024-07
{
  "rollup": {
    "source_index": "sharej-data-2024-07-25",
    "target_index": "rollup-sharej-data-2024",
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes"
      }
    },
    "description": "Rollup data for 2024",
    "enabled": true,
    "page_size": 200,
    "delay": 0,
    "continuous": false,
    "dimensions": [
      {
        "date_histogram": {
          "source_field": "timestamp",
          "fixed_interval": "1h",
          "timezone": "Europe/Oslo"
        }
      },
      {
        "terms": {
          "source_field": "city.keyword",
          "target_field": "city.keyword"
        }
      }
    ],
    "metrics": [
      {
        "source_field": "numberOfCalls",
        "metrics": [
          {
            "sum": {}
          },
          {
            "value_count": {}
          }
        ]
      }
    ]
  }
}

POST sharej-data-2024-10-25/_doc
{
  "timestamp": "2024-10-25T14:10:43.1",
  "city": "Oslo",
  "colour": "Blue",
  "numberOfCalls": 1
}

PUT _plugins/_rollup/jobs/sharej-data-2024-10
{
  "rollup": {
    "source_index": "sharej-data-2024-10-25",
    "target_index": "rollup-sharej-data-2024",
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes"
      }
    },
    "description": "Rollup data for 2024",
    "enabled": true,
    "page_size": 200,
    "delay": 0,
    "continuous": false,
    "dimensions": [
      {
        "date_histogram": {
          "source_field": "timestamp",
          "fixed_interval": "1h",
          "timezone": "Europe/Oslo"
        }
      },
      {
        "terms": {
          "source_field": "city.keyword",
          "target_field": "city.keyword"
        }
      },
      {
        "terms": {
          "source_field": "colour.keyword",
          "target_field": "colour.keyword"
        }
      }
    ],
    "metrics": [
      {
        "source_field": "numberOfCalls",
        "metrics": [
          {
            "sum": {}
          },
          {
            "value_count": {}
          }
        ]
      }
    ]
  }
}

POST sharej-data-2025-01-25/_doc
{
  "timestamp": "2025-01-25T14:10:43.1",
  "city": "Helsinki",
  "colour": "Orange",
  "numberOfCalls": 1
}

PUT _plugins/_rollup/jobs/sharej-data-2025
{
  "rollup": {
    "source_index": "sharej-data-2025-01-25",
    "target_index": "rollup-sharej-data-2025",
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes"
      }
    },
    "description": "Rollup data for 2025",
    "enabled": true,
    "page_size": 200,
    "delay": 0,
    "continuous": false,
    "dimensions": [
      {
        "date_histogram": {
          "source_field": "timestamp",
          "fixed_interval": "1h",
          "timezone": "Europe/Oslo"
        }
      },
      {
        "terms": {
          "source_field": "city.keyword",
          "target_field": "city.keyword"
        }
      },
      {
        "terms": {
          "source_field": "colour.keyword",
          "target_field": "colour.keyword"
        }
      }
    ],
    "metrics": [
      {
        "source_field": "numberOfCalls",
        "metrics": [
          {
            "sum": {}
          },
          {
            "value_count": {}
          }
        ]
      }
    ]
  }
}

## Wait for all the rollup jobs to finish before continuing


## Example searches against the source indexes
GET sharej-data-*/_search
{
  "aggs": {
    "2": {
      "terms": {
        "field": "city.keyword",
        "order": {
          "1": "desc"
        },
        "size": 5
      },
      "aggs": {
        "1": {
          "sum": {
            "field": "numberOfCalls"
          }
        }
      }
    }
  },
  "size": 0
}

## Still works across all source indexes even if the field doesn't exist in any documents for the index
GET sharej-data-*/_search
{
  "aggs": {
    "2": {
      "terms": {
        "field": "colour.keyword",
        "order": {
          "1": "desc"
        },
        "size": 5
      },
      "aggs": {
        "1": {
          "sum": {
            "field": "numberOfCalls"
          }
        }
      }
    }
  },
  "size": 0
}

## Searching the rollup index also fine for the field in all documents/rollups
GET rollup-sharej-data-*/_search
{
  "aggs": {
    "2": {
      "terms": {
        "field": "city.keyword",
        "order": {
          "1": "desc"
        },
        "size": 5
      },
      "aggs": {
        "1": {
          "sum": {
            "field": "numberOfCalls"
          }
        }
      }
    }
  },
  "size": 0
}

## Failure of all three shards when trying the same for the field that was added in 2024

GET rollup-sharej-data-*/_search
{
  "aggs": {
    "2": {
      "terms": {
        "field": "colour.keyword",
        "order": {
          "1": "desc"
        },
        "size": 5
      },
      "aggs": {
        "1": {
          "sum": {
            "field": "numberOfCalls"
          }
        }
      }
    }
  },
  "size": 0
}