Consider changing API to not use regexp pattern

[potiuk]

Body

Our public REST API specification has the possibility of using regexp pattern (dag_id_pattern). This is potentially dangerous as free regexp patterns might have potentially some problems (like infinite recursion or non-linear performance of some patterns) and regexp parsing libraries might have unknown bugs, We should consider making a backwards-incompatible release of the API with the pattern feature removed.

This wouls be a breaking change, and breaking the promise we made to keep our Stable REST API the most stable part of the Public Airflow Interface that we promised our users to rely on, so should carefully consider if removing it is worth breaking the promise. It would also require strong justification and unsolvable security/performance issue to make such change without bumping the REST API version to new MAJOR version and it would disconnect Airflow MAJOR version from the API MAJOR version, so likely the only good time we can do it, is when we release Airflow 3.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

cc: @bolkedebruin

[pierrejeambrun]

Nice, thanks for opening :)

[Bowrna]

@kaxil @potiuk Can i take a look into this?

[kaxil]

@Bowrna This is now assigned to you :)

[Bowrna]

@kaxil I will raise my PR before this weekends

[pierrejeambrun]

I just saw the Pull request associated to this and it makes me wonder. The actual dag_id_pattern is not used to do regexp matching, but db ilike macthing which does not leverage the standard regexp engine matcher but its own thing which is safer.

For instance for postgres some informations are available here https://www.postgresql.org/docs/current/functions-matching.html#FUNCTIONS-LIKE, and as mentioned in the note just above seems safer from a security point of view. In opposition the paragraph right after highlights the similar to regular expression syntax, which is vulnerable.

I am second guessing if we need to do something here.

[potiuk]

You are very right @pierrejeambrun ! We can close it.