feat: add to_string method to SparkLikeExprDateTimeNamespace

[Dhanunjaya-Elluri]

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Related issue [Enh]: Spark Expr missing methods #1714
• Closes #<issue number>

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below

[MarcoGorelli]

what does this do?

[Dhanunjaya-Elluri]

I faced the date migration issue from spark 2.0 to 3.0 (basically to match the behaviour datetime parsing of spark versions <3). I refered this issue here https://stackoverflow.com/questions/62602720/string-to-date-migration-from-spark-2-0-to-3-0-gives-fail-to-recognize-eee-mmm

[EdAbati]

Instead of setting this, I suggest using a valid pattern for Spark 3.0: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
Otherwise we need to ask users to also set this config

When we remove the config we get the error:

pyspark.errors.exceptions.captured.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.DATETIME_PATTERN_RECOGNITION] You may get a different result due to the upgrading to Spark >= 3.0:
    Fail to recognize 'YYYY' pattern in the DateTimeFormatter. 1) You can set "spark.sql.legacy.timeParserPolicy" to "LEGACY" to restore the behavior before Spark 3.0. 2) You can form a valid datetime pattern with the guide from 'https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html'.

we need to substitute YYYY to yyyy (as we do in [strptime_to_pyspark_format](https://github.com/narwhals-dev/narwhals/blob/main/narwhals/_spark_like/expr_str.py#L136)?)

[MarcoGorelli]

thanks @Dhanunjaya-Elluri! @EdAbati , does this PR look good to you?

[EdAbati]

Suggested change

	year = F.date_format(_input, "YYYY")
	year = F.date_format(_input, "yyyy")

[EdAbati]

Suggested change

	year = F.date_format(_input, "YYYY")
	year = F.date_format(_input, "yyyy")

[EdAbati]

Thanks @Dhanunjaya-Elluri for this PR! 🙏

Unfortunately when I run these tests on my machine (I am not in UTC timezone), I get all these errors:

FAILED tests/expr_and_series/dt/to_string_test.py::test_dt_to_string_iso_local_date_expr[pyspark-data0-2020-01-09] - AssertionError: Mismatch at index 0: 2020-01-08 23:00:00 != 2020-01-09 00:00:00
FAILED tests/expr_and_series/dt/to_string_test.py::test_dt_to_string_expr[pyspark-%Y/%m/%d %H:%M:%S] - AssertionError: Mismatch at index 0: 2020/01/02 01:04:14 != 2020/01/02 02:04:14
FAILED tests/expr_and_series/dt/to_string_test.py::test_dt_to_string_expr[pyspark-%Y-%m-%d %H:%M:%S] - AssertionError: Mismatch at index 0: 2020-01-02 01:04:14 != 2020-01-02 02:04:14
FAILED tests/expr_and_series/dt/to_string_test.py::test_dt_to_string_iso_local_datetime_expr[pyspark-data2-2020-01-09T12:34:56.123456] - AssertionError: Mismatch at index 0: 2020-01-09 11:34:56.123456 != 2020-01-09 12:34:56.123456
FAILED tests/expr_and_series/dt/to_string_test.py::test_dt_to_string_iso_local_datetime_expr[pyspark-data1-2020-01-09T12:34:56.000123] - AssertionError: Mismatch at index 0: 2020-01-09 11:34:56.000123 != 2020-01-09 12:34:56.000123
FAILED tests/expr_and_series/dt/to_string_test.py::test_dt_to_string_iso_local_datetime_expr[pyspark-data0-2020-01-09T12:34:56.000000] - AssertionError: Mismatch at index 0: 2020-01-09 11:34:56 != 2020-01-09 12:34:56
FAILED tests/expr_and_series/dt/datetime_attributes_test.py::test_datetime_attributes[pyspark-hour-expected4] - AssertionError: Mismatch at index 0: 1 != 2

I didn't have time to look more into this, I leave this resource that could be useful https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html#timestamp-with-time-zone-semantics

[MarcoGorelli]

Unfortunately when I run these tests on my machine (I am not in UTC timezone), I get all these errors:

Just to confirm, the rest of the tests all pass on your machine?

[Dhanunjaya-Elluri]

Unfortunately when I run these tests on my machine (I am not in UTC timezone), I get all these errors:

Just to confirm, the rest of the tests all pass on your machine?

Apparently, this line is already set to UTC, but it doesn't work somehow.

narwhals/tests/conftest.py

Line 161 in 0f38b77

.config("spark.sql.session.timeZone", "UTC")

I did a work around to set the timezone:

export TZ=UTC

The simplest setup is to use os.environ["TZ"] = "UTC". I'll push these changes now