Improve the upgrade-python script

[FlorentClarret]

What does this PR do?

Improve the upgrade-python script to:

• Update the classifiers in the pyproject.toml files
• Update the classifiers in the setup.py files (if any)
• Update the CI files
• Update the ddev pyproject.toml file to bump the versions for ruff and black
• Update the template files for new integrations
• Warn the user if there's an override that the script can't handle automatically (I tried, I don't manage to have the right format for the output file, I don't know why. Feel free to let me know if you know why, but I did not want to waste too much time on this)
• Remind the user to update the documentation. I did not want to update it with the script as our documentation can be modified pretty often and this would make the script not working as expected. I prefer having a script warning that the user has something to do manually (very minor stuff imo) rather than letting them believe that everything was done automagically.

Motivation

• Simplify the bump process for the integration repos
• https://datadoghq.atlassian.net/browse/AITS-283

Additional Notes

• I used this script to open Bump the Python version from py3.9 to py3.11 #15997
• Needs Use a dummy repository to run the upgrade_python tests #16019 to be merged first

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Changelog entries must be created for modifications to shipped code
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.

[codecov]

Codecov Report

Merging #16000 (131174f) into master (5543e0b) will increase coverage by 0.24%.
Report is 20 commits behind head on master.
The diff coverage is 100.00%.

Flag	Coverage Δ
active_directory	100.00% <ø> (+17.64%)	⬆️
activemq	52.87% <ø> (ø)
activemq_xml	82.31% <ø> (ø)
ambari	85.75% <ø> (ø)
apache	95.08% <ø> (ø)
arangodb	98.23% <ø> (ø)
argocd	87.81% <ø> (ø)
aspdotnet	100.00% <ø> (ø)
avi_vantage	91.35% <ø> (ø)
azure_iot_edge	82.08% <ø> (ø)
boundary	100.00% <ø> (ø)
btrfs	82.91% <ø> (ø)
cacti	87.90% <ø> (ø)
calico	84.61% <ø> (ø)
cassandra	65.51% <ø> (ø)
cert_manager	77.41% <ø> (ø)
cilium	77.63% <ø> (+0.87%)	⬆️
cisco_aci	95.27% <ø> (ø)
citrix_hypervisor	87.50% <ø> (ø)
cloud_foundry_api	96.35% <ø> (+0.12%)	⬆️
cloudera	99.49% <ø> (ø)
cockroachdb	91.52% <ø> (ø)
confluent_platform	81.96% <ø> (ø)
coredns	94.57% <ø> (ø)
crio	89.79% <ø> (ø)
datadog_checks_base	89.67% <ø> (+0.90%)	⬆️
datadog_checks_dev	82.49% <ø> (+0.07%)	⬆️
datadog_checks_downloader	81.65% <ø> (ø)
datadog_cluster_agent	90.19% <ø> (ø)
dcgm	97.40% <ø> (ø)
ddev	99.48% <100.00%> (+<0.01%)	⬆️
directory	95.87% <ø> (+0.43%)	⬆️
disk	91.36% <ø> (ø)
dns_check	93.90% <ø> (ø)
dotnetclr	91.39% <ø> (+12.90%)	⬆️
druid	98.47% <ø> (ø)
ecs_fargate	82.91% <ø> (ø)
eks_fargate	94.05% <ø> (ø)
envoy	95.09% <ø> (+0.41%)	⬆️
etcd	95.56% <ø> (ø)
exchange_server	96.85% <ø> (+11.81%)	⬆️
external_dns	89.28% <ø> (ø)
fluentd	94.77% <ø> (ø)
foundationdb	78.50% <ø> (ø)
gitlab_runner	91.94% <ø> (ø)
gunicorn	92.10% <ø> (ø)
hazelcast	92.39% <ø> (ø)
hdfs_namenode	86.72% <ø> (ø)
hive	51.42% <ø> (ø)
hivemq	61.90% <ø> (ø)
http_check	95.51% <ø> (+2.05%)	⬆️
hudi	73.91% <ø> (ø)
ibm_ace	91.89% <ø> (ø)
ibm_db2	95.35% <ø> (ø)
ibm_i	81.91% <ø> (ø)
ibm_mq	51.98% <ø> (ø)
ignite	46.66% <ø> (ø)
iis	95.05% <ø> (ø)
impala	97.97% <ø> (ø)
istio	77.74% <ø> (+0.52%)	⬆️
jboss_wildfly	47.36% <ø> (ø)
kafka	63.63% <ø> (ø)
kong	87.56% <ø> (ø)
kube_apiserver_metrics	98.14% <ø> (ø)
kube_controller_manager	96.07% <ø> (ø)
kube_dns	95.97% <ø> (ø)
kube_metrics_server	94.87% <ø> (ø)
kube_proxy	96.80% <ø> (ø)
kube_scheduler	97.63% <ø> (ø)
kubelet	90.97% <ø> (ø)
kubernetes_state	89.18% <ø> (ø)
kyototycoon	85.96% <ø> (ø)
lighttpd	83.64% <ø> (ø)
linkerd	85.14% <ø> (+1.14%)	⬆️
linux_proc_extras	96.22% <ø> (ø)
mapr	82.70% <ø> (ø)
mapreduce	81.81% <ø> (+0.46%)	⬆️
marathon	83.43% <ø> (ø)
mcache	93.50% <ø> (ø)
mesos_master	89.75% <ø> (ø)
mesos_slave	93.63% <ø> (ø)
nagios	89.01% <ø> (ø)
network	93.64% <ø> (+1.08%)	⬆️
nfsstat	95.20% <ø> (ø)
nginx	95.24% <ø> (+0.54%)	⬆️
nginx_ingress_controller	98.36% <ø> (ø)
openldap	96.79% <ø> (ø)
openmetrics	98.08% <ø> (ø)
openstack	51.45% <ø> (ø)
openstack_controller	91.12% <ø> (ø)
oracle	88.63% <ø> (ø)
pdh_check	97.82% <ø> (ø)
pgbouncer	91.33% <ø> (ø)
php_fpm	90.53% <ø> (+1.12%)	⬆️
postfix	88.04% <ø> (ø)
powerdns_recursor	96.65% <ø> (ø)
presto	59.09% <ø> (ø)
process	85.42% <ø> (-0.72%)	⬇️
prometheus	94.17% <ø> (ø)
proxysql	98.97% <ø> (ø)
pulsar	100.00% <ø> (ø)
ray	96.45% <ø> (ø)
redisdb	87.78% <ø> (ø)
rethinkdb	97.93% <ø> (ø)
riak	99.22% <ø> (ø)
riakcs	93.61% <ø> (ø)
silk	93.82% <ø> (ø)
singlestore	90.81% <ø> (ø)
snowflake	96.66% <ø> (+0.04%)	⬆️
solr	56.25% <ø> (ø)
spark	93.91% <ø> (ø)
squid	100.00% <ø> (ø)
statsd	87.36% <ø> (+1.05%)	⬆️
strimzi	89.70% <ø> (ø)
supervisord	90.14% <ø> (ø)
system_core	92.66% <ø> (ø)
system_swap	98.30% <ø> (ø)
tcp_check	92.92% <ø> (ø)
temporal	100.00% <ø> (ø)
teradata	94.06% <ø> (ø)
tls	92.18% <ø> (+0.82%)	⬆️
tokumx	58.40% <ø> (ø)
tomcat	61.29% <ø> (ø)
torchserve	97.33% <ø> (ø)
traffic_server	96.13% <ø> (ø)
twemproxy	79.45% <ø> (ø)
twistlock	79.62% <ø> (ø)
varnish	84.39% <ø> (+0.26%)	⬆️
voltdb	96.85% <ø> (ø)
weaviate	76.27% <ø> (ø)
weblogic	71.73% <ø> (ø)
windows_performance_counters	98.36% <ø> (ø)
windows_service	98.00% <ø> (ø)
wmi_check	92.91% <ø> (ø)
yarn	89.59% <ø> (ø)
zk	82.62% <ø> (+1.33%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[github-actions]

Test Results

    2 files      2 suites   4m 42s ⏱️
428 tests 423 ✔️   5 💤 0 ❌
856 runs  827 ✔️ 29 💤 0 ❌

Results for commit 131174f.

♻️ This comment has been updated with latest results.

[iliakur]

Just a couple suggestions, nothing major. Will approve once we settle on something for the hatch.toml updates

[iliakur]

Mostly good, just one simplification proposal left to consider.