Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce TravisCI Test Runtime #2541

Merged
merged 17 commits into from
Nov 21, 2018
Merged

Reduce TravisCI Test Runtime #2541

merged 17 commits into from
Nov 21, 2018

Conversation

dlstadther
Copy link
Collaborator

@dlstadther dlstadther commented Sep 28, 2018 •
edited
Loading

Travis tests are taking way too long (and we're being too greedy with their free resources). Trying to reduce the overhead here.

Description

Brute force nose attributes of (apache, aws, contrib) excluded from core (previously nonhdfs). (apache, aws) will be added for most recent Python 2.X and 3.X versions.

Motivation and Context

CI needs to be quicker - seeing what can be done to segment differently.

Have you tested this? If so, how?

TravisCI :)

@dlstadther
Copy link
Collaborator Author

py27-core took ~5m to complete. This suggests that a large portion of the test runtime is the contrib tests. I'm going to segment these even further (maybe into apache, aws, and contrib) and checkout runtimes.

Generally, I propose we only push most commonly used contrib/ functionality through TravisCI. (we already don't run cdh, minicluster, and gcloud anymore).

@dlstadther dlstadther requested a review from honnix September 28, 2018 17:25
@dlstadther
Copy link
Collaborator Author

py-27-aws only adds 2:30m

@dlstadther
Copy link
Collaborator Author

py27-apache <2m

@dlstadther dlstadther changed the title [WIP] TravisCI Test Refactor [WIP] TravisCI Test Runtime Reduction Sep 28, 2018
@dlstadther dlstadther changed the title [WIP] TravisCI Test Runtime Reduction [WIP] Reduce TravisCI Test Runtime Sep 28, 2018
@dlstadther
Copy link
Collaborator Author

dlstadther commented Sep 28, 2018 •
edited
Loading

Main changes here:

  • python 3.5
    • no contrib/ tests
  • python 2.7/3.7
    • [specific contrib (non-apache/aws]
      • Cascading
      • Docker
      • External Program
      • Kubernetes
      • LSF
      • Mongo
      • Opener
      • Redis
      • Salesforce
      • Scalding
      • SGE
      • SQLA
      • SSH

Also, the following nose attr haven't been running through TravisCI for some time now:

  • gcloud
  • postgres
  • minicluster/cdh

Tarrasch
Tarrasch approved these changes Sep 29, 2018
View reviewed changes
Copy link
Contributor

@Tarrasch Tarrasch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A billion thanks!

@dlstadther
Merge branch 'upstream-master' into feature/refactor-tests
aed07c8 
* upstream-master:
  Add Big Data's name to the list of companies who use Luigi (spotify#2545)
  Fix S3Client.remove - add max batch size (spotify#2529)
  Fix S3Client's _path_to_bucket_and_key to support keys with question marks (spotify#2534)
@dlstadther
Merge branch 'upstream-master' into feature/refactor-tests
cf335c7 
* upstream-master:
  Make Worker parameter task_process_context an OptionalParameter (spotify#2468) (spotify#2574)
  Version 2.8.0
  Implement configurable CORS.
  Add HdfsFlagTarget (spotify#2559)
  Fix HdfsAtomicWriteDirPipe.close() when using snakebite and the file do not exist. (spotify#2549)
  Small fix to logging in contrib/ecs.py (spotify#2556)
  [ImgBot] optimizes images (spotify#2555)
  Add CopyToTable task for MySQL (spotify#2553)
  Make capture_output non-positional in ExternalProgramTask (spotify#2547)
  Add Movio to list of Luigi users (spotify#2551)
  Interpolate environment variables in .cfg config files (spotify#2527)
  Fix ReadTheDocs build (spotify#2546)
@dlstadther
Test if previous moto version fixes failures
6163100
@dlstadther
Revert moto version downgrade
9663df2
@dlstadther
Merge branch 'upstream-master' into feature/refactor-tests
20493b7 
* upstream-master:
  Downgrade moto from 1.3.7 to 1.3.6 (spotify#2586)
@dlstadther dlstadther changed the title [WIP] Reduce TravisCI Test Runtime Reduce TravisCI Test Runtime Nov 19, 2018
@dlstadther
Copy link
Collaborator Author

@Tarrasch @honnix @ulzha
Looking to get one more 👍 before merging.

The only contribution negative is that new TestCases in /tests/contrib/ will need to add a Nose Attr label, otherwise will be included in core

@honnix
Copy link
Member

honnix commented Nov 19, 2018

Looks awesome. Good job!

Just one minor comment. apache refers to Apache Hadoop Ecosystem, right? Wonder whether this name would be too generic. On the other hand, I think people will get it.

@dlstadther
Copy link
Collaborator Author

Correct. apache -> apache hadoop ecosystem

Thanks for taking a look!

@Tarrasch Tarrasch merged commit 70f5380 into spotify:master Nov 21, 2018
@Tarrasch
Copy link
Contributor

Awesome! 💃 🎉 🎈 💯 🏆

@dlstadther dlstadther deleted the feature/refactor-tests branch November 21, 2018 19:47
@dlstadther dlstadther mentioned this pull request Jun 8, 2020
Merged
@honnix honnix mentioned this pull request Jun 8, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Tarrasch Tarrasch Tarrasch approved these changes

@honnix honnix honnix approved these changes

@ulzha ulzha Awaiting requested review from ulzha

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dlstadther @honnix @Tarrasch