Retry datasources on error

[stefanoboriero]

Not all data_source elements were retrying on rate limiting error, I reproduced the retrying structure already present in some of them and extended the time between retries due to rate limiting to 30s as suggested by PagerDuty https://developer.pagerduty.com/docs/rest-api-v2/rate-limiting/#what-are-possible-workarounds-to-the-events-api-rate-limit

Test results, I had to run tests for priorities with a different key because the endpoint was not available for my account

TF_ACC=1 go test -run "TestAccDataSource" ./pagerduty -v -timeout 120m
=== RUN   TestAccDataSourcePagerDutyBusinessService_Basic
--- PASS: TestAccDataSourcePagerDutyBusinessService_Basic (6.80s)
=== RUN   TestAccDataSourcePagerDutyEscalationPolicy_Basic
--- PASS: TestAccDataSourcePagerDutyEscalationPolicy_Basic (13.46s)
=== RUN   TestAccDataSourcePagerDutyExtensionSchema_Basic
--- PASS: TestAccDataSourcePagerDutyExtensionSchema_Basic (5.11s)
=== RUN   TestAccDataSourcePagerDutyPriority_Basic
    TestAccDataSourcePagerDutyPriority_Basic: testing.go:654: Step 0 error: errors during refresh:
        
        Error: GET API call to https://api.pagerduty.com/priorities failed: 404 Not Found
        
          on /var/folders/mb/g8mm073d67n7gngsx7_d0rh8jdx2kh/T/tf-test747950839/main.tf line 2:
          (source code not available)
        
        
--- FAIL: TestAccDataSourcePagerDutyPriority_Basic (1.82s)
=== RUN   TestAccDataSourcePagerDutyPriority_P2
    TestAccDataSourcePagerDutyPriority_P2: testing.go:654: Step 0 error: errors during refresh:
        
        Error: GET API call to https://api.pagerduty.com/priorities failed: 404 Not Found
        
          on /var/folders/mb/g8mm073d67n7gngsx7_d0rh8jdx2kh/T/tf-test088133098/main.tf line 2:
          (source code not available)
        
        
--- FAIL: TestAccDataSourcePagerDutyPriority_P2 (0.73s)
=== RUN   TestAccDataSourcePagerDutyRuleset_Basic
--- PASS: TestAccDataSourcePagerDutyRuleset_Basic (6.79s)
=== RUN   TestAccDataSourcePagerDutySchedule_Basic
--- PASS: TestAccDataSourcePagerDutySchedule_Basic (12.41s)
=== RUN   TestAccDataSourcePagerDutyService_Basic
--- PASS: TestAccDataSourcePagerDutyService_Basic (16.94s)
=== RUN   TestAccDataSourcePagerDutyTeam_Basic
--- PASS: TestAccDataSourcePagerDutyTeam_Basic (8.41s)
=== RUN   TestAccDataSourcePagerDutyUser_Basic
--- PASS: TestAccDataSourcePagerDutyUser_Basic (9.76s)
=== RUN   TestAccDataSourcePagerDutyVendor_Basic
--- PASS: TestAccDataSourcePagerDutyVendor_Basic (5.58s)
=== RUN   TestAccDataSourcePagerDutyVendor_ExactMatch
--- PASS: TestAccDataSourcePagerDutyVendor_ExactMatch (5.77s)
=== RUN   TestAccDataSourcePagerDutyVendor_SpecialChars
--- PASS: TestAccDataSourcePagerDutyVendor_SpecialChars (5.56s)

TF_ACC=1 go test -run "TestAccDataSourcePagerDutyPriority" ./pagerduty -v -timeout 120m
=== RUN   TestAccDataSourcePagerDutyPriority_Basic
--- PASS: TestAccDataSourcePagerDutyPriority_Basic (5.92s)
=== RUN   TestAccDataSourcePagerDutyPriority_P2
--- PASS: TestAccDataSourcePagerDutyPriority_P2 (4.65s)
PASS
ok      github.com/terraform-providers/terraform-provider-pagerduty/pagerduty   11.155s

[stmcallister]

Thanks for adding retry to the rest of the data sources and for making the sleep time uniform. I think 30 seconds is suggested to be super safe, but I can't argue with documentation. ;)

Looks good!

[stefanoboriero]

Sweet, thanks for approving and merging this!
I believe they suggest such conservative value because rate limit for the REST API is shared across the whole account rather than per API key, so consuming it will affect also other people