SCC: adding datasources for automated WP connection and Profile attachments #5179

tyao117 · 2024-03-06T21:59:47Z

Updates:

expose a new datasource ibm_scc_control_libraries that returns a list of control_libraries available to an instance.
- Users can now use the datasource to help create a profile resource given from the datasource.
expose a new datasource ibm_scc_profiles that returns a list of profiles available to an instance
- Users can now use the datasource to help create a profile_attachment resource in relation to the data given from the datasource
expose a new datasource ibm_scc_provider_types that returns a list of provider available to an instance
- Users can use the datasource to find the ibm_scc_provider_type.id needed to create a resource of scc_provider_type_instance.
fixes the import of documentation of scc

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #5186
Closes #5185
Closes #5208

Output from acceptance testing:

$ make testacc TEST=./ibm/service/scc       
=== RUN   TestAccIbmSccControlLibrariesDataSourceBasic
--- PASS: TestAccIbmSccControlLibrariesDataSourceBasic (51.25s)
=== RUN   TestAccIbmSccControlLibrariesDataSourceAllArgs
--- PASS: TestAccIbmSccControlLibrariesDataSourceAllArgs (41.44s)
=== RUN   TestAccIbmSccControlLibraryDataSourceBasic
--- PASS: TestAccIbmSccControlLibraryDataSourceBasic (56.03s)
=== RUN   TestAccIbmSccControlLibraryDataSourceAllArgs
--- PASS: TestAccIbmSccControlLibraryDataSourceAllArgs (55.85s)
=== RUN   TestAccIbmSccInstanceSettingsDataSourceBasic
--- PASS: TestAccIbmSccInstanceSettingsDataSourceBasic (42.82s)
=== RUN   TestAccIbmSccLatestReportsDataSourceBasic
--- PASS: TestAccIbmSccLatestReportsDataSourceBasic (44.97s)
=== RUN   TestAccIbmSccProfileAttachmentDataSourceBasic
--- PASS: TestAccIbmSccProfileAttachmentDataSourceBasic (66.66s)
=== RUN   TestAccIbmSccProfileAttachmentDataSourceAllArgs
--- PASS: TestAccIbmSccProfileAttachmentDataSourceAllArgs (64.63s)
=== RUN   TestAccIbmSccProfileDataSourceBasic
--- PASS: TestAccIbmSccProfileDataSourceBasic (65.19s)
=== RUN   TestAccIbmSccProfileDataSourceAllArgs
--- PASS: TestAccIbmSccProfileDataSourceAllArgs (58.10s)
=== RUN   TestAccIbmSccProfilesDataSourceBasic
--- PASS: TestAccIbmSccProfilesDataSourceBasic (44.98s)
=== RUN   TestAccIbmSccProfilesDataSourceAllArgs
--- PASS: TestAccIbmSccProfilesDataSourceAllArgs (48.66s)
=== RUN   TestAccIbmSccProviderTypeCollectionDataSourceBasic
--- PASS: TestAccIbmSccProviderTypeCollectionDataSourceBasic (40.12s)
=== RUN   TestAccIbmSccProviderTypeInstanceDataSourceBasic
--- PASS: TestAccIbmSccProviderTypeInstanceDataSourceBasic (51.32s)
=== RUN   TestAccIbmSccProviderTypeInstanceDataSourceAllArgs
--- PASS: TestAccIbmSccProviderTypeInstanceDataSourceAllArgs (51.91s)
=== RUN   TestAccIbmSccProviderTypeDataSourceBasic
--- PASS: TestAccIbmSccProviderTypeDataSourceBasic (39.37s)
=== RUN   TestAccIbmSccProviderTypesDataSourceBasic
--- PASS: TestAccIbmSccProviderTypesDataSourceBasic (39.61s)
=== RUN   TestAccIbmSccReportControlsDataSourceBasic
--- PASS: TestAccIbmSccReportControlsDataSourceBasic (42.73s)
=== RUN   TestAccIbmSccReportEvaluationsDataSourceBasic
--- PASS: TestAccIbmSccReportEvaluationsDataSourceBasic (42.10s)
=== RUN   TestAccIbmSccReportResourcesDataSourceBasic
--- PASS: TestAccIbmSccReportResourcesDataSourceBasic (39.08s)
=== RUN   TestAccIbmSccReportRuleDataSourceBasic
--- PASS: TestAccIbmSccReportRuleDataSourceBasic (39.16s)
=== RUN   TestAccIbmSccReportSummaryDataSourceBasic
--- PASS: TestAccIbmSccReportSummaryDataSourceBasic (44.01s)
=== RUN   TestAccIbmSccReportTagsDataSourceBasic
--- PASS: TestAccIbmSccReportTagsDataSourceBasic (38.63s)
=== RUN   TestAccIbmSccReportDataSourceBasic
--- PASS: TestAccIbmSccReportDataSourceBasic (38.64s)
=== RUN   TestAccIbmSccReportViolationDriftDataSourceBasic
--- PASS: TestAccIbmSccReportViolationDriftDataSourceBasic (39.60s)
=== RUN   TestAccIbmSccRuleDataSourceBasic
--- PASS: TestAccIbmSccRuleDataSourceBasic (56.10s)
=== RUN   TestAccIbmSccRuleDataSourceAllArgs
--- PASS: TestAccIbmSccRuleDataSourceAllArgs (55.48s)
=== RUN   TestAccIbmSccControlLibraryBasic
--- PASS: TestAccIbmSccControlLibraryBasic (87.26s)
=== RUN   TestAccIbmSccControlLibraryAllArgs
--- PASS: TestAccIbmSccControlLibraryAllArgs (105.08s)
=== RUN   TestAccIbmSccInstanceSettingsBasic
--- PASS: TestAccIbmSccInstanceSettingsBasic (56.88s)
=== RUN   TestAccIbmSccInstanceSettingsAllArgs
--- PASS: TestAccIbmSccInstanceSettingsAllArgs (111.14s)
=== RUN   TestAccIbmSccProfileAttachmentBasic
--- PASS: TestAccIbmSccProfileAttachmentBasic (63.77s)
=== RUN   TestAccIbmSccProfileAttachmentAllArgs
--- PASS: TestAccIbmSccProfileAttachmentAllArgs (104.06s)
=== RUN   TestAccIbmSccProfileBasic
--- PASS: TestAccIbmSccProfileBasic (92.74s)
=== RUN   TestAccIbmSccProfileAllArgs
--- PASS: TestAccIbmSccProfileAllArgs (97.96s)
=== RUN   TestAccIbmSccProviderTypeInstanceBasic
--- PASS: TestAccIbmSccProviderTypeInstanceBasic (89.87s)
=== RUN   TestAccIbmSccProviderTypeInstanceAllArgs
--- PASS: TestAccIbmSccProviderTypeInstanceAllArgs (98.29s)
=== RUN   TestAccIbmSccRuleBasic
--- PASS: TestAccIbmSccRuleBasic (104.38s)
=== RUN   TestAccIbmSccRuleAllArgs
--- PASS: TestAccIbmSccRuleAllArgs (99.30s)
PASS
ok      github.com/IBM-Cloud/terraform-provider-ibm/ibm/service/scc     2409.749s

jor2 · 2024-03-14T12:01:00Z

could we get controls in the schema too pls, like here

terraform-provider-ibm/ibm/service/scc/data_source_ibm_scc_profile.go

Line 104 in 5dfec70

"controls": {

.

tyao117 · 2024-03-14T21:25:42Z

could we get controls in the schema too pls, like here

terraform-provider-ibm/ibm/service/scc/data_source_ibm_scc_profile.go

Line 104 in 5dfec70

"controls": {

.

I can implement this but it will slow down this datasource significantly. I would need to query every single profile for their controls.

here is a run without the controls included:

make testacc TEST=./ibm/service/scc  TESTARGS="-run TestAccIbmSccProfilesData*"
=== RUN   TestAccIbmSccProfilesDataSourceBasic
--- PASS: TestAccIbmSccProfilesDataSourceBasic (50.92s)
PASS
ok      github.com/IBM-Cloud/terraform-provider-ibm/ibm/service/scc     51.344s

my run with the control doesn't even finish in a reasonable time:

=== RUN   TestAccIbmSccProfilesDataSourceBasic
^Csignal: interrupt <---- me pressing Ctrl+C
FAIL    github.com/IBM-Cloud/terraform-provider-ibm/ibm/service/scc     2246.568s
FAIL

After running it the same make command, my cpu time being after a 30 mins run shoots up:

i would advise using the datasource once you find the profile_id as an alternative.

tyao117 · 2024-03-15T11:49:50Z

needing to push documentation out for the new datasources

tyao117 · 2024-03-15T15:36:11Z

need to add datasource_ibm_control_libraries

tyao117 · 2024-03-18T21:14:29Z

datasource documentation added.

hkantare · 2024-03-21T14:08:32Z

ibm/service/scc/resource_ibm_scc_control_library.go

@@ -876,6 +887,17 @@ func resourceIbmSccControlLibraryControlsInControlLibToMap(model *securityandcom
 	return modelMap, nil
 }

+// using the assessment_id for comparison
+func compareAssessmentSetFunc(v interface{}) int {
+	m := v.(map[string]interface{})


Can you add some checks to check len of map and element of map not nil to eliminate any nil pointer exceptions

In schema assessment_id is defined as optional if user define
assessments = {
assessment_method = "XXXXX"
}

hkantare · 2024-03-21T14:09:22Z

ibm/service/scc/resource_ibm_scc_profile_attachment.go

@@ -321,6 +324,16 @@ func resourceIbmSccProfileAttachmentCreate(context context.Context, d *schema.Re
 	return resourceIbmSccProfileAttachmentRead(context, d, meta)
 }

+func cmpAttachParamSetFunc(v interface{}) int {


same above commands holds here also

hkantare · 2024-03-21T14:09:49Z

ibm/service/scc/resource_ibm_scc_profile_attachment.go

+	assId := (*id)[5:18]
+	var i big.Int
+	i.SetString(strings.Replace(assId, "-", "", 4), 16)
+	val, _ := strconv.Atoi(i.String())


don't supress errors log/throw back the error

tyao117 · 2024-03-21T21:44:44Z

i have implemented the checks that you have asked for.

…hments (IBM-Cloud#5179) * init: pushing out the scc_profiles_datasource * feat: adding the data_source provider_types * fix: using TypeSet for unique assessments * bug: working on resource_ibm_scc_profile_test * Changing various resources to typeSet. * adding the datasource * adding the datasource control libraries and profiles * Updating some documentation * adding the updates to documentation and fmt'ing * Adding the err checks for interface conversion * making both fields to be required if instantiated * fixing the formatting of the test --------- Co-authored-by: Timothy-Yao <[email protected]>

tyao117 added 5 commits March 6, 2024 15:58

init: pushing out the scc_profiles_datasource

4eb6726

feat: adding the data_source provider_types

06c184d

fix: using TypeSet for unique assessments

6da421a

bug: working on resource_ibm_scc_profile_test

32156b3

Changing various resources to typeSet.

5dfec70

This was referenced Mar 13, 2024

SCC Control Library Data Lookup By Name #5186

Closed

SCC Profile Data Lookup By Name #5185

Closed

tyao117 changed the title ~~SCC: adding datasources for the Deployable Architecture team~~ SCC: adding datasources for automated WP connection and Profile attachments Mar 15, 2024

tyao117 added 3 commits March 15, 2024 13:44

adding the datasource

c5ca1b0

adding the datasource control libraries and profiles

3ded53b

Updating some documentation

e8198f4

tyao117 marked this pull request as ready for review March 18, 2024 21:14

adding the updates to documentation and fmt'ing

1983154

hkantare reviewed Mar 21, 2024

View reviewed changes

Adding the err checks for interface conversion

7cbb7a0

tyao117 requested a review from hkantare March 21, 2024 21:46

tyao117 added 2 commits March 22, 2024 12:36

making both fields to be required if instantiated

269a417

fixing the formatting of the test

f8bc5cf

hkantare approved these changes Mar 24, 2024

View reviewed changes

hkantare merged commit 46b9b7f into IBM-Cloud:master Mar 24, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SCC: adding datasources for automated WP connection and Profile attachments #5179

SCC: adding datasources for automated WP connection and Profile attachments #5179

tyao117 commented Mar 6, 2024 •

edited

Loading

jor2 commented Mar 14, 2024

tyao117 commented Mar 14, 2024 •

edited

Loading

tyao117 commented Mar 15, 2024

tyao117 commented Mar 15, 2024

tyao117 commented Mar 18, 2024

hkantare Mar 21, 2024

hkantare Mar 22, 2024

hkantare Mar 21, 2024

hkantare Mar 21, 2024

tyao117 commented Mar 21, 2024

SCC: adding datasources for automated WP connection and Profile attachments #5179

SCC: adding datasources for automated WP connection and Profile attachments #5179

Conversation

tyao117 commented Mar 6, 2024 • edited Loading

Community Note

jor2 commented Mar 14, 2024

tyao117 commented Mar 14, 2024 • edited Loading

tyao117 commented Mar 15, 2024

tyao117 commented Mar 15, 2024

tyao117 commented Mar 18, 2024

hkantare Mar 21, 2024

Choose a reason for hiding this comment

hkantare Mar 22, 2024

Choose a reason for hiding this comment

hkantare Mar 21, 2024

Choose a reason for hiding this comment

hkantare Mar 21, 2024

Choose a reason for hiding this comment

tyao117 commented Mar 21, 2024

tyao117 commented Mar 6, 2024 •

edited

Loading

tyao117 commented Mar 14, 2024 •

edited

Loading