-
Notifications
You must be signed in to change notification settings - Fork 517
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot fetch AWS s3 #157
Comments
Can you run cloudquery with only s3 buckets resource (update the resources in config.hcl). Also please share with me, if possible, the cloudquery.log / any error that returned from that execution? You can also use the Edit: can you please note the Cloudquery version (--version) and plugin version you are using? |
The log is following below. And I am using cloudquery version 0.13.8
And the config.hcl
|
Hi @Rex-office, thanks for the info, I just executed cloudquery v0.13.8 + aws v0.4.13 with your config, and I do see the buckets were inserted.
Did you cut this or it didn't show up? could you please, execute the cloudquery with --verbose flag so we can see debug logs as well? If it didn't insert there should be an error somewhere, we did fix an issue with s3 in v0.14.0 with v0.5.0-beta4 of aws you can try that version, but will require you to delete the whole schema since v0.14.0 introduced a breaking change. |
After execute
|
I am trying to duplicate this on my side, I created a new version aws v0.5.0 and you can use it with cloudquery v0.14.0 (execute it on a clean DB, since it has some breaking changes). Ill try to add ignore errors for this API (we added a fix there in v0.5.0, but I am not sure if its sufficient for this specific edge case. Thanks for the info, will update if I found the correct ignore for this error. Note: v0.14.0 added more verbose error reporting so we should get a more precise error received by the API |
After I installed the v0.14.0, It still does not work in my environment. Seems the same issue.
|
Hi, @Rex-office, thanks for the log. please try AWS v0.5.1, it shouldn't fix the error, but the plugin won't cause the panic and we should get a more detailed error what is causing the failure. |
I changed to AWS v.0.5.2. Seems it still not worked in my enviromnment. The reason is "api error AccessDenied: Access Denied". The log is following below.
|
Great I can see the problem now, will create a fix later today. |
@Rex-office aws provider v0.5.3 should fix this issue, note that some buckets will have less information on them and you will see warning messages. |
I have using the AWS provider v0.5.3. It seems to fix the s3 issue. But I run
|
Hi, @Rex-office I have added in v0.5.4 of AWS provider to ignore access denied in s3 relations, this should solve our issue with the s3 buckets. |
What cloudquery version are you running ? seems like a bug with the console UI. try with Thanks, sorry for the long issue fix. Update: found the bug it was on v0.14.2 -> fixed in v0.14.4 |
Hi, latest version of AWS v0.5.6 should resolve this error, hopefully this is the last one and it should fully execute s3 buckets. |
Synced local file(s) with [cloudquery/.github](https://github.com/cloudquery/.github). --- This PR was created automatically by the [repo-file-sync-action](https://github.com/BetaHuhn/repo-file-sync-action) workflow run [#2540965963](https://github.com/cloudquery/.github/actions/runs/2540965963)
Synced local file(s) with [cloudquery/.github](https://github.com/cloudquery/.github). --- This PR was created automatically by the [repo-file-sync-action](https://github.com/BetaHuhn/repo-file-sync-action) workflow run [#2604938158](https://github.com/cloudquery/.github/actions/runs/2604938158)
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/hashicorp/go-hclog](https://togithub.com/hashicorp/go-hclog) | require | patch | `v1.2.1` -> `v1.2.2` | --- ### Release Notes <details> <summary>hashicorp/go-hclog</summary> ### [`v1.2.2`](https://togithub.com/hashicorp/go-hclog/releases/tag/v1.2.2) [Compare Source](https://togithub.com/hashicorp/go-hclog/compare/v1.2.1...v1.2.2) #### What's Changed - fix various typos in comments by [@​marco-m](https://togithub.com/marco-m) in [https://github.com/hashicorp/go-hclog/pull/115](https://togithub.com/hashicorp/go-hclog/pull/115) - Omit empty colon when message is empty. Fixes [#​109](https://togithub.com/hashicorp/go-hclog/issues/109) by [@​evanphx](https://togithub.com/evanphx) in [https://github.com/hashicorp/go-hclog/pull/116](https://togithub.com/hashicorp/go-hclog/pull/116) **Full Changelog**: hashicorp/go-hclog@v1.2.1...v1.2.2 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4xMzMuMCIsInVwZGF0ZWRJblZlciI6IjMyLjEzMy4wIn0=-->
🤖 I have created a release *beep* *boop* --- ## [0.5.2](cloudquery/cq-provider-terraform@v0.5.1...v0.5.2) (2022-08-07) ### Bug Fixes * **deps:** Update module github.com/aws/aws-sdk-go to v1.44.66 ([#156](cloudquery/cq-provider-terraform#156)) ([3d54470](cloudquery/cq-provider-terraform@3d54470)) * **deps:** Update module github.com/cloudquery/cq-provider-sdk to v0.14.6 ([#154](cloudquery/cq-provider-terraform#154)) ([48cb44f](cloudquery/cq-provider-terraform@48cb44f)) * **deps:** Update module github.com/hashicorp/go-hclog to v1.2.2 ([#157](cloudquery/cq-provider-terraform#157)) ([3a1cc8a](cloudquery/cq-provider-terraform@3a1cc8a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
After running
./cloudquery fetch
, All information is fetched from AWS. But only the information of the s3 backet cannot fetch.I have already got the read-only permission. I run
aws s3 ls
. It cannot show all the backet. So I know it is not a permission issue. May I know how can I solve this issue?The text was updated successfully, but these errors were encountered: