Documentation for Google Cloud Data Loss Prevention

[OmairK]

---

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Target Github ISSUE in description if exists
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[potiuk]

Still some checks to fix :(

[OmairK]

Still some checks to fix :(

I fixed the static checks, one of the integration check is breaking.

[turbaszek]

This information can be outdate in some future. So I would suggest to skip this section and let the user use the Google docs

[turbaszek]

I have the impression that there are more DLP operators but they are not covered in the example DAG. I think we should add the documentation. I'm ok with not extending the DAG itself.

[OmairK]

I have the impression that there are more DLP operators but they are not covered in the example DAG. I think we should add the documentation. I'm ok with not extending the DAG itself.

I have added documentation for all the DLP operators. Should I add other DAGs to extend the example?

[mik-laj]

@OmairK Example tasks for the rest operators will be helpful.

[potiuk]

Hey @OmairK - I re-run the latest commit run and there are few errors but not a mypy one :).

1. You will need to rebase to latest master, there were some changes in Dockerfile/requirements recently that will make your CI checks run much faster when you rebase.

2. You have some trailing whitespace - that's why the static checks fail. But if you install pre-commit locally and run it with --all-files those should get fixed automatically.

3. There are some doc errors - all of them have good description, except one that is "document missing in a TOC tree" - you need to add a link to the dlp index in the relevant place in one of the doc/*.rst files.

4. In the "Backport packages" you have an import error from the example dag of dlp - and the reason is that task_id parameter is missing in the example_dlp.

I hope it's helpful.

J.

[OmairK]

Hey @OmairK - I re-run the latest commit run and there are few errors but not a mypy one :).

Thanks alot @potiuk, but this wasn't the PR with mypy errors. I have mentioned you there, sorry for the trouble.
I have extended the example dag and fixed the issue with the docs (I forgot to build docs after the rebase). Is there anything I could do to make the documentation better?

[turbaszek]

This task fails with:

  File "/airflow/airflow/providers/google/cloud/hooks/dlp.py", line 432, in create_stored_info_type
    metadata=metadata,
  File ".virtualenvs/airflow/lib/python3.6/site-packages/google/cloud/dlp_v2/gapic/dlp_service_client.py", line 2625, in create_stored_info_type
    parent=parent, config=config, stored_info_type_id=stored_info_type_id
TypeError: Parameter to MergeFrom() must be instance of same class: expected google.privacy.dlp.v2.StoredInfoTypeConfig got list.

[turbaszek]

I'm happy to help with solving this but probably it would be best if the same person who develops the DAG run the DAG to verify if the example works. @mik-laj @kaxil

[kaxil]

I agree

[turbaszek]

This task fails with this error:

  File "/airflow/airflow/providers/google/cloud/operators/dlp.py", line 433, in execute
    metadata=self.metadata,
  File "/airflow/airflow/providers/google/common/hooks/base_google.py", line 376, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/airflow/airflow/providers/google/cloud/hooks/dlp.py", line 375, in create_job_trigger
    metadata=metadata,
  File "/.virtualenvs/airflow/lib/python3.6/site-packages/google/cloud/dlp_v2/gapic/dlp_service_client.py", line 2557, in create_job_trigger
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 `StorageConfig` must be set.

[turbaszek]

@OmairK would you mind rebasing? I'm ok with merging this PR with not working system tests. We need them to debug the DLP operators that are reported to be not working properly.

[mik-laj]

@OmairK Can you do a rebase and leave a comment so we get notifications? Without it, we don't know that you've found the time to respond to our request.

[OmairK]

@OmairK Can you do a rebase and leave a comment so we get notifications? Without it, we don't know that you've found the time to respond to our request.

@mik-laj I did the rebase. I informed @turbaszek via slack, I thought against sending two notifications for the same thing.

[turbaszek]

@OmairK will you be able to fix the statics checks?

[OmairK]

@OmairK will you be able to fix the statics checks?

@turbaszek Done 😅