CSV imports (UI) #3845

ruslandoga · 2024-02-28T16:34:42Z

Changes

This PR adds UI for CSV imports via S3.

screen-recording-2024-03-20-at-183854_Esviayqm.mp4

Tests

Automated tests have been added

Changelog

Entry has been added to changelog

Documentation

Docs have been updated

Dark mode

The UI has been tested both in dark and light mode

Makefile

zoldar

Can you rebase the branch against current master? 🙏

zoldar · 2024-03-21T11:05:30Z

lib/plausible_web/controllers/site_controller.ex

@@ -708,7 +708,7 @@ defmodule PlausibleWeb.SiteController do
    |> redirect(external: Routes.site_path(conn, :settings_integrations, site.domain))
  end

-  def export(conn, _params) do


lib/plausible_web/live/csv_import.ex

zoldar · 2024-03-21T13:21:46Z

The upload dialog does not let me select particular files, I'm not sure why 🤔. The only way I can select the files is entering the (extracted) export directory and clicking "Upload". FF gives a warning that I'm trying to upload all the files from a given folder but they appear in the upload list once I confirm it.
I have first made an export from freshly loaded seeds, made sure minio container is running, extracted the export, uploaded it and initiated import. The worker seems to start and it reaches https://github.com/ruslandoga/analytics/blob/50f6a3d7bab940a2f3bc11cd4e26590e0533019f/lib/plausible/imported/csv_importer.ex#L41 for the first upload but remains stuck there for several minutes without any activity (the task is hanging in "executing" state for that time as well). Have you experienced anything like that?
I'm not sure if we have discussed this, but are we planning to support uploading zipped archive as well? That would make the export->import process more streamlined for the common scenario of moving between hosted and cloud, one way or another.

ruslandoga · 2024-03-21T13:26:24Z

I'll try using Firefox tomorrow.
No
Yeah, but the importer would need to be reworked to support Zip archives, maybe https://clickhouse.com/blog/clickhouse-release-23-08#direct-import-from-archives-nikita-keba-antonio-andelic-pavel-kruglov could be used once it lands: Direct import from archives don't work for table function s3 ClickHouse/ClickHouse#54726

ruslandoga · 2024-03-22T07:29:29Z

Removing webkitdirectory added in e803909 makes it "work" in FF. Here're some other problems I noticed:

with webkitdirectory set FF doesn't support directory drag and drop
with webkitdirectory set FF doesn't support picking individual files
with webkitdirectory set Chrome doesn't support picking individual files, but supports directory drag and drop
with webkitdirectory set both Chrome and FF warn about multiple files upload (and they both wrongly show the number of files as 10, I guess they count directory as a file)

I removed webkitdirectory in a4840c1 as it seems less intuitive than a basic multiple file upload.

Thank you very much for spotting it! I should've tested browsers other than Safari before committing.

ruslandoga · 2024-03-22T08:15:46Z

Re 2, are there any errors on the Oban job? Maybe ClickHouse S3 timeout or similar. In that case it probably means you need to set S3_CLICKHOUSE_HOST=<MinIO ip address, from docker network inspect bridge> to make MinIO visible from ClickHouse container.

More info:

the first upload but remains stuck there for several minutes without any activity

Hm, but S3 timeout happens after 60s, so maybe it's something else. With S3 timeout, it would also appear to get stuck on a different line, right above Ch.query!

zoldar · 2024-03-22T09:36:22Z

Oban Job kept executing without any errors.

I have resolved the issue by swapping "host.docker.internal" for "localhost" in S3 URLs sent to import worker:

diff --git a/lib/plausible/imported/csv_importer.ex b/lib/plausible/imported/csv_importer.ex
index 6376d851d..6f6c422ee 100644
--- a/lib/plausible/imported/csv_importer.ex
+++ b/lib/plausible/imported/csv_importer.ex
@@ -65,7 +65,9 @@ defmodule Plausible.Imported.CSVImporter do
           "end_date" => end_date
         }
 
+      IO.inspect upload, label: :ABOUT_TO_QUERY
       Ch.query!(ch, statement, params, timeout: :infinity)
+      IO.inspect upload, label: :AFTER_QUERY # THIS WAS NEVER SHOWN UNTIL I CHANGED THE HOST
     end)
   rescue
     # we are cancelling on any argument or ClickHouse errors
diff --git a/lib/plausible_web/live/csv_import.ex b/lib/plausible_web/live/csv_import.ex
index 38c97ca17..f538b3e5a 100644
--- a/lib/plausible_web/live/csv_import.ex
+++ b/lib/plausible_web/live/csv_import.ex
@@ -168,6 +168,8 @@ defmodule PlausibleWeb.Live.CSVImport do
     %{s3_url: s3_url, presigned_url: upload_url} =
       Plausible.S3.import_presign_upload(socket.assigns.site_id, entry.client_name)
 
+    s3_url = String.replace(s3_url, "localhost", "host.docker.internal")
+
     {:ok, %{uploader: "S3", s3_url: s3_url, url: upload_url}, socket}
   end

After the change, the import proceeds basically immediately.

In CH logs I can see connection failing with "connection refused" when running the insert query but it surprisingly seems to not propagate back to the client somehow? I'm not sure, I haven't digged too deep into this.

zoldar · 2024-03-22T09:40:19Z

BTW something for a follow-up perhaps but IMHO it would make sense to hide CSV export and import UI when S3 is not configured, at least until local storage version is implemented. WDYT?

ruslandoga · 2024-03-22T10:13:37Z

When I don't set S3_CLICKHOUSE_HOST or replace localhost in the S3 URLs Oban jobs fail for me:

iex> Plausible.Repo.all Oban.Job
[
  %Oban.Job{
    __meta__: #Ecto.Schema.Metadata<:loaded, "oban_jobs">,
    id: 19,
    state: "discarded",
    queue: "analytics_imports",
    worker: "Plausible.Workers.ImportAnalytics",
    args: %{
      "end_date" => "2024-01-23",
      "import_id" => 10,
      "start_date" => "2023-02-09",
      "uploads" => [
        %{
          "filename" => "imported_visitors_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_visitors_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_sources_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_sources_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_pages_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_pages_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_operating_systems_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_operating_systems_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_locations_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_locations_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_exit_pages_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_exit_pages_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_entry_pages_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_entry_pages_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_devices_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_devices_20230209_20240123.csv"
        },
        %{
          "filename" => "imported_browsers_20230209_20240123.csv",
          "s3_url" => "http://localhost:10000/dev-imports/3/imported_browsers_20230209_20240123.csv"
        }
      ]
    },
    meta: %{},
    tags: [],
    errors: [
      %{
        "at" => "2024-03-22T10:11:52.453813Z",
        "attempt" => 1,
        "error" => "** (Oban.PerformError) Plausible.Workers.ImportAnalytics failed with {:discard, \"Code: 499. DB::Exception: Failed to get object info: Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.3.7.5 (official build)). HTTP response code: 18446744073709551615. (S3_ERROR) (version 23.3.7.5 (official build))\\n\"}"
      }
    ],
    attempt: 1,
    attempted_by: ["192", "eac88a52-9e1c-4360-af93-b4655763d6a2"],
    max_attempts: 3,
    priority: 0,
    attempted_at: ~U[2024-03-22 10:11:26.687453Z],
    cancelled_at: nil,
    completed_at: nil,
    discarded_at: ~U[2024-03-22 10:11:52.453798Z],
    inserted_at: ~U[2024-03-22 10:11:26.652690Z],
    scheduled_at: ~U[2024-03-22 10:11:26.652690Z],
    conf: nil,
    conflict?: false,
    replace: nil,
    unique: nil,
    unsaved_error: nil
  }
]

I wonder what's different between our setups (I'm running make clickhouse-prod and make minio). Ideally the jobs would fail properly for everyone.

ruslandoga · 2024-03-22T10:15:22Z

BTW something for a follow-up perhaps but IMHO it would make sense to hide CSV export and import UI when S3 is not configured, at least until local storage version is implemented. WDYT?

~~It's already implemented :) I'll just copy it from #3692 once this PR is finished.~~ I've started experimenting with various export/import implementations for CE. I'll keep them in draft for now but I'm sure they would be ready by the time the feature flag goes away.

ruslandoga commented Feb 28, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

ruslandoga changed the title ~~add basic CSV import~~ CSV import UI Feb 28, 2024

ruslandoga force-pushed the allow-csv-imports-ui branch from 94d52f8 to a3abbe5 Compare February 29, 2024 17:16

This was referenced Mar 12, 2024

CSV exports (no UI) #3836

Merged

CSV exports (UI) #3875

Merged

CSV imports (no UI) #3895

Merged

ruslandoga changed the title ~~CSV import UI~~ CSV imports (UI) Mar 14, 2024

ruslandoga mentioned this pull request Mar 15, 2024

Allow using verified routes #3900

Closed

4 tasks

ruslandoga force-pushed the allow-csv-imports-ui branch 8 times, most recently from e25903d to a05b801 Compare March 20, 2024 10:35

ruslandoga marked this pull request as ready for review March 20, 2024 10:35

ruslandoga requested a review from zoldar March 20, 2024 10:35

zoldar reviewed Mar 21, 2024

View reviewed changes

ruslandoga added 5 commits March 21, 2024 20:04

add basic ui

8b19c13

remove TODO

7f15c7e

credo

f694f3e

allow folder upload

e803909

redirect external

Loading
Loading status checks…

c817a13

ruslandoga force-pushed the allow-csv-imports-ui branch from d9cec1e to c817a13 Compare March 21, 2024 12:07

mention folder, use folder icon for file picker

Loading
Loading status checks…

50f6a3d

ruslandoga commented Mar 21, 2024

View reviewed changes

lib/plausible_web/live/csv_import.ex Outdated Show resolved Hide resolved

ruslandoga added 2 commits March 22, 2024 15:38

back to multiple file upload

Loading
Loading status checks…

a4840c1

mention zip

Loading
Loading status checks…

081e5d7

This was referenced Mar 25, 2024

"on-the-fly" CSV export #3937

Closed

"temp file" CSV export #3938

Closed

escape dots in archive filename

Loading
Loading status checks…

af97453

ruslandoga mentioned this pull request Mar 25, 2024

Local CSV exports/imports and S3/UI updates #3944

Closed

9 tasks

zoldar approved these changes Mar 26, 2024

View reviewed changes

zoldar merged commit c263df5 into plausible:master Mar 26, 2024
9 checks passed

ruslandoga deleted the allow-csv-imports-ui branch June 10, 2024 16:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CSV imports (UI) #3845

CSV imports (UI) #3845

ruslandoga commented Feb 28, 2024 •

edited

Loading

zoldar left a comment

zoldar Mar 21, 2024

zoldar commented Mar 21, 2024

ruslandoga commented Mar 21, 2024 •

edited

Loading

ruslandoga commented Mar 22, 2024 •

edited

Loading

ruslandoga commented Mar 22, 2024 •

edited

Loading

zoldar commented Mar 22, 2024

zoldar commented Mar 22, 2024

ruslandoga commented Mar 22, 2024 •

edited

Loading

ruslandoga commented Mar 22, 2024 •

edited

Loading

CSV imports (UI) #3845

CSV imports (UI) #3845

Conversation

ruslandoga commented Feb 28, 2024 • edited Loading

Changes

Tests

Changelog

Documentation

Dark mode

zoldar left a comment

Choose a reason for hiding this comment

zoldar Mar 21, 2024

Choose a reason for hiding this comment

zoldar commented Mar 21, 2024

ruslandoga commented Mar 21, 2024 • edited Loading

ruslandoga commented Mar 22, 2024 • edited Loading

ruslandoga commented Mar 22, 2024 • edited Loading

zoldar commented Mar 22, 2024

zoldar commented Mar 22, 2024

ruslandoga commented Mar 22, 2024 • edited Loading

ruslandoga commented Mar 22, 2024 • edited Loading

ruslandoga commented Feb 28, 2024 •

edited

Loading

ruslandoga commented Mar 21, 2024 •

edited

Loading

ruslandoga commented Mar 22, 2024 •

edited

Loading

ruslandoga commented Mar 22, 2024 •

edited

Loading

ruslandoga commented Mar 22, 2024 •

edited

Loading

ruslandoga commented Mar 22, 2024 •

edited

Loading