download-logs commands returning old/inaccurate data

[molson504x]

Description

I'm debugging our migration automation scripts, and came across an interesting issue. I'm intentionally causing the gh bbs2gh migrate-repo script to fail, and I see it coming back as expected:

gh bbs2gh migrate-repo --bbs-project 'BEAUT' \
  --bbs-repo 'hello-world-obj-c' \
  --github-org '[REDACTED]' \
  --github-repo 'BEAUT_hello-world-obj-c' \
  --bbs-server-url "$BB_SERVER_URL" \
  --bbs-username "$BB_ACCT_USER" \
  --bbs-password "$BB_ACCT_PASSWORD" \
  --ssh-user "$BB_SSH_USER" \
  --bbs-shared-home "$BB_SHARED_HOME_DIR" \
  --ssh-private-key "$BB_SSH_KEY" \
  --ssh-port "$BB_SSH_PORT" \
  --target-repo-visibility private

EXIT_CODE=$(echo $?)

[2025-02-17 15:42:06] [INFO] You are running an up-to-date version of the bbs2gh extension [v1.12.0]
[2025-02-17 15:42:06] [INFO] GITHUB ORG: [REDACTED]
[2025-02-17 15:42:06] [INFO] GITHUB REPO: BEAUT_hello-world-obj-c
[2025-02-17 15:42:06] [INFO] TARGET REPO VISIBILITY: private
[2025-02-17 15:42:06] [INFO] BBS SERVER URL: [REDACTED]
[2025-02-17 15:42:06] [INFO] BBS PROJECT: BEAUT
[2025-02-17 15:42:06] [INFO] BBS REPO: hello-world-obj-c
[2025-02-17 15:42:06] [INFO] BBS USERNAME: [REDACTED]
[2025-02-17 15:42:06] [INFO] BBS PASSWORD: ***
[2025-02-17 15:42:06] [INFO] BBS SHARED HOME: /var/atlassian/application-data/bitbucket/shared
[2025-02-17 15:42:06] [INFO] SSH USER: sftpuser
[2025-02-17 15:42:06] [INFO] SSH PRIVATE KEY: [REDACTED]
[2025-02-17 15:42:06] [INFO] SSH PORT: 2223
[2025-02-17 15:42:07] [ERROR] An unexpected error happened. Please see the logs for details.

This failure is expected... HOWEVER, as a follow-up step, we also run gh bbs2gh download-logs and, well, notice the date?

gh bbs2gh download-logs --github-org '[REDACTED]' \
  --github-pat "$GH_TOKEN" \
  --github-repo 'BEAUT_hello-world-obj-c' \
  --migration-log-file "migration-log_[REDACTED]_BEAUT_hello-world-obj-c.log"

[2025-02-13T21:30:47Z] INFO -- Migration started by molson504x from http://[REDACTED]/projects/BEAUT/repos/hello-world-obj-c/browse to [REDACTED]/BEAUT_hello-world-obj-c
[2025-02-13T21:30:47Z] INFO -- Migration ID: [REDACTED]
[2025-02-13T21:30:47Z] INFO -- -----------------------------
[2025-02-13T21:30:47Z] INFO -- Git source migration started
[2025-02-13T21:32:25Z] INFO -- Git source migration completed
[2025-02-13T21:32:25Z] INFO -- -----------------------------
[2025-02-13T21:32:26Z] INFO -- Extraction started
[2025-02-13T21:32:33Z] INFO -- Extraction complete
[2025-02-13T21:32:33Z] INFO -- -----------------------------
[2025-02-13T21:32:33Z] INFO -- Transformation started
[2025-02-13T21:32:34Z] INFO -- Transformation complete
[2025-02-13T21:32:34Z] INFO -- -----------------------------
[2025-02-13T21:32:35Z] INFO -- Import started
[2025-02-13T21:32:36Z] WARN -- 'Allow Forking' setting could not be enabled for the repository because it is not enabled at the organization level.
[2025-02-13T21:32:47Z] INFO -- Import complete
[2025-02-13T21:32:47Z] INFO -- -----------------------------
[2025-02-13T21:32:47Z] INFO -- Migration complete
[2025-02-13T21:32:47Z] INFO -- -----------------------------

This was run at around 2025-02-17 15:42:10, so I'm not sure why it's giving me logs back from 2025-02-13... AND I thought these logs were only available for 24h? What's even more interesting is that I'm causing a failure on the export side of the script by passing it an invalid set of credentials for the azure storage account, which means it should never hit the import step, which makes the download-logs output extra wrong since the API call to the GraphQL startRepositoryMigration mutator wouldn't have happened yet.

The reason there's migration data here (I suspect) is, since I'm working in a "lab" environment for developing the automation, we are deleting and re-creating the same repos repeatedly, so there's a chance that this is leftover (old) data from the last successful previous attempt before I started intentionally causing the gh bbs2gh migrate-repo step to fail.

Reproduction Steps

1. [optional to produce bad data] Run a successful migration using the gh bbs2gh migrate-repo command, followed by a gh bbs2gh download-logs command. Observe logs coming back indicating success.
2. Delete the migrated repo from GitHub.
3. Intentionally cause the gh bbs2gh migrate-repo step to fail. In my case, I'm passing a bad set of credentials for the AZURE_STORAGE_ACCOUNT_CONNECTION_STRING env variable to ensure that the error I'm causing is happening before the import step begins. Observe the gh bbs2gh migrate-repo failure, and it produces appropriate failure logs. Also observe that the repo to be imported doesn't exist yet.
4. Run the gh bbs2gh download-logs command. Observe logs are returned indicating a successful import... Old leftover data, ok. Also observe the warning that logs are only available for 24 hours.
5. Wait a couple of days (like over the weekend, which is what I did)
6. Run the gh bbs2gh download-logs command again and observe that the same logs come back indicating success, despite being over 24h past when the migration occurred and the repo not existing in GitHub and the previous gh bbs2gh migrate-repo attempts failing.

[ArinGhazarian]

👋

This is by design as download-logs just downloads the latest log. In your case, since you have already performed a successful migration, it downloads the latest available log that pertains to your most recent migration. Also as you mentioned the one that was run on 2025-02-17 at 15:42:10 didn't even get to the actual migration so there were no logs available for it.
The only problem that I can see is the fact that the log is available for more that 24 hours. I will follow it up with the team and get back to you.

[molson504x]

Hey @ArinGhazarian - thanks for the response. That makes sense, the migration logs aren't actually tied to the migrated repo (they're stored separately). Is that something that can be changed, so the migration logs are tied to the actual repo? Or is there a way to make an API call (or an extension to the gei tooling) to purge the migration logs?

[ArinGhazarian]

The migration logs aren't actually tied to the migrated repo (they're stored separately)

Apologies for not being clear. The logs are stored separately but they are tied to the migrated repo. The download-logs command requires a github-org and a github-repo to be able to download the logs for that repo so it downloads the latest log associated to a repo. Here is the GraphQL query that the GetMigrationLogUrl method calls to get the migration logs associated with a repo.

Is that something that can be changed, so the migration logs are tied to the actual repo?

As I described above they are tied to a repo but deleting a repo doesn't remove the migration log. If you run a new migration for that repo again and it reaches to the actual migration a new migration log will be uploaded for it and can be downloaded.

Is there a way to make an API call (or an extension to the gei tooling) to purge the migration logs?

No. The logs are read-only and maintained by the GEI's backend.

Also as a quick note, the migration log is also available as an issue in the migrated repo but it only reflects the backend migration events in a user-friendly fashion. The gh gei CLI does extra pre migration stuff that won't be reflected in the migration log. Like in case of bbs migrations (gh bbs2gh), depending on the provided args, it may generate, download and upload the migration archive, then it calls the backend GQL mutation to perform the migration.