Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support migration from AWS CodeCommit #31981

Merged
merged 12 commits into from
Sep 10, 2024
Merged

Support migration from AWS CodeCommit #31981

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b57d633
add codecommit downloader
Zettat123 Aug 30, 2024
d37835d
impl GetComments
Zettat123 Aug 30, 2024
a0e68f8
fix
Zettat123 Sep 4, 2024
c9b195f
fix baseURL
Zettat123 Sep 4, 2024
960068b
improvements
Zettat123 Sep 5, 2024
55dc9d5
add codecommit.tmpl
Zettat123 Sep 5, 2024
fb7ab0e
make tidy
Zettat123 Sep 5, 2024
fdde5b5
generate swagger
Zettat123 Sep 5, 2024
c506736
update logo
Zettat123 Sep 5, 2024
44939a8
Merge branch 'main' into feat/migration-from-codecommit
Zettat123 Sep 6, 2024
4143b68
improve GetComments
Zettat123 Sep 10, 2024
53f9c28
Merge branch 'main' into feat/migration-from-codecommit
Zettat123 Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions assets/go-licenses.json
View file
Open in desktop

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ require (
github.com/PuerkitoBio/goquery v1.9.2
github.com/SaveTheRbtz/zstd-seekable-format-go/pkg v0.7.2
github.com/alecthomas/chroma/v2 v2.14.0
github.com/aws/aws-sdk-go v1.43.21
github.com/aws/aws-sdk-go-v2/credentials v1.17.30
github.com/aws/aws-sdk-go-v2/service/codecommit v1.25.1
github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb
github.com/blevesearch/bleve/v2 v2.4.2
github.com/buildkite/terminal-to-html/v3 v3.12.1
Expand Down Expand Up @@ -146,6 +149,10 @@ require (
github.com/andybalholm/cascadia v1.3.2 // indirect
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aws/aws-sdk-go-v2 v1.30.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.16 // indirect
github.com/aws/smithy-go v1.20.4 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bits-and-blooms/bitset v1.13.0 // indirect
Expand Down
18 changes: 18 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,20 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPd
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aws/aws-sdk-go v1.43.21 h1:E4S2eX3d2gKJyI/ISrcIrSwXwqjIvCK85gtBMt4sAPE=
github.com/aws/aws-sdk-go v1.43.21/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
github.com/aws/aws-sdk-go-v2 v1.30.4 h1:frhcagrVNrzmT95RJImMHgabt99vkXGslubDaDagTk8=
github.com/aws/aws-sdk-go-v2 v1.30.4/go.mod h1:CT+ZPWXbYrci8chcARI3OmI/qgd+f6WtuLOoaIA8PR0=
github.com/aws/aws-sdk-go-v2/credentials v1.17.30 h1:aau/oYFtibVovr2rDt8FHlU17BTicFEMAi29V1U+L5Q=
github.com/aws/aws-sdk-go-v2/credentials v1.17.30/go.mod h1:BPJ/yXV92ZVq6G8uYvbU0gSl8q94UB63nMT5ctNO38g=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16 h1:TNyt/+X43KJ9IJJMjKfa3bNTiZbUP7DeCxfbTROESwY=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16/go.mod h1:2DwJF39FlNAUiX5pAc0UNeiz16lK2t7IaFcm0LFHEgc=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.16 h1:jYfy8UPmd+6kJW5YhY0L1/KftReOGxI/4NtVSTh9O/I=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.16/go.mod h1:7ZfEPZxkW42Afq4uQB8H2E2e6ebh6mXTueEpYzjCzcs=
github.com/aws/aws-sdk-go-v2/service/codecommit v1.25.1 h1:mOOALIM4JzhYkq3voCBbmZqmyEVEhHsfasMTbVxLkNs=
github.com/aws/aws-sdk-go-v2/service/codecommit v1.25.1/go.mod h1:6zf5j3mIUXKM0s2iz5ttR2Qwq+o47D0jotpAyaKgZRA=
github.com/aws/smithy-go v1.20.4 h1:2HK1zBdPgRbjFOHlfeQZfpC4r72MOb9bZkiFwggKO+4=
github.com/aws/smithy-go v1.20.4/go.mod h1:irrKGvNn1InZwb2d7fkIRNucdfwR8R+Ts3wxYa/cJHg=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
Expand Down Expand Up @@ -509,6 +523,9 @@ github.com/jessevdk/go-flags v1.5.0 h1:1jKYvbxEjfUl0fmqTCOfonvskHHXMjBySTLW4y9LF
github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
github.com/jhillyerd/enmime v1.2.0 h1:dIu1IPEymQgoT2dzuB//ttA/xcV40NMPpQtmd4wslHk=
github.com/jhillyerd/enmime v1.2.0/go.mod h1:FRFuUPCLh8PByQv+8xRcLO9QHqaqTqreYhopv5eyk4I=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
Expand Down Expand Up @@ -913,6 +930,7 @@ golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81R
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
Expand Down
3 changes: 3 additions & 0 deletions modules/migration/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,7 @@ type MigrateOptions struct {
ReleaseAssets bool
MigrateToRepoID int64
MirrorInterval string `json:"mirror_interval"`

AWSAccessKeyID string
AWSSecretAccessKey string
}
25 changes: 16 additions & 9 deletions modules/structs/repo.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,15 +291,16 @@ type GitServiceType int

// enumerate all GitServiceType
const (
NotMigrated GitServiceType = iota // 0 not migrated from external sites
PlainGitService // 1 plain git service
GithubService // 2 github.com
GiteaService // 3 gitea service
GitlabService // 4 gitlab service
GogsService // 5 gogs service
OneDevService // 6 onedev service
GitBucketService // 7 gitbucket service
CodebaseService // 8 codebase service
NotMigrated GitServiceType = iota // 0 not migrated from external sites
PlainGitService // 1 plain git service
GithubService // 2 github.com
GiteaService // 3 gitea service
GitlabService // 4 gitlab service
GogsService // 5 gogs service
OneDevService // 6 onedev service
GitBucketService // 7 gitbucket service
CodebaseService // 8 codebase service
CodeCommitService // 9 codecommit service
)

// Name represents the service type's name
Expand All @@ -325,6 +326,8 @@ func (gt GitServiceType) Title() string {
return "GitBucket"
case CodebaseService:
return "Codebase"
case CodeCommitService:
return "CodeCommit"
case PlainGitService:
return "Git"
}
Expand Down Expand Up @@ -361,6 +364,9 @@ type MigrateRepoOptions struct {
PullRequests bool `json:"pull_requests"`
Releases bool `json:"releases"`
MirrorInterval string `json:"mirror_interval"`

AWSAccessKeyID string `json:"aws_access_key_id"`
AWSSecretAccessKey string `json:"aws_secret_access_key"`
}

// TokenAuth represents whether a service type supports token-based auth
Expand All @@ -382,6 +388,7 @@ var SupportedFullGitService = []GitServiceType{
OneDevService,
GitBucketService,
CodebaseService,
CodeCommitService,
}

// RepoTransfer represents a pending repo transfer
Expand Down
5 changes: 5 additions & 0 deletions options/locale/locale_en-US.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1176,6 +1176,11 @@ migrate.gogs.description = Migrate data from notabug.org or other Gogs instances
migrate.onedev.description = Migrate data from code.onedev.io or other OneDev instances.
migrate.codebase.description = Migrate data from codebasehq.com.
migrate.gitbucket.description = Migrate data from GitBucket instances.
migrate.codecommit.description = Migrate data from AWS CodeCommit.
migrate.codecommit.aws_access_key_id = AWS Access Key ID
migrate.codecommit.aws_secret_access_key = AWS Secret Access Key
migrate.codecommit.https_git_credentials_username = HTTPS Git Credentials Username
migrate.codecommit.https_git_credentials_password = HTTPS Git Credentials Password
migrate.migrating_git = Migrating Git Data
migrate.migrating_topics = Migrating Topics
migrate.migrating_milestones = Migrating Milestones
Expand Down
1 change: 1 addition & 0 deletions public/assets/img/svg/gitea-codecommit.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions routers/api/v1/repo/migrate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,10 @@ func Migrate(ctx *context.APIContext) {
opts.PullRequests = false
opts.Releases = false
}
if gitServiceType == api.CodeCommitService {
opts.AWSAccessKeyID = form.AWSAccessKeyID
opts.AWSSecretAccessKey = form.AWSSecretAccessKey
}

repo, err := repo_service.CreateRepositoryDirectly(ctx, ctx.Doer, repoOwner, repo_service.CreateRepoOptions{
Name: opts.RepoName,
Expand Down
4 changes: 4 additions & 0 deletions routers/web/repo/migrate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,10 @@ func MigratePost(ctx *context.Context) {
opts.PullRequests = false
opts.Releases = false
}
if form.Service == structs.CodeCommitService {
opts.AWSAccessKeyID = form.AWSAccessKeyID
opts.AWSSecretAccessKey = form.AWSSecretAccessKey
}

err = repo_model.CheckCreateRepository(ctx, ctx.Doer, ctxUser, opts.RepoName, false)
if err != nil {
Expand Down
2 changes: 2 additions & 0 deletions services/convert/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ func ToGitServiceType(value string) structs.GitServiceType {
return structs.OneDevService
case "gitbucket":
return structs.GitBucketService
case "codecommit":
return structs.CodeCommitService
default:
return structs.PlainGitService
}
Expand Down
3 changes: 3 additions & 0 deletions services/forms/repo_form.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ type MigrateRepoForm struct {
PullRequests bool `json:"pull_requests"`
Releases bool `json:"releases"`
MirrorInterval string `json:"mirror_interval"`

AWSAccessKeyID string `json:"aws_access_key_id"`
AWSSecretAccessKey string `json:"aws_secret_access_key"`
}

// Validate validates the fields
Expand Down
Loading