Running on App Engine

[kubaraczkowski]

Hi,

First of all - great thanks! This is a great little project, exactly what I was looking for - go for quick startup, simple code, GCS backend. Super!

I am trying to setup the gopypi server on App Engine, just as you've suggested. However, the templates seem to have trouble loading. What I get is

panic: html/template: pattern matches no files: `/layers/google.go.appengine_gomod/srv/*/*.gohtml`

goroutine 1 [running]:
html/template.Must(...)
	/usr/local/go/src/html/template/template.go:372
github.com/MadJlzz/gopypi/internal/pkg/template.New(0xc000102120)
	/layers/google.go.appengine_gomod/srv/internal/pkg/template/template.go:19 +0xea
main.main()
	/layers/google.go.appengine_gomod/srv/main.go:23 +0x199

It works just fine locally, of course.
I tried running from cmd/gcp/main.go as well as copying the main.go to the root folder.

The app.yaml is:

runtime: go115
service: gopypi
#main: ./cmd/gcs # either this or default from root

entrypoint: main -bucket *bucketname* -port 8080 -credentials ""

Could you perhaps share an example app.yaml?
Any other hints ?

Thanks again!

[MadJlzz]

Hello @kubaraczkowski, thanks for being interrested on to my project! I didn't thought someone would find it usefull so I let him be for a moment.

Actually there is some flaws you need to consider and I would like to fix those before gopypi can be deployed to production.

First, I am using service account private key to authenticate clients for GCS API which is bad. It means credentials are deployed within AppEngine and it is really not needed.

Second, I am creating signed URL for files inside the GCS. This is also not needed and I could use authenticated URL to expose packages.

Last but not least, there is no authentication setup actually. pip is only supporting user/password or password tokens to authenticate clients to package repository. In order to make package retrieving "safe", I need to create firewall rules to only allow certain IPs (which is not good enough or even bad)

Now that I am thinking of it, I should have put everything in the README.md to explain that.

TL;DR

I will create one issue per topic I have to fix and hopefully it will be safer to use gopypi for production grade. In the meantime, I recommand you don't use it and wait I make those fix first!

I'll try to advance on that this weekend and see if I can deliver a version for next week 😉

[kubaraczkowski]

Haha, this is awesome! So honest! :)
There are some alternatives to your project, of course, like pypicloud for instance, but i've seen it be quite slow in startup (important with super low traffic) and well... too complex for what's needed for a small deployment.

I did see your service account requirement, however from what i've read up on GCS in app engine you shouldn't need anything at all (https://cloud.google.com/appengine/docs/standard/go/using-cloud-storage#setting_bucket_and_object_permissions). I think the call to the storage.NewClient() could go without the option and done! I even thought that passing an empty string as the option is going to result in the same, but the program didn't get that far as it crashed on the templates.

I guess you could also 'proxy' the packages through the service not requiring anything special from GCS. That might be less than ideal for larger services though I guess.

Lastly - sure, the security is a concern! In fact in our use case the server would be for running behind a VPC connected to 'the office' via VPN - nothing exposed over the internet.

Cool in any case! Good luck with the project, it's certainly a great case to use go ;)

[MadJlzz]

I made good progress with fixing the mentionned issue. Well, I went a bit extreme and I took the opportunity to review everything that I did. My work is on branch code-structure-ddd-hexagon.

You can find there the app.yaml I used to deploy gopypi on my own GCP project.

Work is still under progress as I have two problems:

1. pip is having trouble installing an available package (pip install resolves package but cannot install it #4)
2. I don't know how to authenticate pip so that he can retrieve packages from a non-public bucket. Actually, I put in place Cloud IAP as a proxy to authenticate requests from a browser but no idea for pip. (pip supports only basic authentication #5)

[MadJlzz]

@kubaraczkowski the issues I had has been solved with PR #6.

Please reopen this issue if you have still trouble deploying,