Unable to Test OAuth2 Workflow Locally Due to Fixed redirect_uri Constraint

[kunfang98927]

Description

I am working on integrating the OAuth2 workflow for publishing new instrument names/images to Wikidata. While implementing the workflow, I encountered an issue with the redirect_uri:

• The application is currently developing locally (e.g., http://localhost:8000).
• Wikidata cannot redirect to localhost, as it requires a publicly accessible URL for the redirect_uri, which is https://vim.simssa.ca/oauth/callback. (This uri was registered when I applied for the OAuth2 consumer, and can not be changed.) Here is the screenshot of the approved OAuth2 consumer:

Steps to Reproduce

1. In UMIL's instrument list page, when users try to add new instrument names directly to wikidata, they will click on a Authorize button.
2. Initiate the OAuth2 flow to redirect users to the Wikidata authorization endpoint.
3. After granting permission, Wikidata attempts to redirect to the fixed redirect_uri (https://vim.simssa.ca/oauth/callback), which is inaccessible when developing locally.

Questions

• Are there any recommended ways to test the OAuth2 workflow locally without modifying the fixed redirect_uri?
• Is there a development-friendly approach to handle this situation?

[kunfang98927]

Current Solution after Scrum meeting this morning (2025-01-10): push the code directly to production server and test this workflow, but quickly revert changes if something goes wrong.

[dchiller]

When you attempt this workflow locally, are you being redirected to vim.simssa.ca after successful authentication? If so, then isn't this working as expected? In other words, if you are being redirected correctly by the service, then there really isn't anything else to test. It may be that when you are developing locally you are getting redirected to vim.simssa.ca rather than the local version, but that's not really a problem, right? Just navigate back to localhost....

[kunfang98927]

Thank you for the comments. @dchiller Maybe I didn't describe it clearly. Let me try to clarify. If I'm still not clear enough, or if I'm doing anything which is obviously wrong, please let me know.

• I'm following this workflow https://www.mediawiki.org/wiki/OAuth/For_Developers#Authorisation_2.
• Currently I can use our client_id to finish Step 1. This means that clicking on this url https://www.wikidata.org/w/rest.php/oauth2/authorize?response_type=code&client_id=4b8617100ef01f44476115fb787d9d18 will take you to the wikidata login page, where you can grant permission. Here is the screenshot of how I can grant permission:

• After I click the "Allow" button, it will take me to https://vim.simssa.ca/oauth/callback?code=<a_temporary_code>, as shown in this screenshot. This code will only send to https://vim.simssa.ca/oauth/callback because this was registered as the OAuth "callback URL", which means I can't get such a code to localhost.

• When developing, I need to fetch this code, back to local, in order to send a POST request to oauth2/access_token using code as query parameter. The response of this POST request will contain an access_token, which I need to make the consequent API calls (e.g., APIs for publishing names/images). That's where I got stuck.

[dchiller]

Gotcha! So what I'm suggesting is that, in that last screenschot, you manually change vim.simssa.ca to localhost in your address bar, you're replicating the workflow for local development. (1) You know that the oauth process sends you to the right place (vim.simssa.ca/oauth/callback?code=....) and (2) once you change vim.simssa.ca to localhost in the address bar you'll be able to see how UMIL does or does not handle the callback in development.

If your local site doesn't work when you make that manual change, then, if I'm understanding the issue correctly, the production site won't either.

[kunfang98927]

Thank you for the suggestion! I changed the vim.simssa.ca to localhost and got this error message...

I should ask ChatGPT for troubleshooting.

[kunfang98927]

The following is from ChatGPT

Verify the redirect_uri
The redirect_uri in your POST request must match exactly the redirect_uri registered in your Wikidata OAuth2 consumer settings.
Registered URL: https://vim.simssa.ca/oauth/callback
If you're testing locally, manually changing it to http://localhost:8000 in the address bar won't work because it's not registered with Wikidata.
Solution: The only way to test locally without modifying the registered redirect_uri is to use a tunneling tool (e.g., ngrok) or test directly on the production server where the redirect_uri matches.

[ahankinson]

Make your local machine vim.simssa.ca by editing the hosts file on your machine? Since it’s all client side your browser won’t know that it’s not communicating with the “real” service.

[kunfang98927]

Make your local machine vim.simssa.ca by editing the hosts file on your machine? Since it’s all client side your browser won’t know that it’s not communicating with the “real” service.

Thank you for the suggestion! I'll try to learn how to edit hosts file on my machine.

[dchiller]

@kunfang98927 RE: the discussion in meeting

Did you map vim.simssa.ca to 127.0.0.1? And then what port is the vim server listening on your local machine? Are you trying to access that port?

And then, can you share where the code is that handles the redirect?

[kunfang98927]

@kunfang98927 RE: the discussion in meeting

Did you map vim.simssa.ca to 127.0.0.1? And then what port is the vim server listening on your local machine? Are you trying to access that port?

And then, can you share where the code is that handles the redirect?

@dchiller Yes, I add 127.0.0.1 vim.simssa.ca in my hosts file as administrator. And when I run ping vim.simssa.ca, I got "Pinging vim.simssa.ca [127.0.0.1] with 32 bytes of data: ...", which means that the mapping does work.

When developing, the vim server is running on localhost:8000.

After I add this line (127.0.0.1 vim.simssa.ca), when I run the redirecting code, it will go to vim.simmsa.ca/oauth/callback?code=<...> but I got this:

Same if I go to vim.simssa.ca:8000

Here's the code of how I implement the redirecting:

1. addNameModal.html

<button type="button" class="btn btn-primary" id="confirmPublishBtn">
  Publish
</button>

2. The listener of the above button (in AddName.js)

document
  .getElementById('confirmPublishBtn')
  .addEventListener('click', function () {
    ...
    // Send the request to publish
    fetch('/publish_name/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'X-CSRFToken': document.querySelector('[name=csrfmiddlewaretoken]')
              .value,
      },
        body: JSON.stringify({
            wikidata_id: wikidataId,
            entries: entries,
            publish_to_wikidata: publishToWikidata,
            account_option: accountOption,
        }),
    })
    .then((response) => response.json())
    ...
...

3. urls.py

...
    path("oauth/authorize/", wikidata_authorize, name="wikidata_authorize"),
    path("oauth/callback/", wikidata_callback, name="wikidata_callback"),
    path("publish_name/", publish_name, name="publish_name"),
...

4. views/publish_name.py

@login_required
def publish_name(request):
...
      access_token = request.session.get("wikidata_access_token")
      if not access_token:
          redirect_url = (
              f"{reverse('wikidata_authorize')}?next={request.path}"
          )
          return redirect(redirect_url)
...

5. views/wiki_apis.py

WIKIDATA_OAUTH_URL = "https://www.wikidata.org/w/rest.php/oauth2"
WIKIDATA_REDIRECT_URI = "https://vim.simssa.ca/oauth/callback"

def wikidata_authorize(request):
    # Get the current target URL from the request
    next_url = request.GET.get("next", "/")
    request.session["next"] = next_url
    # Construct the authorization URL
    base_url = f"{WIKIDATA_OAUTH_URL}/authorize"
    params = {
        "response_type": "code",
        "client_id": WIKIDATA_CLIENT_ID,
        "redirect_uri": WIKIDATA_REDIRECT_URI,
    }
    authorization_url = f"{base_url}?{urllib.parse.urlencode(params)}"
    return redirect(authorization_url)

def wikidata_callback(request):
    authorization_code = request.GET.get("code")
    if not authorization_code:
        return JsonResponse({"error": "Authorization code not provided"}, status=400)

    # Exchange the authorization code for an access token
    base_url = f"{WIKIDATA_OAUTH_URL}/access_token"
    data = {
        "grant_type": "authorization_code",
        "code": authorization_code,
        "client_id": WIKIDATA_CLIENT_ID,
        "redirect_uri": WIKIDATA_REDIRECT_URI,
    }

    response = requests.post(base_url, data=data, timeout=50)
    if response.status_code == 200:
        access_token = response.json().get("access_token")
        request.session["wikidata_access_token"] = access_token
        next_url = request.session.get("next", "/")
        return redirect(next_url)
    else:
        return JsonResponse({"error": response.json()}, status=response.status_code)

[kunfang98927]

What I want to add is that I think the wikidata_authorize is successfully called, as it will generate the correct url https://www.wikidata.org/w/rest.php/oauth2/authorize?response_type=code&client_id=4b8617100ef01f44476115fb787d9d18

I think where we are stuck is that the wikidata_callback wasn't called properly.

[ahankinson]

You need a web server to proxy your Django app from port 443 to port 8000 on your local machine.

You will also probably need the SSL certificates locally so that ssl works. You could also do a self signed certificate.

[kunfang98927]

I generated the ".crt" certificate and ".key", and then I added these lines in my "nginx.conf" file:

     server {
         listen 443 ssl;
         server_name vim.simssa.ca;

         ssl_certificate      ../certs/vim.simssa.ca.crt; # Replace with path to your certificate
         ssl_certificate_key  ../certs/vim.simssa.ca.key; # Replace with path to your key

         location / {
             proxy_pass http://127.0.0.1:8000;  # Forward requests to your Django server

             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         }
     }

And now after authorizing wikidata, the call back url will go to

I've been using ChatGPT to troubleshoot for a long time, but still can't find the solution...
@dchiller Could you have a look when you have time? Really appreciate your help in advance! I've updated the wikidata oauth module in "img-upload" branch b5aa583.

[dchiller]

I can take a look...what do you see in the nginx logs for this request?

Two things I immediately notice:

1. You have those certificate files...are you making them accessible to nginx, which is running inside a Docker container, in some way? In the branch you mention, I don't see any modification to either the nginx/Dockerfile or docker-compose.yml that would copy those files into the container? Are you generating them inside the container or doing this locally somehow?
2. Your nginx configuration is listening on port 443, but I'm not sure that port is mapped to your local port. In docker-compose.yml the only port that is mapped is local port 8000 to the container's port 80...

[kunfang98927]

I can take a look...what do you see in the nginx logs for this request?

Two things I immediately notice:

1. You have those certificate files...are you making them accessible to nginx, which is running inside a Docker container, in some way? In the branch you mention, I don't see any modification to either the nginx/Dockerfile or docker-compose.yml that would copy those files into the container? Are you generating them inside the container or doing this locally somehow?

I did this locally. Do I need to create the certificate files in docker containers?

2. Your nginx configuration is listening on port 443, but I'm not sure that port is mapped to your local port. In docker-compose.yml the only port that is mapped is local port 8000 to the container's port 80...

Okay... So I'll check if there is anything wrong with the "nginx.conf".

[dchiller]

I did this locally. Do I need to create the certificate files in docker containers?

You need to make sure the certificate files are inside the docker container. nginx is going to look for the certificate files at ../certs/vim.simssa.ca.crt inside the container. So you either need to generate them from inside the container, or generate them outside of the container and copy them into the container. You could use the COPY command in nginx/Dockerfile to do this. You are also using a relative path for those certificate files, so make sure that same relative location holds inside the container, or better yet, use an absolute path.

So I'll check if there is anything wrong with the "nginx.conf".

I don't think there's anything wrong with nginx.conf... it's just a question of whether your configuration of nginx.conf, which is listening on port 443 for ssl, matches the port mapping you have in your docker configuration.

In the screenshot, you are getting the error at vim.simssa.ca -- is this mapped to localhost in your hosts file?

[kunfang98927]

I did this locally. Do I need to create the certificate files in docker containers?

You need to make sure the certificate files are inside the docker container. nginx is going to look for the certificate files at ../certs/vim.simssa.ca.crt inside the container. So you either need to generate them from inside the container, or generate them outside of the container and copy them into the container. You could use the COPY command in nginx/Dockerfile to do this. You are also using a relative path for those certificate files, so make sure that same relative location holds inside the container, or better yet, use an absolute path.

Ok. I'll do this. Thank you!

So I'll check if there is anything wrong with the "nginx.conf".

I don't think there's anything wrong with nginx.conf... it's just a question of whether your configuration of nginx.conf, which is listening on port 443 for ssl, matches the port mapping you have in your docker configuration.

In the screenshot, you are getting the error at vim.simssa.ca -- is this mapped to localhost in your hosts file?

Yes... In my hosts file I added 127.0.0.1 vim.simssa.ca

[kunfang98927]

Update: Here are some of the codes and the results. If there is anything wrong, please let me know.

1. Dockerfile

FROM nginx:1.25.2
COPY ./nginx.conf /etc/nginx/nginx.conf
COPY ./vim.conf.template /etc/nginx/templates/vim.conf.template
COPY ./vim.simssa.ca.crt /etc/nginx/templates/vim.simssa.ca.crt
COPY ./vim.simssa.ca.key /etc/nginx/templates/vim.simssa.ca.key

2. vim.conf.template

  upstream vim-app {
    # define the vim-app upstream server
    server vim-app:8001 fail_timeout=0;
  }

  server {
    # if no host match, close connection
    listen 80 default_server;
    return 444;
  }


server {
    listen 80;
    listen 443 ssl;
    client_max_body_size 4G;
    server_name ${HOST_NAME:-vim.simssa.ca} 127.0.0.1 localhost;

    # SSL Configuration
    ssl_certificate      /etc/nginx/templates/vim.simssa.ca.crt;
    ssl_certificate_key  /etc/nginx/templates/vim.simssa.ca.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location /static/ {
        root /virtual-instrument-museum/;
    }

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;

        proxy_pass http://vim-app; # Proxy request to upstream
    }
}

3. hosts file on my computer:

...
127.0.0.1 vim.simssa.ca

The results

1. The 127.0.0.1:8000 works properly:

2. vim.simssa.ca has 502 bad gateway error:

3. oauth/authorize works well:

4. After clicking on "Allow", it goes to the right callback url, but encounters 502 bad gateway error:

Other information

1. When running docker logs vim-app, docker logs vim-nginx, and docker exec:-it vim-nginx curl -I http://vim-app:8001

No issue detected.

[ahankinson]

This is strange:

COPY ./vim.conf.template /etc/nginx/templates/vim.conf.template

Your nginx.conf is looking for:

include /etc/nginx/conf.d/vim.conf; # include virtual host configurations

[ahankinson]

The 127.0.0.1:8000 works properly:

BUT

upstream vim-app {
    # define the vim-app upstream server
    server vim-app:8001 fail_timeout=0;
  }

Is your app on 8000, or 8001? Also, how does vim-app:8001 resolve? Shouldn't this be localhost:8000?

(I'm not sure about cross-docker communication here, so @dchiller is better able to answer that.)

[ahankinson]

• Remove the listen 80; line -- it's going to confuse you.
• Change server_name to just server_name vim.simssa.ca;

server {
    listen 80;
    listen 443 ssl;
    client_max_body_size 4G;
    server_name ${HOST_NAME:-vim.simssa.ca} 127.0.0.1 localhost;

    # SSL Configuration
    ssl_certificate      /etc/nginx/templates/vim.simssa.ca.crt;
    ssl_certificate_key  /etc/nginx/templates/vim.simssa.ca.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location /static/ {
        root /virtual-instrument-museum/;
    }

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;

        proxy_pass http://vim-app; # Proxy request to upstream
    }
}

[dchiller]

vim.simssa.ca has 502 bad gateway error:

Yes. This is expected based on the configuration you've copied. So before we work on the OATH redirection, let's just get this to show up correctly (if vim.simssa.ca has a gateway error, then it's makes sense to me that the callback url would run into problems).

I'll run through what (as I understand your configuration files) is happening when you make a request to vim.simssa.ca on your browser.

1. Your hosts file resolves a request to vim.simssa.ca to 127.0.0.1. When you point your browser to http://vim.simssa.ca, this host is resolved to 127.0.0.1:80. A request to https://vim.simssa.ca resolves to 127.0.0.1:443.
2. So that's where your requests are going, and we'll look now at where there is a server listening.
   • In docker-compose.yml, we can see where each of the container services are listening with the ports configuration:
     • the docker network is routing requests to port 8000 of your local machine to port 80 of the nginx container
     • the docker network is routing requests to port 8983 of your local machine to port 8983 of the solr container.
       So right away, I'm noticing that nothing in the vim app is listening on either port 80 or 443 on your local machine. Unless you have nginx running somewhere else, or your browser has cached something somewhere, I'm surprised you are even seeing anything nginx-related when you go to vim.simssa.ca. Can you figure that out?
   • If you get rid of the listen 80 line in your nginx configuration as Andrew suggests, then your nginx server will listen on port 443 of the nginx docker container. It will then (for the most part) proxy the request through to vim-app upstream, which routes the request to the vim-app docker container at port 8001.
   • If you look at the django server configurations (https://github.com/DDMAL/VIM/blob/develop/web-app/django-startup.sh), you can see that it (or in production, the gunicorn server) is listening at port 8001 (exactly the place that the nginx server is routing requests to).

Put all of this together, and you can see there are two points where you are sending requests to places where there is no listening server:

1. Your browser is resolving requests to vim.simssa.ca to 127.0.0.1 to either port 80 or 443, but nothing in the vim-app is listening at either of those ports on your local machine. You need to adjust where you are sending those requests or you need to adjust docker-compose.yml's port configuration to ensure that requests to these local ports are mapped to requests to a docker container.
2. Your nginx docker container is open at port 80 (that's the 8000:80 mapping again), but your nginx.conf configuration is now listening at port 443. So you need these to match up.

[kunfang98927]

Thank you for all the comments!

Update:

I did these two things:

1. Remove listen 80; in vim.conf.template
2. Modify docker-compose.yml by adding 443:443

...
  nginx:
    build: ./nginx
    container_name: vim-nginx
    restart: unless-stopped
    environment:
      - HOST_NAME=${HOST_NAME}
    ports:
      - "${PORT}:80"
      - "443:443"
    volumes:
      - vim-static:/virtual-instrument-museum/static
    depends_on:
      - app
...

Currently vim.simssa.ca works well.

I think the redirecting issue is currently resolved. I can get the response from the callback now.

This invalid_client error is not relevant to redirecting issue, so I'll continue looking into this as a new issue.

[kunfang98927]

This is strange:

COPY ./vim.conf.template /etc/nginx/templates/vim.conf.template

Your nginx.conf is looking for:

include /etc/nginx/conf.d/vim.conf; # include virtual host configurations

According to nginx's logs below, I think this is because Nginx’s default entrypoint script (/docker-entrypoint.sh) looks for .template files inside /etc/nginx/templates/. It runs envsubst, which replaces environment variables inside the template and saves the result as a .conf file.

[ahankinson]

I should have realized I shouldn’t be so naive as to think it was that straightforward with Docker involved.🙄