Failures in getJSON crash the build instead of allowing the user to handle it.

[Lnaden]

As suggested in #5643 I am creating a new issue for this.

getJSON causes builds to fail and halt in the versions 0.68.3, 0.69.1 and 0.69.2 which I have tested it on. This happens when accessing a remote file at a HTTP address (have not tested for local json files, not my use case).

The use case I have is that we're checking a database through its REST API for the existence of an entry to pull some metadata, and falling back to local data if not present. However, the only way we can check for the existence is to make a query, and if it's not there, the getJSON errors, and the build halts.

Specific error is this: Failed to get JSON resource "https://data.rcsb.org/rest/v1/core/entry/": Failed to retrieve remote file: Not Found.

Supposedly #5648 was meant to fix this, but from what I'm reading never did it properly?

I don't know Go so I'm not sure exactly how to fix it, but I think I can trace its stack:

• getJson calls getResource and then if an error is thrown its supposed to log said error but not throw the error:

  hugo/tpl/data/data.go

  Lines 118 to 122 in c2d9fd1

  	err = ns.getResource(cache, unmarshal, req)
  	if err != nil {
  	ns.deps.Log.ERROR.Printf("Failed to get JSON resource %q: %s", url, err)
  	return nil, nil
  	}

• getResource attempts to call getRemote because the a URL was passed in, so the "" case is skipped and it goes to the default:

  hugo/tpl/data/resources.go

  Lines 109 to 122 in c2d9fd1

  	switch req.URL.Scheme {
  	case "":
  	url, err := url.QueryUnescape(req.URL.String())
  	if err != nil {
  	return err
  	}
  	b, err := getLocal(url, ns.deps.Fs.Source, ns.deps.Cfg)
  	if err != nil {
  	return err
  	}
  	_, err = unmarshal(b)
  	return err
  	default:
  	return ns.getRemote(cache, unmarshal, req)

• getRemote detects an HTTPError and actually throws the error, which passes back up the stack:

  hugo/tpl/data/resources.go

  Lines 55 to 57 in c2d9fd1

  	if isHTTPError(res) {
  	return nil, errors.Errorf("Failed to retrieve remote file: %s", http.StatusText(res.StatusCode))
  	}

• getResource and getJSON behave correctly and log the error, but the thing actually halting the build came from the getRemote call and the HTTPError handling.

I have a couple ideas on what behavior I would like to see to make this more user and use-case friendly.

1. Is it possible to make the getRemote on HTTPError not actually throw the error and instead log it, pass it back up and let the calling function handle it? Especially since getRemote isn't a publicly available function in Hugo's shortcodes?
2. Could there be some switch to tell the getRemote to pass it back up (like if there is another function invoking which could handle it?)
3. Could the getRemote or getJson throw a warning instead? This does leave it up to the site developer to properly handle the case when the JSON they get back isn't what they expect, but I think that's a reasonable thing to ask for.
4. Use a with...else...end construct to handle the return from getJSON when it errors. This actually already works as expected. I can test it with the shortcode construct of

   {{ with getJson "...." }}
       <!-- Do stuff on success -->
   {{ else }}
       {{ warnf "It failed" }}
   {{ end }}
   

   This will correctly throw not only the getJSON error, but also throw the warnf warning in the console. However, the fact that an error was thrown causes the built to fail in the end, but the getJSON error clearly is not fatal to the overall build code.
5. Have a function to probe an external site? Basically a function to see if the url exists. I think getRemote does this already to some extent, but make this return a boolean instead of the result or error?

What version of Hugo are you using (hugo version)?

0.68.3, 0.69.1 and 0.69.2 (currently using)

Sepcifically: Hugo Static Site Generator v0.69.2/extended darwin/amd64 installed via brew

Does this issue reproduce with the latest release?

Yes: 0.69.2

[Lnaden]

For additional data, the remote API service I'm calling does return the following when you give it a non-existent data field (using 6ZLE as the requested object, which doesn't exist):

{"status":404,"message":"No data found for entryId: 6ZLE","link":"https://data.rcsb.org/redoc/index.html"}

However, I feel that even if the API was down or the URL someone fed into simply did not exist at all, the getJSON function should still allow a more graceful handling.

[maelle]

Related to this (and partly repeating ideas from above, sorry), would it be possible to have

• an argument to getJSON/getRemote (and maybe even built-in shortcodes) that'd allow to give a warning instead of an error with a default behavior to error -- and maybe even a fallback option;

• a hugo build/server option to change that error/warning option site-wide. (so for instance the build could be less strict in branches, more strict on master, depending on a Netlify config)

Use case

• Being able to ignore that a tweet/instagram post disappeared when working on something else in the website. see Build should not fail when Instagram post is missing in shortcode #6954

It's great to get some message that a resource couldn't be fetched, to e.g. remove deleted tweets IDs from content, but sometimes it halts work on other issues (e.g. I'm editing a post, and the build will fail because a tweet ID in another post is obsolete).

[butaca]

I'm getting this error when a user deletes a tweet (custom template, not using a shortcode: I have a list of tweets ids and fetching them from the Twitter public API). It would be great to be able to handle the error instead of crashing the build.

[robsonsobral]

Tested on 0.72.0. The issue remains

[SimZhou]

Same issue here:

ERROR 2020/06/24 13:58:13 Failed to get JSON resource "https://api.instagram.com/oembed/?url=https://instagram.com/p/BWNjjyYFxVx/&hidecaption=1": Get "https://api.instagram.com/oembed/?url=https://instagram.com/p/BWNjjyYFxVx/&hidecaption=1": dial tcp 31.13.97.245:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Built in 30695 ms
Error: Error building site: "C:\Users\XXX\My Programs\Hugo\testsite\content\posts\theme-documentation-built-in-shortcodes\index.zh-cn.md:1:1": timed out initializing value. You may have a circular loop in a shortcode, or your site may have resources that take longer to build than the `timeout` limit in your Hugo config file.

Had a quick search, some effort was already devoted since #5643 , but the issue remains.
Similar issues: #5659 #6954

Better to mark this issue as a bug.

[onedrawingperday]

Since fdfa4a5 the following can be entered in a project's config to allow building a project even if a remote source called by getJSON is missing.

If you feel that this should not be logged as an ERROR, you can ignore it by adding this to your site config:
ignoreErrors = ["error-remote-getjson"]

P.S. This issue needs to be closed.

[maelle]

Good to know!
However isn't this issue about being able to add some sort of fallback?

[onedrawingperday]

@maelle

With the above in a project's config the console will show a WARN for a missing getJSON remote. It's up to you what you're going to do with it. The project build will no longer be halted.

[Lnaden]

That's great to hear! I'd need to test it more thoroughly to make sure it works correctly.

@maelle when I originally made the issue, you can establish a fallback with a with...else block such as this:

{{ with getJson "...." }}
    <!-- Do stuff on success -->
{{ else }}
    {{ warnf "It failed" }}
{{ end }}

which would fail since the getJson still raised an error.

@onedrawingperday do you know if a warning raised by getJSON will still trigger the else block of that construct, or would it be considered success enough and run the with block?

[onedrawingperday]

@Lnaden

I don't know the answer to your question and cannot test it at the moment.

[Lnaden]

For cross reference: #7867

[bep]

There will be no warning/error for that particular error situation if you add this to your config, which means that you need to do any error handling yourself (if you want, e.g.:

{{ with getJson "...." }}
    <!-- Do stuff on success -->
{{ else }}
    {{ warnf "It failed" }}
{{ end }}

[Lnaden]

@onedrawingperday @bep @maelle

I just did a local build with the current master and can confirm that #7867 does fix the problem and the with...else...end construct works correctly.

For anyone looking for this solution in the future for some version after 0.76.5

• Add ignoreErrors = ["error-remote-getjson"] to your config file
• Trap getJSON calls you expect to fail with the following

  {{ with getJson "...." }}
      <!-- Do stuff on success -->
  {{ else }}
      <!-- Do stuff on failure -->
  {{ end }}
  

Thanks to everyone who replied and thanks to @bep for implementing a fix.

[Lnaden]

I know its not a fix-fix, and the implementation was not meant for this problem, but it does work for this use case and does produce the behavior I was asking for, so I'll count it as a fix.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.