[gatsby-plugin-feed] Allow additional feed options #3413

mikefowler · 2018-01-04T17:28:04Z

Overview

Refactors gatsby-plugin-feed so that the query option handles the bulk of configuration.

Basic configuration

Scenario

I want an RSS feed that includes all of my Markdown files, and I don't need any custom configuration.

Configuration

gatsby-plugin-feed must be listed in gatsby-config, and all MarkdownRemark nodes must have a slug field assigned to them.

{
  resolve: `gatsby-plugin-feed`,
}

Result

A feed is generated at /rss.xml. It includes all edges in allMarkdownRemark, sorted by frontmatter date, in descending order.

Custom feed options

Feeds may specify their own query option, the result of which will be deeply merged with the default query. As such, custom queries need not replicate the entire query, but can override specific keys as desired.

Scenario

The default set of nodes included in the feed work fine for me, but I need to customize some feed options.

Configuration

The same configuration as described under [Basic Configuration][#basic-configuration), but with a custom query.

{
  siteMetadata: {
    title: 'Default Feed Title',
    customFeedTitle: 'Custom Feed Title',
  },
  plugins: [{
    resolve: `gatsby-plugin-feed`,
    feeds: [{
      query: `
        {
          site {
            siteMetadata {
              title: customFeedTitle
            }
          }
        }
      `,
    }],
  }],
}

Multiple feeds

Scenario

The default feed doesn't work for me. I need more than one, and each will include different nodes.

Configuration

{
  siteMetadata: {
    title: 'Default Feed Title',

    audioFeedTitle: 'Audio RSS',
    audioFeedDescription: 'Duis mollis, est non commodo luctus, nisi erat porttitor.',

    journalFeedTitle: 'Journal RSS',
    journalFeedDescription: 'Cras mattis consectetur purus sit amet fermentum.',
  },
  plugins: [{
    resolve: `gatsby-plugin-feed`,
    feeds: [{
      output: 'audio.xml',
      query: `
        {
          site {
            siteMetadata {
              title: audioFeedTitle
              description: audioFeedDescription
            }
          }
          entries: allMarkdownRemark {
            # Only nodes returned in the entries key will be
            # included in the feed.
          }
        }
      `,
    }, {
      output: 'journal.xml',
      query: `
      {
        site {
          siteMetadata {
            title: journalFeedTitle
            description: journalFeedDescription
          }
        }
        entries: allMarkdownRemark {
          # Only nodes returned in the entries key will be
          # included in the feed.
        }
      }
      `,
    }],
  }],
}

Addresses #1808

Reviewers

to: @nicholaswyoung
cc: @roundedbygravity

Updates `gatsby-plugin-feed` to accept additional options on a per-plugin or per-feed basis. Available options are documented in [node-rss][1]. Any options specified in `feedOptions` are overriden by matching keys returned by a feed's `query`, if provided. [1]: https://github.com/dylang/node-rss#feedoptions

gatsbybot · 2018-01-04T17:33:46Z

Deploy preview ready!

Built with commit 257dae4

https://deploy-preview-3413--gatsbygram.netlify.com

secretfader · 2018-01-04T17:46:24Z

Thanks for jumping on this!

I would rather use GraphQL data to populate options for the RSS module. Since we can query for all site options, what about a standardized GraphQL query instead of passing a separate feedOptions parameter? Then, anything out of the normal parameters can be considered { ...extras } for the RSS module?

For instance, what if options.query was:

{
  site {
    siteMetadata {
      title
      description
      siteUrl
      site_url: siteUrl
    }
  }
  entries: {
    // edge and node fields
   }
}

That represents the simplest configuration possible, and we could easily derive everything from that single option. The output would be /rss.xml by default (which I think is the current behavior).

Additionally, you could override that query exactly to gain additional options. Like so:

{
  resolve: 'gatsby-plugin-feed',
  options: {
    feeds: [
      output: '/test.xml',
      query: ``
    ]
  }
}

I think we can get a lot of what we want by simply standardizing the query, then making sure the context passed into the setup and serialize functions is a deep merged version of both queries' data, should more than one query exist. This is actually the issue at the heart of the plugin's malfunction.

Does this make sense? If so, what do you think? Aliasing our entries query to be a specific key would seem to get us a lot of the way. Then, all we need to do is make sure the setup/serialize context (when looping through individual feed configuration objects) is properly merged.

secretfader · 2018-01-04T22:26:00Z

I've pushed some ideas and referenced them in #2820.

As for #1808, I never got a repro case. It should be considered closed.

mikefowler · 2018-01-04T22:27:15Z

@nicholaswyoung for sure, that makes sense! I was wondering if that was the direction you were heading.

In the case of something like the generator option, is your thinking that the key should just exist in the query results, whether in siteMetadata or as a top-level key in another query?

{
  site {
    siteMetadata {
      title
      description
      siteUrl
      site_url: siteUrl
      generator
    }
  }
  someOtherQuery {
    generator
  }
}

If that's the case I'll just amend this PR to include better instructions on how to pass those extra options, because I believe this should already work.

secretfader · 2018-01-04T22:46:22Z

@mikefowler The site > siteMetadata > fields hierarchy is standard across most Gatsby sites. That's where I'd store all of the global configuration, but it could be handy to offer overrides.

I've been thinking much about how to make configuration incredibly simple. I could see users wanting to override on a per-feed basis (for instance, in the case of a podcast or different media type), so maybe we allow them to override like so?

feeds: [
  {
    query: `{
      site {
        siteMetadata {
          // pull in fields you need to configure or remap for this rss feed
        }
      }
      entries: allMarkdownRemark() {} // for example, the only thing that matters is the key, "entries"
    }`,
  }
]

Then, when merged with the top-level query, any duplicated fields are overridden by local results. And honestly, I'm leaning towards dropping the query in our context object (which is passed to setup / serialize) entirely. I mean, the query + options are all we really need in there.

I'll amend this PR with a better commit message before merging.

mikefowler · 2018-01-06T06:53:14Z

@nicholaswyoung: I took the ideas you outlined in https://github.com/uptimeventures/gatsby/commit/50368c3d76025619f277fbc37da7452a48044573 and jammed on them for a while tonight. The most recent commit on this branch has the plugin in a working state for generating multiple feeds using queries alone. The default config is No Config™, and adding a second (third, fourth, etc) feed looks something like this:

module.exports = {
  siteMetadata: {
    title: 'My Website',
    fireAntsTitle: 'Fire Ants!',
  },
  plugins: [
    // ...
    {
      resolve: 'gatsby-plugin-feed',
      options: {
        feeds: [{
          // This will use the default entries & metadata
          output: 'rss.xml',
        }, {
          output: 'fireants.xml',
          query: `
            {
              site {
                siteMetadata {
                  title: fireAntsTitle
                }
              }
              entries: allMarkdownRemark() {
                # Etc, etc.
              }
            }
          `,
        }],
      },
    },
    // ...
  ],
};

secretfader · 2018-01-06T18:19:28Z

@mikefowler That's exactly the direction I was thinking of going. What do you think? Does it accomplish the goal of feeling natural when overriding global config with local options? Looking over that branch, it really does feel right.

I just need to flesh out the logic for automatic configuration, since it'd be nice to have media frontmatter translated into <enclosure> tags by default, plus make iTunes and other advanced tags a simple boolean. (By default, anyway. I think we should still allow passing custom setup/serialize callbacks or some other mechanism of customization.) I don't think I did any of that in my initial PR.

The feed plugin has become a popular addition to most Gatsby sites, but I didn't do the best job designing the options API in our first release. This PR is very close to what I envisioned.

secretfader · 2018-01-06T18:21:54Z

packages/gatsby-plugin-feed/src/serialize.js

@@ -0,0 +1,16 @@
+export default function serialize({


I'm inclined to think that even though the query format is standardized, we should still include a way for users to customize feed options via a callback.

secretfader · 2018-01-06T18:23:03Z

packages/gatsby-plugin-feed/src/gatsby-ssr.js

    ...defaultOptions,
    ...pluginOptions,
  }

-  const links = feeds.map(({ output }, i) => {
+  const links = options.feeds.map(({ output }, i) => {


Just curious: is this a style change only, or was the other implementation harming performance?

Ah, just stylistic. I'm going to revert these changes because they're just adding noise to the diff.

secretfader · 2018-01-06T18:28:26Z

packages/gatsby-plugin-feed/src/createFeed.js

+  }
+
+  // @TODO: is this useful? We should think of an example that would use this.
+  const context = defaultsDeep(options, rest)


The original intent of context was to provide a shared object, passed to functions that created feed options or serialized nodes to entries. This allowed every parameter the option of being dynamic: either retrieved from the query data or specified options.

I think the idea is sound, though the prior implementation certainly wasn't. See the comment below on serialize. I think we need to retain some of the ideas from context, by allowing users to pass in their own setup and serialize functions in addition to our smart defaults.

secretfader · 2018-01-06T18:37:16Z

packages/gatsby-plugin-feed/package.json

@@ -23,7 +23,9 @@
  },
  "dependencies": {
    "babel-runtime": "^6.26.0",
-    "lodash.merge": "^4.6.0",
+    "deepmerge": "^2.0.1",
+    "lodash.defaultsdeep": "^4.6.0",


I think we can get away with not including lodash. I know there's a gatsby plugin that integrates the babel plugin, but as a general rule, I'm trying to stay away from pulling in extras that I don't need. deepmerge is probably the most important, since most other config is dealt with through object rest spread.

@KyleAMathews Do you have an official policy regarding lodash in gatsby plugins? Object rest spread seems to do most of what is needed here, and deepmerge fills in the gaps. But if it'd be better to rely on lodash if it's included in several plugins (and perhaps creating a smaller build), I could see the case for using it.

@nicholaswyoung Most recent commit drops both of the lodash packages. I replaced defaultsDeep with deepmerge and reversed the argument order, which I think achieves the same result.

secretfader · 2018-01-06T19:01:37Z

packages/gatsby-plugin-feed/src/createFeed.js

+}
+
+export default function createFeed({
+  query: {


I have questioned if there's a reason why query data shouldn't just be merged in with other parameters. Does it need to live under the key query when most of the data is being derived from that object? I don't care either way, It's just another potential simplification I thought of.

mikefowler · 2018-01-06T21:25:22Z

Thanks for the feedback, @nicholaswyoung!

I've been trying to work backwards from my ideal metadata configuration for managing multiple feeds, and I think my ideal is something like the following, but I'm having a hard time figuring out how that would square with the direction we're heading.

{
  siteMetadata: {
    title: 'Website',

    audioFeed: {
      title: 'Audio',
      // Any additional options get passed to node-rss
    },

    journalFeed: {
      title: 'Journal',
    },
  }
}

It may be the engineer in me that doesn't like the idea of storing related feed options, for separate feeds, as sibling keys in the same object, like this:

{
  siteMetadata: {
    title: 'Website',

    audioFeedTitle: 'Audio',
    journalFeedTitle: 'Journal',
  }
}

This doesn't seem so bad at the outset, but with 5-10 options for each feed (in an advanced configuration, say), I would much prefer the first example. Thoughts?

secretfader · 2018-01-06T22:09:26Z

@mikefowler To address that scenario, I currently return a setup function in my config that returns options for each feed. That's one pattern (and it works with the current implementation), though I'm sure it can be improved upon.

I'm open to changing anything that makes configuration easier to reason about. This is one of the hard spots everyone has gone through with this plugin.

mikefowler · 2018-01-06T23:28:33Z

@nicholaswyoung pushed one more commit adding the serialize and setup methods back. As implemented this would be a breaking change, because it seems to make more sense to just pass the result of the query to serialize and setup, rather than the entire set of options. You have more practical experience with setting up feeds using this plugin, though, so let me know if you don't think that's quite right.

secretfader · 2018-01-06T23:44:20Z

@mikefowler That's exactly what I think is right.

Even though it's a breaking change, there's really no reason to keep query separate from other options.

agarrharr · 2018-01-10T20:51:01Z

Would this allow for multiple rss feeds on one site? And would it allow you to customize it to work for a podcast feed? I'm trying to figure out how to do this right now: orbitfm/orbit.fm#6

secretfader · 2018-01-11T00:44:40Z

@agarrharr It currently supports multiple feeds per site. Take a look at the configuration options I linked above. If you need help beyond that, I'll gladly answer specific questions. I run multiple feeds on my site, so I know this is easily within reach. :)

mikefowler · 2018-01-11T22:11:11Z

@nicholaswyoung I'll have some time this weekend to work on the media integration, but if you have any work-in-progress, or want to take a stab at it yourself, feel free to push commits to this branch!

Alternatively: I could temporarily revert to the previous implementations of serialize and setup, so as not to warrant a breaking change, release this branch as a patch version, and then release a followup major version bump that breaks serialize and setup, and adds media support. Thoughts?

secretfader · 2018-01-12T01:38:31Z

@mikefowler I've been sick and/or out most of the week thus far, so I haven't had a chance to dig in. I'm going to pull this branch and take a look tomorrow when my head is clearer, but I have a hunch we can easily wrap it up. (Side note, sorry that the links to my repo above are now dead.)

I saw another config change that was suggested by @agarrharr in #3489, but I think there has to be a cleaner API for generating dynamic feeds. I'm hesitant to make every configuration option an optional function callback, mostly because I'm concerned about further complicating the configuration object.

I'm using the plugin to generate podcast feeds, too, but the implementation is incredibly hacky. The number one question I get is concerning how to configure the plugin, so that's obviously the most pressing issue. If I can clean up the implementations of everybody's RSS generation workflow all at once, rather than landing a bunch of potentially bothersome changes... that's my goal.

agarrharr · 2018-01-12T02:57:49Z

Thanks @nicholaswyoung. I'm very interested in figuring out a cleaner API for generating dynamic feeds.

KyleAMathews · 2018-02-09T00:10:41Z

@mikefowler any chance you could finish this up soon? Lot of great fixes here! :-)

mikefowler · 2018-02-09T00:23:22Z

@KyleAMathews, yea, it's still on my radar! My Gatsby contributions seem to be coming at the speed that I'm able to work on converting my website over from Jekyll :P

I think this is quite close to being ready to ship. For sake of fixing the original confusion that this work stemmed from, we can add the new functionality (media/podcast feeds, for instance) in a separate changeset.

secretfader · 2018-02-09T04:18:11Z

I've been trying to find time to wrap this up, too. I've just been pulled in too many directions.

After repeatedly considering what should be the heuristic for determining media entries from regular entries, I keep coming back to some version of the configuration we currently have. I don't know if a zero (or lightweight) configuration option is sensible after all.

KyleAMathews · 2018-02-09T04:43:54Z

It's tough because you can make a feed from any data.

agarrharr · 2018-02-09T16:15:17Z

It is tough. This is what I ended up changing in this plugin to make it work for me: orbitfm/orbit.fm@4bdfb2c#diff-5c11e5724db58b8e0e82c5472c465be4R41-64

But then my config for this plugin is huge! https://github.com/agarrharr/orbit.fm/blob/master/gatsby-config.js#L5-L239

pieh · 2018-06-07T15:23:31Z

@mikefowler It'd be great to get this merged if you have the chance to update it? In order to clean up the PR list I'll close hanging PRs like this over the next few days. But it's always better to merge a PR, so drop a comment here if you'd like to keep this open :) Or feel free to re-open this later.

m-allanson · 2018-06-27T13:30:59Z

Closing as this seems inactive. Please open a new PR if you'd like to pick this up again.

Let query option do the heavy lifting

88b2136

I'll amend this PR with a better commit message before merging.

secretfader reviewed Jan 6, 2018

View reviewed changes

secretfader mentioned this pull request Jan 6, 2018

Revert #2571 #2820

Merged

mikefowler force-pushed the mrf--allow-additional-options branch from 850d3e5 to f987baf Compare January 6, 2018 21:57

Drop lodash dependency

859a104

mikefowler force-pushed the mrf--allow-additional-options branch from f987baf to 859a104 Compare January 6, 2018 22:07

Reimplement setup and serialize methods

257dae4

secretfader mentioned this pull request Jan 11, 2018

Gatsby Feed Plugin and Feed Example Issues #1808

Closed

secretfader mentioned this pull request Jan 12, 2018

gatsby-plugin-feed feature request - dynamic feeds #3489

Closed

KyleAMathews mentioned this pull request Feb 21, 2018

[gatsby-plugin-feed] Document slug fields creation #3873

Closed

m-allanson closed this Jun 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[gatsby-plugin-feed] Allow additional feed options #3413

[gatsby-plugin-feed] Allow additional feed options #3413

mikefowler commented Jan 4, 2018 •

edited

Loading

gatsbybot commented Jan 4, 2018 •

edited

Loading

secretfader commented Jan 4, 2018 •

edited

Loading

secretfader commented Jan 4, 2018

mikefowler commented Jan 4, 2018 •

edited

Loading

secretfader commented Jan 4, 2018 •

edited

Loading

mikefowler commented Jan 6, 2018

secretfader commented Jan 6, 2018 •

edited

Loading

secretfader Jan 6, 2018

secretfader Jan 6, 2018

mikefowler Jan 6, 2018

secretfader Jan 6, 2018 •

edited

Loading

secretfader Jan 6, 2018

secretfader Jan 6, 2018 •

edited

Loading

mikefowler Jan 6, 2018

secretfader Jan 6, 2018

mikefowler commented Jan 6, 2018

secretfader commented Jan 6, 2018 •

edited

Loading

mikefowler commented Jan 6, 2018

secretfader commented Jan 6, 2018

agarrharr commented Jan 10, 2018

secretfader commented Jan 11, 2018

mikefowler commented Jan 11, 2018 •

edited

Loading

secretfader commented Jan 12, 2018

agarrharr commented Jan 12, 2018

KyleAMathews commented Feb 9, 2018

mikefowler commented Feb 9, 2018

secretfader commented Feb 9, 2018

KyleAMathews commented Feb 9, 2018

agarrharr commented Feb 9, 2018

pieh commented Jun 7, 2018

m-allanson commented Jun 27, 2018

[gatsby-plugin-feed] Allow additional feed options #3413

[gatsby-plugin-feed] Allow additional feed options #3413

Conversation

mikefowler commented Jan 4, 2018 • edited Loading

Overview

Basic configuration

Scenario

Configuration

Result

Custom feed options

Scenario

Configuration

Multiple feeds

Scenario

Configuration

Reviewers

gatsbybot commented Jan 4, 2018 • edited Loading

secretfader commented Jan 4, 2018 • edited Loading

secretfader commented Jan 4, 2018

mikefowler commented Jan 4, 2018 • edited Loading

secretfader commented Jan 4, 2018 • edited Loading

mikefowler commented Jan 6, 2018

secretfader commented Jan 6, 2018 • edited Loading

secretfader Jan 6, 2018

Choose a reason for hiding this comment

secretfader Jan 6, 2018

Choose a reason for hiding this comment

mikefowler Jan 6, 2018

Choose a reason for hiding this comment

secretfader Jan 6, 2018 • edited Loading

Choose a reason for hiding this comment

secretfader Jan 6, 2018

Choose a reason for hiding this comment

secretfader Jan 6, 2018 • edited Loading

Choose a reason for hiding this comment

mikefowler Jan 6, 2018

Choose a reason for hiding this comment

secretfader Jan 6, 2018

Choose a reason for hiding this comment

mikefowler commented Jan 6, 2018

secretfader commented Jan 6, 2018 • edited Loading

mikefowler commented Jan 6, 2018

secretfader commented Jan 6, 2018

agarrharr commented Jan 10, 2018

secretfader commented Jan 11, 2018

mikefowler commented Jan 11, 2018 • edited Loading

secretfader commented Jan 12, 2018

agarrharr commented Jan 12, 2018

KyleAMathews commented Feb 9, 2018

mikefowler commented Feb 9, 2018

secretfader commented Feb 9, 2018

KyleAMathews commented Feb 9, 2018

agarrharr commented Feb 9, 2018

pieh commented Jun 7, 2018

m-allanson commented Jun 27, 2018

mikefowler commented Jan 4, 2018 •

edited

Loading

gatsbybot commented Jan 4, 2018 •

edited

Loading

secretfader commented Jan 4, 2018 •

edited

Loading

mikefowler commented Jan 4, 2018 •

edited

Loading

secretfader commented Jan 4, 2018 •

edited

Loading

secretfader commented Jan 6, 2018 •

edited

Loading

secretfader Jan 6, 2018 •

edited

Loading

secretfader Jan 6, 2018 •

edited

Loading

secretfader commented Jan 6, 2018 •

edited

Loading

mikefowler commented Jan 11, 2018 •

edited

Loading