feeds-rs on nfl.com contains all you could possibly ask for

[mickeelm]

So after some friendly help over at Reddit, turns out that what I posted in #28 really was only the tip of the iceberg. Not even that.

http://www.nfl.com/feeds-rs?_wadl

If you check that link, you'll get all the endpoints that are under feeds-rs. There are dozens and dozens more. Just some examples:

#The 1983 NFL teams (goes back to 1970)
https://feeds.nfl.com/feeds-rs/teams/1983.json

#The current roster of the Arizona Cardinals
https://feeds.nfl.com/feeds-rs/roster/3800.json

#All draft picks in the 4th round of 2015
https://feeds.nfl.com/feeds-rs/draft/byRound/4/2015.json

#Player stats on Khalil Mack (one of the many player-related endpoints, depending on what kind of info you want and how you want it presented/filtered)
https://feeds.nfl.com/feeds-rs/playerStats/2543463.json

#Player transactions for the Denver Broncos in september:
https://feeds.nfl.com/feeds-rs/transactions/team/1400/2018/9.json

#Colts@Patriots 2 weeks ago
http://www.nfl.com/feeds-rs/boxscorePbp/2018100400.json

And the list goes on, and on, and on. You can get info on officials, blog posts, free agency...

In the way nflgame is used now, this doesn't really change things drastically. There are things that could be simplified though, especially the update-player logic which is pretty advanced and relies on web scraping. So that would benefit from this.

I gladly admit when I'm wrong, I think it is safe to say that the NFL does have an API (apart from api.nfl.com), this is way more than a few "accidental" feeds.

[JimHewitt]

This is great, thank you.

[derek-adair]

Fantastic work. I think we should start planning out how we can leverage this.

I'm, unfortunately, quite busy trying to keep the lights on, so to speak so i've still been unable to contribute much lately. I would personally like to see these in place for next season.

Have you put any thought into how this may look @mickeelm

[mickeelm]

No worries, I'm not sure I'm gonna be able to contribute anything here for a long time, a lot of other stuff going on.

Just to be clear, the links above was just a few out of zillions of examples of the kind of data you can get, I just picked a few randomly but the amount of endpoints is HUGE. I haven't even had time to check all the endpoints because they're so many and I think that to sum it up - whatever you can see on NFL.com, you can get from the API (or whatever we decide to call it), and possibly even more than that.

So "to see these in place for next season" really depends on what you mean. For instance just simplifying/solidifying the update-player-logic and/or investigating if for instance #34 can be solved by using another feed then it sounds reasonable. Making use of the whole API requires a full-paid project team of x members 🙂

But yeah, the doors are definitely open if one wants to develop nflgame further with new functionality. I haven't really thought of what that could be myself though.

[doctor3w]

I would love to help out with updating nflgame to use these resources (as well as updating to python3). Let me know if there's anything I can do.

[derek-adair]

If you have any suggestions on how one or both of these may be accomplished I'm all about it. I'm unfortunately having to focus my time on keeping food on the table.

Let me, @ochawkeye or @mickeelm know if you have any questions.

[doctor3w]

I'll take a look at the update player and schedule, see if there are any obvious ways to leverage this for simplification.

[derek-adair]

Hopefully this will also increase accuracy and prevent issues like #55. I am assuming it is a more pristine data source... time shall tell!

[mickeelm]

I don't think that these resources are more reliable than the ones nflgame uses right now but just to take an example - scheduling. IMO, nflgame should simply use https://feeds.nfl.com/feeds-rs/schedules/2018.json (insert 2019 next season, or just omit the parameter to get the current season), plain and simple. This contains the whole schedule for this season, and changes throughout the season when games are moved. The logic today in nflgame is built during other circumstances, where only one week at a time could be fetched. This is a limitation that no longer needs to be. So issues like the one mentioned above can easily be removed by simply removing a lot of now unnecessary logic and just fetching the whole schedule every now and then. Of course, if NFL indeed was feeding the wrong data, then there's nothing we can do about it, regardless of data source.

As mentioned before but as another example, player logic relies on web scraping. As sophisticated and great it may have been written during different circumstances with different conditions, it is no longer necessary to have this kind of advanced logic and it requires many more HTTP calls than necessary (which have been up to debate before).

As for my own contribution, I'm pretty swamped with other stuff so I'm not gonna take the lead here (also, in my application I just go straight towards these endpoints, so I don't use nflgame personally, I just like the project). But, I'll be glad to help out when I can if you want me to, when specific tasks/questions pop up.

[doctor3w]

@mickeelm it sounds like it could be a fairly large restructuring of the update schedule and player logic. I'll take a whack at it, although I haven't done much on nflgame myself so I'll have to read through and try and understand exactly how it's working. I'll be sure to stop back with any questions.

[derek-adair]

It's worth noting that there is a TON of really useful comments in the __init__.py

[derek-adair]

Working on this now.

It seems /playbyplay/{eid}.json is what we want. Its the same or similar data but formatted differently.

Old
New

[doctor3w]

@derek-adair Are you integrating this into this repo? Or are you working on it for a new project?
I attempted adding some of these into this library, but I began work on a Swift version instead of updating this one.

[derek-adair]

I plan on integrating parts of this into this project, however, I'm a bit hesitant to do so w/o a really good reason... in hindsight i'm not 100% sure its worth it.

I've committed an update that uses this to grab the current week/season state so far,.. which was absolutely worth it. I'm open to any suggestions on how to move forward on this.

[degravek]

Hey guys, just wondering if anyone has experience pulling weather data from the feeds.nfl.com/feeds-rs/ api. It looks like there are /weather/... endpoints, but I can't extract the data. Any help is much appreciated.

[mickeelm]

Hmm, maybe they're not used. Given how most other endpoints work, for instance the request https://feeds.nfl.com/feeds-rs/weather/byGame/2015080951.json "should work".

Also, the request: https://feeds.nfl.com/feeds-rs/weather/byWeek does return <gameWeathers/> (or an empty array in the JSON equivalent), and so does for instance https://feeds.nfl.com/feeds-rs/weather/byWeek/2019/REG/15.

My hypothesis is that data simply isn't loaded to them. Given the big amount of endpoints, it might be that weather data is provided somewhere else?

[AccidentalGuru]

Some of the links give this error:

http://feeds.nfl.com/ is deprecated. Please use https://api.nfl.com/ moving forward. Credentials are required, please reach out to [email protected] to request them.

I mentioned that I emailed for credentials in #112, and will post results.

[mickeelm]

That's really sad news. We'll see for how long the ones that work, will continue to work. Interesting that they put the email-address in there, I haven't even been able to find an email before (on api.nfl.com). I just sent them a few lines too.

[JimHewitt]

It should be http://www.nfl.feeds/... instead of http://feeds.nfl.com/...

[derek-adair]

Ya the www is required, but this is still an interesting development. Which endpoints where you probing?

This is actually good news IMO. ALTHOUGH it's reported to be a dud anyways.

EXPENSES BE DAMNED! I would be more than willing to set up a simple proxy for us all to hit that will forward requests... assuming that we are able to leverage the "official" api, LEGALLY and free.

[JimHewitt]

The endpoints all seem to be working, however as mentioned before not all return data (weather, depth charts to name a few). PBP, Rosters, Teams and Schedule all seem to work. Also, Transactions seem to work, but not sure how often they're updated.

[derek-adair]

@AccidentalGuru which endpoints were you getting these messages on?

[AccidentalGuru]

I couldn't find which endpoint it was in my history, or they may have tightened things up as sadly I got this response:

"Thanks for reaching out. As of Feb. 18th, the NFL has secured the APIs and will only grant access to contractual partners. Unfortunately, if you are NOT a NFL Partner you will no longer have access to the feeds or able to obtain access to this feed."

Bummer, though I told them I'm available to the right bidder...

[derek-adair]

Would you mind emailing me the transcript please, its worth looking at to know where we stand on this project in general.

[mickeelm]

For clarity, as @JimHewitt pointed out. If you just do http://www.nfl.com/feeds-rs and not https://feeds.nfl.com/feeds-rs, they all work as before. Only https://feeds.nfl.com/feeds-rs gives the deprecation message.

Then of course, the http://www.nfl.com/feeds-rs may just stop working at anytime as well - we don't know.

[derek-adair]

Ya this really highlights the need to simplify how the data is pulled. If we start to normalize how this is done and treat this project as a "wrapper" it will allow us to adjust easier (e.g. feeds-rs closes).

For what it's worth I am confident that the NFL is not concerned about any individual, non-commercial use of the data we are scraping. If you are using this data for commercial use you are likely breaking their terms (which are not to be found anywhere). You can see how the MLB feels about this.

tl;dr; I'm comfortable driving this project forward in that regard and more tightly coupling this project to official nfl apis. Just be warned nflgame is very likely illegal to use for commercial use.

[derek-adair]

Oh how wrong I was to count on feeds-rs.

[mickeelm]

I mean - nflgame has always existed under this threat, that the data sources might vanish at any time. There is nothing specific about feeds-rs, the other endpoints/feeds used to feed nflgame are also gone.

So I don't really see why "counting on feeds-rs" was/would have been (not very much integration has been done anyway, has it?) a bad decision - what was the alternative?

[derek-adair]

1. I was wrong; the nfl cares about the number of requests being sent
2. I was wrong to more closely tie this project to the specific api

The idea going forward is to reverse both of these decisions.

[mickeelm]

Sounds great - although for the record/for what it's worth I don't think your decisions/assumptions were bad at all - they were perfectly sound given the reality that we lived in up until 4-5 days ago. I'm comfortable you'll make good decisions moving forward too!

[brbeaird]

Yeah I agree - it sucks, but don't be hard on yourself. This kind of stuff happens all the time with these unofficial integration projects.