Display extra information for packages hosted in GitHub

[richorama]

The ProjectURL could be parsed, and used to determine if GitHub is used to host the package's source code.

Several things could then be done:

• Add a link on the left hand side of the package page to the source code and issues in GitHub
• Pull through the number of stars, open issues and PRs, as an indicator of project health.
• Display the readme.md file from github
• Detect other packages authored by the same GitHub user.
• Display the date the code was last updated (indicates that the package is behind)
• Display the programming language used to author the package

These ideas are implemented on http://nugot.net

[maartenba]

On NuGot, are you storing these or fetching on the fly?

[richorama]

Storing.

When the background process indexes all the packages, it detects a github link in the project URL. It then adds a message to an azure queue.

The github API is then called 3 times:

1. retrieve the repo information
2. retrieve the readme markdown (call 1 gives you the url).
3. ask github to convert the markdown to html.

The HTML is then stored as an azure blob.

The GitHub API is rate limited, so the process backs off and waits when the limit is approached.

If the repo has been moved (i.e. new owner or rename), the API gives you a redirect response, so there's a little bit of nuance.

I also have a list of popular packages that are on GH, but the URL isn't set to GH.

There is a reasonably high percentage (can't remembre the precise number) of packages that have GH repos, so I think it's worthwhile.

Happy to share this code if that helps (you could probably lift it an run it as is to create the blobs).

[yishaigalatzer]

@richorama do you want to start a PR.

We can break it down to backend - Grab the data etc.
UI - Where we would probably need to grab a UI designer to work on the layout.

A basic PR would help assess the cost of taking this to production. We can also probably chat with GH on perhaps allowing less rate limiting for nuget.org

[maartenba]

Also interested in learning when you update the readme info? Open to exploring this for NuGet.org. I guess this would need a back-end job as well? Or could we do this on front-end? (and when rate limit is reached, just have the current NuGet behaviour where we don't fetch the GitHub info)?

[richorama]

Great! Would a node.js app be an acceptable starting point for the background process? It could be rewritten in c# later...

Sent from my iPhone

On 16 Mar 2016, at 01:45, Yishai Galatzer [email protected] wrote:

@richorama do you want to start a PR.

We can break it down to backend - Grab the data etc.
UI - Where we would probably need to grab a UI designer to work on the layout.

A basic PR would help assess the cost of taking this to production. We can also probably chat with GH on perhaps allowing less rate limiting for nuget.org

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[maartenba]

I think we can probably start off of a node.js one. Perhaps we can first look at the current implementation and then look at how to light it up in NuGet.org?

[304NotModified]

Is iframing the readme not a possibility?

[richorama]

@304NotModified I don't think that would look very nice. It wouldn't help from an SEO perspective either.

@maartenba I haven't given up on this, I've just be busy, and on a few wild goose chases with the bitbucket libraries I've found. I think I might just call the API directly...

[maartenba]

@richorama Keep us posted! I tried the current version with GitHub and think it's great so far!

[harikmenon]

Thanks @richorama. This is pretty awesome. Some minor feedback. Disclosure, havent built this on my end, just looking at your screenshot:)

• We probably should not display the readme.md in the first version, there are a number of projects, where readme.md is tied to showing how to build and run the project instead of how to use the API. This is probably something we dont need to show directly in nuget.org
• Instead of showing GitHub as it is, we should do something similar to how cocoapods does it with a logo and hyperlinked project name. E.g https://cocoapods.org/?q=network.

[maartenba]

My take is we should show it as-is :-) (just as in the screenshot)

[harikmenon]

I'm fan of having concise information dense info as much as possible, especially given the fact the we have around 12 lines of info on the left pane. For new items we are adding to the page, we should consider tightening it up so that you dont have to vertically scroll :)

[skofman1]

@jonwchu ?

[akamud]

Doing something similar to what Marketplace does would be a great MVP.

[magol]

In addition to readme.md, also read changelog.md.
If it find that file, try to parse out the changes for each version.

This require that the file is following the "standard" in http://keepachangelog.com

[cdrnet]

Consider to support other ways of associating packages with (GitHub, ...) repositories, as not all of them host their website on github pages (e.g. none of my projects/packages does). An additional field in the nuspec might be a better idea. Trying to guess this from the ProjectURL is a rather poor design.

[cdrnet]

In addition to readme.md, also read changelog.md.

Wouldn't this be redundant with the release notes in the nuspec (which is typically generated form the change log on package creation - at least that's what I do)?

[richorama]

@cdrnet Just to be clear, the code currently attempts to find the url to your GitHub repo, not GitHub pages.

I think adding repo URL as a first class field in the nuspec is a good idea, perhaps if this field isn't present, the code could fail over to attempting to read it from the project url field, as this is already set for a good proportion of packages.

Trying to guess this from the ProjectURL is a rather poor design.

It's not guessing, it's decoding a URL and seeing if it matches a well known pattern (i.e. http://github.com/USERNAME/REPO)

[cdrnet]

@richorama it is guessing, not decoding, since ProjectURL is semantically not the same as the mainline repository. It is guessing that there is a relationship between ProjectURL and GitHub repository. This assumption is wrong e.g. for all my packages. That's what I mean with poor design. Instead of guessing, we could let packages declare this explicitly in the metadata.

[magol]

@cdrnet

Wouldn't this be redundant with the release notes in the nuspec (which is typically generated form the change log on package creation - at least that's what I do)?

I see it as an alternative easyer way to view a release note. It is hard to use the releaseNotes tag as it require you to manually manage it, or use some script to extract from the changelog.md.

Maybe add a releaseNotesUrl that point to a file with redan release log. That make it more easyer to use, but that should be in a separate issue.

[cdrnet]

Maybe add a releaseNotesUrl that point to a file with redan release log.

That would be great!

[maartenba]

Just FYI, think this issue/PR will be obsolete. Saw the team is working on this: http://blog.nuget.org/20170718/NuGet-Gallery-Gets-A-Facelift.html#package-documentation-integration

[skofman1]

Introducing ReadMe.md support to packages is work in progress, and will be rolled our in the upcoming months. The spec is available here: https://github.com/NuGet/home/wiki/Package-README.md-support

[ffMathy]

@maartenba it is available on NuGet.org, but not from the commandline yet.

[skofman1]

@ffMathy , adding markdown documentation to the package is work in progress. See this announcement:
NuGet/Announcements#29