Skip to content

Latest commit

 

History

History
845 lines (658 loc) · 28 KB

Git_Workflow.md

File metadata and controls

845 lines (658 loc) · 28 KB

Git Workflow

To help get everyone up to speed on the way we use version control, specifically Git on projects like MPICH, we've put together this Wiki page as a reference. Some of the content here may be remedial, but please go through all of it to make sure you're contributing code in the way that everyone else on the team expects. Useful resources on learning the basics of our current VCS, git, can be found below.

Committing

Below you can find information on what is required for creating a commit in MPICH.

Commit Content

Coding Standard

MPICH has a coding stadard that should be followed for all newly added code, along with updating any old code.

One Idea Per Commit

  • Often commits are edits to a single file and only a few lines. Keeping them around this size makes automated testing much easier. If your commit includes features A & B and we have to roll back to fix B, we don't want to lose A as well.

  • Code refactoring should be separate from everything else. If you need to change API calls, that should be in a completely separate commit.

Don't Pollute Code That Isn't Yours

  • Try to make the least intrusive version of your change as possible. Don't try to fix things like whitespace in the same commit as code changes. If it's absolutely necessary to reformat existing code to make it more readable, then it should be done in a separate commit. Not doing this makes it hard to separate out who actually wrote the code.

Commit Message

Title

This should be in the format of <Codebase Section>: <One Line Description>. This should be no longer than 50 characters.

Description

This section should contain a description of what the changes in the commit are actually doing. This could be as small as a single line or paragraph. The most important part is the description is concise and to the point. Descriptions should wrap text at 72 characters. Below are some tips for writing commit descriptions.

  • Use a blank line between the first line and the rest of the description
  • Write a complete description of the code change in the body.
  • Use correct punctuation, spelling, and cApItaLizAtIOn
  • Relevant Blog Post

Example

Below is an example commit:

Docs: Mediawiki Pages Conversion

The pages for the wiki.mpich.org site were converted to match
the format for github wiki pages, where the site will be migrated
to due to end of life of the mediawiki at ANL.

Using Git

Important URLs

Type URL
writeable clone URL [email protected]:pmodels/mpich
read-only clone URL (via git protocol) git://github.com/pmodels/mpich
read-only clone URL (via http) http://github.com/pmodels/mpich

IF YOU CANNOT ACCESS THE GITHUB REPO BUT THINK YOU SHOULD BE ABLE TO, CONTACT [email protected] SO THAT WE CAN ADD YOU AS A COLLABORATOR

Local Environment Setup

If you do not have the actual git tool, you can get that here or your preferred software package management system (brew, apt, etc).

The next step is to add the following (substituting your name and email) into your ~/.gitconfig file:

[user]
    name = Joe Developer
    email = [email protected]

[color]
    diff = auto
    status = auto
    branch = auto
    ui = auto

# optional, but helps to distinguish between changed and untracked files
[color "status"]
    added = green
    changed = red
    untracked = magenta

# optional, but allows git to create 8-character abbreviated hashes, that are 
# "trac-compatible" for automatic link generation in the comments.

[core]
    whitespace = trailing-space,space-before-tab,indent-with-tab
    abbrev = 8

# optional, but allows better view of commits
[alias]
    graph = log --graph --decorate --abbrev-commit --pretty=oneline

Repository Management

Authorized committers (core MPICH developers) can clone the mpich repository using:

git clone --origin mpich [email protected]:pmodels/mpich --recursive

If you do not have access to the writeable repository but think you should (because you are a core MPICH developer or collaborator), contact [email protected] for access.

Other users can clone the repository using:

git clone --origin mpich git://github.com/pmodels/mpich --recursive

This will create an mpich directory in your $PWD that contains a completely functional repository with all the dependencies and full project history.

You can see the branches it contains using:

git branch -a
* main
  remotes/mpich/HEAD -> mpich/main
  remotes/mpich/main

You can also add other repositories to your local clone. If you have forked the MPICH repo, for example, you can add it using:

git remote add <username> [email protected]:<username>/mpich --fetch

Once this is done, you should be able to see branches from both repositories:

git branch -a
* main
  remotes/raffenet/fix-rma-types
  remotes/raffenet/large-count
  remotes/raffenet/main
  remotes/mpich/HEAD -> mpich/main
  remotes/mpich/main

Any number of such remote repositories can be added as needed.

Dependencies and Submodules

MPICH depends on several external libraries. They are imported as submodules in the MPICH project repository. Below is the list of the dependencies of the current main branch.

Library Repository Location in MPICH source
hwloc https://github.com/open-mpi/hwloc modules/hwloc
json-c https://github.com/pmodels/json-c modules/json-c
Libfabric https://github.com/ofiwg/libfabric modules/libfabric
UCX https://github.com/openucx/ucx modules/ucx
Yaksa https://github.com/pmodels/yaksa modules/yaksa

When using the aforementioned command (git clone) to check out the MPICH code, the --recursive option ensures these dependencies are also checked out and put into their corresponding directories under the MPICH code directory.

If the --recursive option was not used when you initially checking out the MPICH code, you can always manually pull these submodules with

git submodule update --init

The use of submodules allows us to easily manage the external libraries and their versions which MPICH depends on. The MPICH project only needs to store the hashes of these external project instead of keeping the entire copies of these libraries in the MPICH repo. When the a upstream MPICH commit updates these submodules, you will need to run git submodule update --recursive to bring your local copies up-to-date.

Possible Problems

If it's been a long time since you updated, you might see a message like this

git submodule update --init --recursive
error: Server does not allow request for unadvertised object 28872b36f01cc8680abaf1ae0850dec5e0c406de
Fetched in submodule path 'src/mpid/ch4/netmod/ucx/ucx', but it did not contain 28872b36f01cc8680abaf1ae0850dec5e0c406de. Direct fetching of that commit failed.

That suggests git is trying to find a revision that only exists in our fork. Maybe things got reorganized while you were off doing something else. The following command sequence should get everything back on track.

git submodule sync
git submodule update --init

Changes and Commits

Let's see an example on how to make changes to the code base and commit them in git.

Edit an existing file (foo.c) and a new file (bar.c):

vim foo.c   # edit an existing tracked file
...
vim bar.c   # edit a new file
...

Check the status of your local git clone:

git status
# On branch main
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   foo.c
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       bar.c
no changes added to commit (use "git add" and/or "git commit -a")

git status -s
 M foo.c
 ?? bar.c

Notice that git knows that foo.c has been modified, but it is not staged for a commit. Git does not know about bar.c and lists it as an untracked file.

You can ask git to start tracking bar.c and stage it for a commit using:

git add bar.c

Once bar.c is added, git status should reflect that bar.c is tracked and staged for a commit.

git status
# On branch main
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       new file:   bar.c
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   foo.c
#

You can also add foo.c to be staged for a commit:

git add foo.c  # This allows git to stage it for a commit

git status
# On branch main
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       new file:   bar.c
#       modified:   foo.c
#

Once all the files to be committed are staged, you can commit them in using:

git commit -m 'fixed bug in foo.c, using new bar.c to do so'
[main f36baae] fixed bug in foo.c, using new bar.c to do so
 1 file changed, 1 insertion(+)
 create mode 100644 bar.c

If you want to commit all tracked files without explicitly staging them, you can use:

git commit -a -m 'fixed bug in foo.c, using new bar.c to do so'

At this stage, you now have created a new commit that is only present in your local repository.

Commit Hashes

Once you have committed changes to your local repository clone, you can view this commit in your commit log:

git log
commit 200039300bd7ad390a1268a08d9a92f13b2a46a5
Author: Pavan Balaji <[email protected]>
Date:   Mon May 6 21:29:46 2013 -0500

    fixed bug in foo.c, using new bar.c to do so

Notice the commit hash on the first line of the log message. This hash represents the actual commit and can be used to reference it.

If you have multiple commits, you can use a single-line format to view the commits without taking up too much space:

git log --format=oneline
200039300bd7ad390a1268a08d9a92f13b2a46a5 fixed bug in foo.c, using new bar.c to do so
1be61fbcad263cffe6dedfb1597be46ccad2b714 Allow f90 mod files to be installed in an arbitrary directory.
3b8afffb24a94858a44c20f6c4fd1ab11838fc1f Add '--with-mpl-prefix=DIR' configure option.
c18320d3188bfc075e04dbcba3e2899451e133ee Fix '--with-openpa-prefix=DIR' so it actually works.
9164d3db668e1f2c0f82cd238599782dfc1fc045 Allow pkgconfig files to be installed in an arbitrary directory.

You can also reduce the commit hash size to a more abbreviated format using:

git log --format=oneline --abbrev-commit
20003930 fixed bug in foo.c, using new bar.c to do so
1be61fbc Allow f90 mod files to be installed in an arbitrary directory.
3b8afffb Add '--with-mpl-prefix=DIR' configure option.
c18320d3 Fix '--with-openpa-prefix=DIR' so it actually works.
9164d3db Allow pkgconfig files to be installed in an arbitrary directory.

Local Branches

Some simple steps first. When you cloned your repository, you have a bunch of remote branches and one local branch (main). Something like this --

git branch -a
  main
  remotes/origin/3.1.x
  remotes/origin/3.2.x
  remotes/origin/3.3.x
  remotes/origin/3.4.x
  remotes/origin/HEAD -> origin/main
  remotes/origin/ecp-ft
  remotes/origin/main
  remotes/upstream/2111_test_ucx
  remotes/upstream/3.1.x
  remotes/upstream/3.2.x
  remotes/upstream/3.3.x
  remotes/upstream/3.4.x
  remotes/upstream/4.0.x
  remotes/upstream/ecp-ft
  remotes/upstream/main

The local "main" branch has a bunch of commits:

git log --format=oneline --abbrev-commit --since='4 hours ago'
6484107e5 (upstream/main, main) Merge pull request #6046 from hzhou/2206_ch4_req
74c35404b ch4: move and rename MPIDIU_request_complete
741e89745 ch4: add MPIDI_REQ_TYPE
7e2ba6876 ch4: remove checking of anysrc_partner in PREQUEST_RECV
c0521cbf0 ch4: remove macro MPIDI_REQUEST_ANYSOURCE_PARTNER
03ab07fff request: move completion_notification into ch4

You can create a new branch called "foobar" from main and checkout this new branch using:

git branch foobar

git checkout foobar

(or git checkout -b foobar for short)

Now, foobar is in the same state as main:

git log --format=oneline --abbrev-commit --since='4 hours ago'
6484107e5 (upstream/main, main, foobar) Merge pull request #6046 from hzhou/2206_ch4_req
74c35404b ch4: move and rename MPIDIU_request_complete
741e89745 ch4: add MPIDI_REQ_TYPE
7e2ba6876 ch4: remove checking of anysrc_partner in PREQUEST_RECV
c0521cbf0 ch4: remove macro MPIDI_REQUEST_ANYSOURCE_PARTNER
03ab07fff request: move completion_notification into ch4

Notice that "foobar" is listed on the top-most hash together with "main".

Now, let's say, you make some changes and commit them into this foobar branch:

git log --format=oneline --abbrev-commit --since='4 hours ago'
cc47a7e3 (HEAD -> foobar) demo change 2 on branch foobar
6484107e5 (upstream/main, main) Merge pull request #6046 from hzhou/2206_ch4_req
74c35404b ch4: move and rename MPIDIU_request_complete
741e89745 ch4: add MPIDI_REQ_TYPE
7e2ba6876 ch4: remove checking of anysrc_partner in PREQUEST_RECV
c0521cbf0 ch4: remove macro MPIDI_REQUEST_ANYSOURCE_PARTNER
03ab07fff request: move completion_notification into ch4

Notice that "foobar" has a new hashes cc47a7e3 that does not belong to "main".

Updating Your Local Clone With Remote Changes

The main MPICH branch can continue to receive chances from various contributors while you are working on your own Pull Request (PR). Before you submit a PR you must apply those changes to your working branch. Below is how to do that:

git checkout main
git pull upstream main
git checkout <Your Branch>
git rebase main
git push -f

Handling Rebase Conflicts

Let's assume we have a conflict on one of the files, foo.c, while performing a rebase.

git rebase upstream/main
First, rewinding head to replay your work on top of it...
Applying: fixed bug in foo.c, using new bar.c to do so
Using index info to reconstruct a base tree...
M       foo.c
Falling back to patching base and 3-way merge...
Auto-merging foo.c
CONFLICT (content): Merge conflict in foo.c
Failed to merge in the changes.
Patch failed at 0001 fixed bug in foo.c, using new bar.c to do so
The copy of the patch that failed is found in:
    /Users/goodell/scratch/git-wiki-example/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

Now we need to resolve the conflict.

vim foo.c
... (search for conflict markers, fix conflict)

git add foo.c

Once all conflicts have been resolved, we simply continue the rebase operation:

git rebase --continue
Applying fixed bug in foo.c, using new bar.c to do so

If you are unable to resolve the conflict and don't want to continue the rebase, you can abort it using:

git rebase --abort

This will take you back to the original setup before you started the rebase.

Pushing Changes Back to the Server

Once all your changes have been locally committed, you have fetched all remote commits and rebased your local commit hashes at the top of the remote branch, you are ready to push your changes to the server.

The general format for pushing changes is:

git push <repo> <local_ref>:<remote_branch>

repo refers to the repository to which you want to push the changes. If you want to push changes to the primary repository you cloned, this will be mpich.

local_ref refers to a local reference which can be a local branch, a local tag, or other non-SHA-1 forms of references (e.g., HEAD, HEAD~3, etc.).

Pull Requests

Changes should be submitted in the form of a "pull request". More info on pull requests can be found here. Some of the more relevant sections for outside collaborators are:

Tips for creating a pull request
- Describe your changes, both in the pull request description and in
  each individual commit message.
    - Describe the underlying issue, whether it be a bug or a new
      feature.
    - Describe the user-visible impact it will have.
- Separate your changes so that there is one logical change per commit.
    - Each commit should stand on its own merits. If a commit depends on
      a previous one, it is helpful to note that in the commit
      message.
    - Each commit in a series should be able to compile and run. This
      way bugs can be traced down using `git bisect` in the future.

Dealing With Development Branches/Repositories

In SVN we had branches in https://svn.mcs.anl.gov/repos/mpi/mpich2/branches/dev, which we would typically refer to as dev/FOO. Many of these branches had restricted permissions, especially when used to collaborate on a research paper or with a vendor. Because of git's distributed nature, it is difficult/impossible to restrict read permissions for a specific branch within a repository. So these development branches that require special permissions are put into their own repositories.

For regular development branches that can be world-readable, a new repository mpich-dev is created. You can add this repository to your local clone using:

% git remote add mpich-dev --fetch [email protected]:mpich-dev.git
Updating mpich-dev
X11 forwarding request failed on channel 0
remote: Counting objects: 213, done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 89 (delta 77), reused 55 (delta 49)
Unpacking objects: 100% (89/89), done.
From git.mpich.org:mpich-dev
 * [new branch]        fix-rma-types -> mpich-dev/fix-rma-types
 * [new branch]        large-count -> mpich-dev/large-count
 * [new branch]        main     -> mpich-dev/main
 * [new branch]        portals-work -> mpich-dev/portals-work

That is, we are actually doing two things:

  • adding a new git "remote" by the name of mpich-dev;
  • fetching its content

You will probably then want to create a local branch to track the remote branch:

git branch fix-rma-types mpich-dev/fix-rma-types
git checkout fix-rma-types

You can now start hacking away on your local fix-rma-types branch.

If you have local branch foobar that you want to create a new branch for in mpich-dev:

git push mpich-dev foobar:foobar

At the current moment we do not have any easy way to list all of the available repositories unless you have permissions to view the gitolite-admin.git repository. If you think you should have access to a particular development branch, contact [email protected] or the specific MPICH core developer with whom you are working. You can list the repositories to which you already have access by running:

ssh [email protected] info
hello balaji, this is git@caveat running gitolite3 v3.2-13-gf89408a on git 1.7.0.4

 R W C  papers/..*
 R W C  u/CREATOR/..*
 R W    dmem
 R W    gitolite-admin
 R W    mpich
 R W    mpich-benchmarks
 R W    mpich-bgq
 R W    mpich-dev
 R W    mpich-ibm
 R W    review

The Pro Git book has more information about working with remotes.

Overall Workflow

Below is a sample workflow for committing changes to the MPICH github repository:

  1. For each new bug fix or feature, create and checkout a new local branch (say ticket-1234) based on upstream/main:
git checkout -b ticket-1234 upstream/main
  1. Make changes as one or more commits on this branch.

In commit messages, if your branch is fixing a ticket, you should use any of the following keywords to make GitHub do magic to automatically attach commits to issues (if your branch is fixing an issue).

Fix <project>#1234
Fixes <project>#1234
Resolves <project>#1234
See <project>#1234
<project>#1234

Where project is: pmodels/mpich

  1. If other users made any changes to the server in the meanwhile, make sure your changes apply cleanly on them. To do this, you need to:
  • Fetch all remote changes:
git fetch --all --prune
  • Move your commits to sit on top of the new changes from other users:
git rebase upstream/main 
// If this detects any conflicts, it'll tell you how to resolve them
  1. Push your changes to your private github fork and open a pull request (PR) to the main MPICH repo. See GitHub docs on how to fork a repo: https://help.github.com/articles/fork-a-repo/

Your PR will automatically be tested with the following tests:

  • contribution-checker
  • mpich-warnings
  • spell-checker
  • whitespace-checker

More tests are available to run, which can be found on our Jenkins MPICH-Review page. To run one of these tests, simply add a comment to your PR in the following format:

test:mpich/ch3/most
test:mpich/ch4/most

This will now run both tests against your PR chances. The results of the review jobs will be added to the status of the PR. Note: only repo admins can trigger Jenkins tests with these phrases.

  1. Assign your PR for someone(s) to review.

  2. The reviewer(s) should review the PR using the GitHub review interface. Comments can be added to specific lines of each file changed, and can request changes to the file before it is accepted. Updating the branch in your github fork will automatically update the PR. Changes should be re-tested and re-reviewed as necessary.

  3. Once the PR is approved by the reviewer(s), an MPICH maintainer will click the button to merge the PR to the target branch in MPICH. Make sure to add Approved-by: <Reviewer(s)> tag in the merge commit message to preserve any review history in the git log.

commit cb944baeb07061759f7c22d85704cc9c673056f9
Merge: f7869f67 14f58e1a
Author: Ken Raffenetti <[email protected]>
Date:   Wed Dec 4 13:18:48 2019 -0600

    Merge pull request #4075 from RozanovAnatoliy/hydra2_timeout

    hydra2: Base implementation of timeout functionality

    Approved-by: Ken Raffenetti <[email protected]>

VCS History

Git

Until January 7, 2013, MPICH used Subversion (SVN) for its version control system (VCS). Now we use git. This page documents important information about the use of git within MPICH.

Why Git?

Git was chosen over another system due to it having arguably the largest slice of the distributed VCS market right now. More of the world will know how to use it to , thus being able to interact with our project. It is also heavily documented as well as having numerous examples and tutorials.

SVN

What has been imported?

Much of the history from our previous SVN repository has been migrated over to git. This includes:

  • All trunk history, with commit messages prefixed by "[svn-rXXXX]". This history lives in the main branch, which is the git convention corresponding to SVN's trunk. The oldest history that was present in SVN was 1.0.6, so that's as far back as the git history goes.
  • All release tags (which are branch-like in SVN), with their history squashed down into a single commit. These commit messages have the format "[svn-synthetic] tags/release/mpich2-1.4.1p1". These commits were then tagged with annotated git tags with names like "v1.4.1p1".

The SVN URL for the original MPICH2 trunk was: https://svn.mcs.anl.gov/repos/mpi/mpich2/trunk. In place of trunk, there were branches and tags. Within those two directories there were also two subdirectories, dev and release. Particular branches and tags were subdirectories of these.

The repository is still operational, however almost all paths have been marked read-only.

Import Process and Caveats

The history was imported by a custom script because the MPICH SVN repository was more complicated than git-svn could handle. Specifically, the use of svn:externals caused a problem. Problem #1 is that git-svn cannot handle any form of SVN external natively. Problem #2 is that our past use of relative SVN externals (e.g., for confdb) was unversioned. This means that svn export -r XYZ $SVN_PATH (nor the pinned-revision variant, @XYZ) would not actually reproduce the correct working copy at revision XYZ if the confdb directory had been changed since XYZ. So the script jumped through a number of hoops in order to provide the expected result from svn export. Branch points were computed by hand, rather than attempt to teach the script to do this.

Why Change?

SVN has numerous, well documented deficiencies:

  • Branching and merging are nightmares.
  • Inspecting history is much more difficult than it is in git.
  • Working offline in SVN is limited.
  • Performance is slow.

The only three things that SVN had in its favor were inertia (we already had it installed, with other infrastructure built around it), support for fine-grained permissions via the MCS "authz" web page, and everyone basically knows how to use it at this point. Eventually our SVN pain began to exceed the inertia benefit and MCS Systems provided gitolite in order to self-administer permissions with finer granularity. The education issue is unfortunate, but this was an issue that simply must be overcome every time that a VCS becomes obsolete (it occurred for the CVS-->SVN migration).

CVS

When we converted to Subversion in November 2007 we simply made a clean break with history. The scripts and tools at the time simply could not handle some aspects of our CVS archive. As a result, tools like "git blame" and "git log" stop with commit 6a1cbdcfc42.

With some manual assistance (basically resolving all CVS modules so that you have a single cvs directory without any sub-modules) one can get 2015-era 'git cvsimport' to generate a git tree from our historical (and only internally available to MCS account holders) CVS archive. One can find such a modified tree here. One could add it to one's remote repositories like so:

git remote add mpich-cvs <https://xgitlab.cels.anl.gov/robl/mpich-cvs.git>

We cannot rebase today's git main onto this historical record. If we were to do so, it would be like rebasing local changes after having published them: everyone "downstream" would have a giant headache next time they picked up changes to the main repository. Instead, we can use git-replace.

In our mpich-main tree every commit has a parent commit... except 6a1cbdcfc42. And in mpich-CVS.git, the main-line tree stops at 7e0e4d706c32cd73e. Actually, there are two post-conversion commits to the CVS repository. We should not use those, but they were made in 2009 so they are easy to spot. A sufficiently new 'git replace' has --graft, which sounds like what we want. If one only has git-1.9.1 (as on RobL's Ubuntu-14.04 machine), one must simply stomp on the initial "put everything in its place" commit

git replace 6a1cbdcfc 7e0e4d706c32cd73e

Now your working repository has all the MPICH history dating back to the birth of MPICH2 in 2001. In fact, because we never re-wrote ROMIO but instead simply carried it forward from MPICH1, you have some history dating back to the birth of ROMIO in 1998.