Merge v2 into master

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "src/vrs-protobuf"]
+	path = src/vrs-protobuf
+	url = https://github.com/ga4gh/vrs-protobuf.git

.mvn/wrapper/maven-wrapper.properties

@@ -1 +1 @@
-distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.6.2/apache-maven-3.6.2-bin.zip
+distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.8.1/apache-maven-3.8.1-bin.zip

CHANGELOG.rst

@@ -0,0 +1,79 @@
+=========
+Changelog
+=========
+
+1.0.0 to 2.0
+--------------
+
+This is no an exhaustive list of changes, but highlights the most important additions and breaking changes.
+
+Phenopacket
+~~~~~~~~~~~
+- Removed `genes` and `variant` fields. These should be sent as part of an `Interpretation`.
+
+
+Biosample
+~~~~~~~~~~~
+- Removed `variant` field. These should be sent as part of an `Interpretation`.
+
+
+TimeElement
+~~~~~~~~~~~
+- Added a new `TimeElement` message to wrap all the previous `oneof` in `Individual`, `Biosample`, `Disease`, `PhenotypicFeature`
+
+.. code-block::
+
+    message TimeElement {
+        oneof element {
+            GestationalAge gestational_age = 6;
+            Age age = 1;
+            AgeRange age_range = 2;
+            OntologyClass ontology_class = 3;
+            google.protobuf.Timestamp timestamp = 4;
+            TimeInterval interval = 5;
+        }
+    }
+
+GestationalAge
+~~~~~~~~~~~~~~
+- Added new `GestationalAge` message for pre-natal individuals and samples
+
+
+Individual
+~~~~~~~~~~
+- Replaced oneof `age_at_collection` with new `TimeElement` message
+- Added new `OntologyClass` `gender` field
+- Added new `VitalStatus` field
+
+
+VitalStatus
+~~~~~~~~~~~
+- Added new message for capturing the vital status of an individual
+
+
+PhenotypicFeature
+~~~~~~~~~~~~~~~~~
+- Replaced oneof `onset` with new `TimeElement` message
+
+
+Disease
+~~~~~~~~~~~~~~~~~
+- Replaced oneof `onset` with new `TimeElement` message
+
+
+Measurement
+~~~~~~~~~~~
+
+- Added new `Measurement` message for capturing quantitative, ordinal (e.g., absent/present), or categorical measurements. This element is available as a repeated field in the `Phenopacket` and `Biosample` top-level elements.
+
+
+MedicalAction
+~~~~~~~~~~~~~
+
+- Added new `MedicalAction` to capture medications, procedures, other actions taken for clinical management. This element is available as a repeated field in the `Phenopacket`.
+
+
+Interpretation
+~~~~~~~~~~~~~~
+
+- `Interpretation` is now a sub-element of a `Phenopacket`, rather than an enclosing element. The change allows for better semantics on the `Gene` and `Variant` types and their relationship to an `Individual` or `Biosample` in the context of a `Diagnosis` based on a `GenomincInterpretation`.

LICENSE

@@ -1,6 +1,6 @@
 BSD 3-Clause License
 
-Copyright (c) 2018, PhenoPackets
+Copyright (c) 2018-2021, Global Alliance for Genomics and Health
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

README.rst

@@ -9,8 +9,9 @@ Phenopacket schema
 .. |Maven Central| image:: https://maven-badges.herokuapp.com/maven-central/org.phenopackets/phenopacket-schema/badge.svg
   :target: https://maven-badges.herokuapp.com/maven-central/org.phenopackets/phenopacket-schema
 
-.. |Documentation| image:: https://readthedocs.org/projects/phenopackets-schema/badge/?version=latest
-  :target: https://phenopacket-schema.readthedocs.io/en/latest/
+.. |Documentation| image:: https://readthedocs.org/projects/phenopacket-schema/badge/?version=v2
+    :target: https://phenopacket-schema.readthedocs.io/en/v2/?badge=v2
+    :alt: Documentation Status
 
 This has been produced as part of the `GA4GH`_ `Clinical Phenotype Data Capture Workstream`_ and it merges the existing `GA4GH metadata-schemas`_ work with a more focused model from the `phenopacket-reference-implementation`_.
 
@@ -27,9 +28,9 @@ Documentation
 
 The core documentation can be found at `Documentation`_
 
-The documentation in this README is primarily for the users of the phenopackets-schema java library.
+The documentation in this README is primarily for the users of the phenopacket-schema java library.
 
-.. _Documentation: https://phenopacket-schema.readthedocs.io/en/latest/index.html
+.. _Documentation: https://phenopacket-schema.readthedocs.io/en/latest
 
 Scope and Purpose
 =================
@@ -53,10 +54,14 @@ about this list is available at https://groups.io/g/phenopackets.
 Usage
 =====
 The Phenopacket schema is defined using `Protobuf`_ which is `"a language-neutral, platform-neutral extensible mechanism for serializing structured data"`.  There are two ways to use this library, firstly using the ``Phenopacket`` as an exchange mechanism, secondly as a schema of basic types on which to build more specialist messages, yet allow for easy interoperability with other resources using the phenopackets schema.
+The following sections describe how to achieve these two things.
 
 .. _Protobuf: https://developers.google.com/protocol-buffers/
 
-Java people can incorporate phenopackets-api into their code by importing the jar using maven:
+Include phenopackets into your project
+--------------------------------------
+
+**Java** people can incorporate phenopackets into their code by importing the jar using maven:
 
 .. code:: xml
 
@@ -66,7 +71,12 @@ Java people can incorporate phenopackets-api into their code by importing the ja
         <version>${phenopacket-schema.version}</version>
     </dependency>
 
-The following sections describe how to achieve these two things.
+Using phenopackets in **Python** is also straightforward::
+
+.. code:: python
+
+    pip install phenopackets
+
 
 Exchanging Phenopackets directly
 --------------------------------
@@ -84,8 +94,11 @@ A Phenopacket can be transformed between the native binary format and JSON using
         <version>${protobuf.version}</version>
     </dependency>
 
+.. code:: python
+
+    pip install protobuf
 
-``protobuf-java-util`` contains simple utility methods to perform these transformations. Usage is shown here:
+``protobuf-java-util`` for java and ``protobuf`` for python contain simple utility methods to perform these transformations. Usage is shown here:
 
 .. code-block:: java
 
@@ -113,12 +126,47 @@ A Phenopacket can be transformed between the native binary format and JSON using
     JsonFormat.parser().merge(jsonPhenopacket, phenoPacketBuilder2);
     Phenopacket fromJson2 = phenoPacketBuilder2.build();
 
+.. code-block:: python
+
+    from google.protobuf.json_format import Parse, MessageToJson
+    from google.protobuf.timestamp_pb2 import Timestamp
+    from phenopackets import Phenopacket, Individual, PhenotypicFeature, OntologyClass
+
+    # Parsing phenopackets from json
+    with open('file.json', 'r') as jsfile:
+        phenopacket = Parse(Phenopacket(), text=jsfile.read())
+
+    # Writing phenopackets to json
+    with open('file.json', 'w') as jsfile:
+        subject = Individual(id="Zaphod", sex="MALE", date_of_birth=Timestamp(seconds=-123456798))
+        phenotypic_features = [PhenotypicFeature(type=OntologyClass(id="HG2G:00001", label="Hoopy")),
+                               PhenotypicFeature(type=OntologyClass(id="HG2G:00002", label="Frood"))]
+
+        phenopacket = Phenopacket(id="PPKT:1", subject=subject, phenotypic_features=phenotypic_features)
+
+        json = MessageToJson(phenopacket)
+        jsfile.write(json)
+
 Building new messages from the schema
 -------------------------------------
 There is an example of how to do this included in the `mme.proto`_ file. Here the Matchmaker Exchange (MME) API has been implemented using the phenopackets schema, defining custom messages as required, but re-using messages from `base.proto`_ where applicable. Using the above example, perhaps the ``Phenopacket.genes`` is a problem as you wish to record not only the gene panels ordered, but also the candidate genes discovered in two separate fields. In this case, a new bespoke message could be created, using the ``Gene`` as a building block.
 
 .. _mme.proto: https://github.com/phenopackets/phenopacket-schema/blob/master/src/test/proto/org/matchmakerexchange/api/v1/mme.proto
 .. _base.proto: https://github.com/phenopackets/phenopacket-schema/blob/master/src/main/proto/org/phenopackets/schema/v1/core/base.proto
+
+Git Submodules
+==============
+This repo uses `git submodules`_ to import the `VRS protobuf` implementation. You may need to use the following command after cloning/update
+for things to build correctly:
+
+.. code:: bash
+
+  $ git submodule update --init --recursive
+
+
+.. _git submodules: https://git-scm.com/book/en/v2/Git-Tools-Submodules
+.. _VRS protobuf: https://github.com/ga4gh/vrs-protobuf
+
 Building
 ========
 The project can be built using the awesome `Takari maven wrapper`_ which requires no local maven installation. The only requirement for the build is to have a working java installation and network access.
@@ -131,7 +179,7 @@ To do this ``cd`` to the project root and run the wrapper scripts:
 
 or
 
-.. code:: cmd
+.. code:: batch
 
     $ ./mvnw.cmd clean install
 
@@ -140,12 +188,24 @@ or
 
 Sign artefacts for release
 ==========================
-There is a ``release-sign-artifacts`` profile which can be triggered with the command
+There is a ``release-sign-artifacts`` profile for **Java** which can be triggered with the command
 
 .. code:: bash
 
     $ ./mvnw clean install -DperformRelease=true
 
+The **Python** artefacts are released by running::
+
+Test
+
+.. code::bash
+    $ bash deploy-python.sh release-test
+
+Production
+
+.. code::bash
+    $ bash deploy-python.sh release-prod
+
 Java, Python and C++ artefacts
 ==============================
 Building the project will automatically compile Java, Python and C++ artefacts. The Java jar file can be directly used in any Java project. For Python or C++ the build artefacts can be found at

deploy-python.sh

@@ -0,0 +1,80 @@
+# Create Temporary Destination
+# Phenopackets folder
+TEMP_DIRECTORY=$(mktemp -d)
+echo "Building phenopacket distribution files in temporary directory at $TEMP_DIRECTORY"
+TEMP_DIRECTORY_PYTHON_MODULE="$TEMP_DIRECTORY/phenopackets"
+TEMP_DIRECTORY_TESTS_MODULE="$TEMP_DIRECTORY/tests"
+TEMP_DIRECTORY_VIRTUAL_ENV="$TEMP_DIRECTORY/phenopackets-venv"
+declare -a pyfiles=("base" "phenopackets" "biosample" "disease" "genome" "individual" "interpretation" "medical_action" "measurement" "meta_data" "pedigree" "phenotypic_feature" "vrsatile")
+# Functions
+createInitFile(){
+  echo "import pkg_resources" >> "$TEMP_DIRECTORY/phenopackets/__init__.py"
+  echo "__version__ = pkg_resources.get_distribution('phenopackets').version"   >> "$TEMP_DIRECTORY/phenopackets/__init__.py"
+  for i in "${pyfiles[@]}"
+  do
+     echo "from .${i}_pb2 import *"  >> "$TEMP_DIRECTORY/phenopackets/__init__.py"
+  done
+}
+
+replaceImports(){
+    for i in "${pyfiles[@]}"
+  do
+      sed -i 's/from phenopackets.schema.v2.core/from . /g' "$TEMP_DIRECTORY_PYTHON_MODULE/${i}_pb2.py"
+      sed -i 's/from ga4gh.vrsatile.v1/from . /g' "$TEMP_DIRECTORY_PYTHON_MODULE/${i}_pb2.py"
+      sed -i 's/from ga4gh.vrs.v1/from . /g' "$TEMP_DIRECTORY_PYTHON_MODULE/${i}_pb2.py"
+  done
+}
+
+createVirtualEnvironment(){
+  echo "Creating Python virtual environment at ${1}"
+  python3 -m venv "${1}" &> /dev/null
+  if [ ${?} = 1 ]; then
+    echo "Setup of Python virtual environment using 'python3 -m venv' failed. Trying 'virtualenv'"
+    virtualenv "${1}" &> /dev/null
+  fi
+  if [ ${?} = 1 ]; then
+    echo "Deployment FAILED. Could not create Python virtual environment"
+    exit 1;
+  fi
+  echo "Virtual environment created successfully";
+}
+
+# Create python module
+mkdir $TEMP_DIRECTORY_PYTHON_MODULE
+createInitFile
+cp ./target/generated-sources/protobuf/python/phenopackets/schema/v2/phenopackets_pb2.py $TEMP_DIRECTORY_PYTHON_MODULE
+cp ./target/generated-sources/protobuf/python/phenopackets/schema/v2/core/* $TEMP_DIRECTORY_PYTHON_MODULE
+cp ./target/generated-sources/protobuf/python/ga4gh/vrsatile/v1/vrsatile_pb2.py $TEMP_DIRECTORY_PYTHON_MODULE
+cp ./target/generated-sources/protobuf/python/ga4gh/vrs/v1/vrs_pb2.py $TEMP_DIRECTORY_PYTHON_MODULE
+replaceImports
+# Create tests module
+mkdir $TEMP_DIRECTORY_TESTS_MODULE
+cp ./src/test/python/* $TEMP_DIRECTORY_TESTS_MODULE
+# Copy Packaging files
+cp requirements.txt setup.py pom.xml LICENSE README.rst $TEMP_DIRECTORY
+
+# Create Python venv in virtual directory
+createVirtualEnvironment $TEMP_DIRECTORY_VIRTUAL_ENV
+cd $TEMP_DIRECTORY || { echo "Deployment FAILED. Couldn't cd to temp directory" ; exit 1; }
+# shellcheck disable=SC1090
+source "$TEMP_DIRECTORY_VIRTUAL_ENV/bin/activate"
+pip install -r "$TEMP_DIRECTORY/requirements.txt"
+# Dependencies for building/deploying
+python3 -m pip install setuptools wheel twine || { echo "Deployment FAILED. Failed to install python dependencies" ; exit 1; }
+# Test
+pip install -e .
+python3 setup.py test || { echo "Deployment FAILED. Unittest Failure" ; exit 1; }
+# Build
+python3 setup.py sdist bdist_wheel || { echo "Deployment FAILED. Building python package" ; exit 1; }
+
+# Deploy - Remove --repository testpypi flag for production.
+if [ $1 = "release-prod" ]; then
+  python3 -m twine upload dist/*
+elif [ $1 = "release-test" ]; then
+  python3 -m twine upload --repository testpypi dist/*
+else
+  echo "Python Release was prepared successfully. No release argument provided, use one of [release-prod, release-test] to make the production/test release."
+fi
+
+
+

docs/Makefile

@@ -4,7 +4,7 @@
 # You can set these variables from the command line.
 SPHINXOPTS    =
 SPHINXBUILD   = python -msphinx
-SPHINXPROJ    = phenopackets-schema
+SPHINXPROJ    = phenopacket-schema
 SOURCEDIR     = .
 BUILDDIR      = _build
 html_static_path = ['..']

docs/_static/custom.css

@@ -10,4 +10,32 @@
   .wy-table-responsive {
     overflow: visible !important;
   }
+
+  .color-interpretation {
+    color: #63498B;
+  }
+
+  .color-base {
+    color: #1b75bc;
+  }
+
+  .color-measurement {
+    color: #4DA8D4;
+  }
+
+  .color-genomic-vrs {
+    color: #5BC4BF;
+  }
+
+  .color-pedigree {
+    color: #8BC751;
+  }
+
+  .color-top-level {
+    color: #faa634;
+  }
+
+  .color-medical-action {
+    color: #AAAAAA;
+  }
 }

docs/_static/style.css

@@ -0,0 +1,3 @@
+.wy-nav-content {
+    max-width: 1000px !important;
+}

docs/_templates/layout.html

@@ -0,0 +1,4 @@
+{% extends "!layout.html" %}
+{% block extrahead %}
+<link href="{{ pathto("_static/style.css", True) }}" rel="stylesheet" type="text/css">
+{% endblock %}