Merge branch 'nexB:develop' into add_new_flag

README.rst

@@ -6,7 +6,11 @@ A typical software project often reuses hundreds of third-party packages.
 License and origin information is not always easy to find and not normalized:
 ScanCode discovers and normalizes this data for you.
 
-Read more about ScanCode here: `scancode-toolkit.readthedocs.io <https://scancode-toolkit.readthedocs.io/en/latest/>`_.
+Read more about ScanCode here: `scancode-toolkit.readthedocs.io 
+<https://scancode-toolkit.readthedocs.io/en/latest/>`_.
+
+Check out the code at https://github.com/nexB/scancode-toolkit
+
 
 Why use ScanCode?
 =================
@@ -25,8 +29,10 @@ Why use ScanCode?
   `RedHat Fabric8 analytics <https://github.com/fabric8-analytics>`_,
   and many more.
 
-- ScanCode detects licenses, copyrights, package manifests, direct
-  dependencies, and more both in **source code** and **binary** files.
+- ScanCode detects licenses, copyrights, package manifests, direct dependencies,
+  and more both in **source code** and **binary** files and is considered as the
+  best-in-class and reference tool in this domain, re-used as the core tools for
+  software composition data collection by several open source tools.
 
 - ScanCode provides the **most accurate license detection engine** and does a
   full comparison (also known as diff or red line comparison) between a database
@@ -37,14 +43,12 @@ Why use ScanCode?
   new and improved scanners, data summarization, package manifest parsers, and
   new outputs.
 
-- You can save your scan results as **JSON, HTML, CSV or SPDX**. And you can use the
-  companion `ScanCode workbench GUI app <https://github.com/nexB/scancode-workbench>`_
-  to review and display scan results, statistics and graphics.
+- You can save your scan results as **JSON, HTML, CSV or SPDX** or create your
+  own format with Jinja templates.
 
 - You can also organize and run ScanCode server-side with the
-  companion `ScanCode.io web app <https://github.com/nexB/scancodeio>`_
-  to organize and store multiple scan projects including scripting scanning piplines.
-
+  companion `ScanCode.io web app <https://github.com/nexB/scancode.io>`_
+  to organize and store multiple scan projects including scripted scanning pipelines.
 
 - ScanCode is **actively maintained**, has a **growing users and contributors
   community**.
@@ -53,29 +57,35 @@ Why use ScanCode?
 
 - ScanCode has an extensive and growing documentation.
 
+- ScanCode can process these packages, build manifest and lockfile formats to extract metadata:
+  Alpine packages, BUCK files, ABOUT files, Android apps, Autotools, Bazel, 
+  JavaScript Bower, Java Axis, MS Cab, Rust Cargo, Chef Chrome apps, 
+  PHP Composer and composer.lock, Conda, CPAN, Debian, Apple dmg,
+  Java EAR, WAR, JAR, FreeBSD packages, Rubygems gemspec, Gemfile and Gemfile.lock, 
+  Go modules, Haxe packages, InstallShield installers, iOS apps, ISO images, 
+  Apache IVY, JBoss Sar, R CRAN, Apache Maven, Meteor, Mozilla extensions, 
+  MSI installers, JavaScript npm packages, package-lock.json, yarn.lock, 
+  NSIS Installers, NugGet, OPam, Cocoapods, Python PyPI setup.py, setup.cfg, and 
+  several related lockfile formats, semi structured README
+  files such as README.android, README.chromium, README.facebook, README.google,
+  README.thirdparty, RPMs, Shell Archives, Squashfs images, Java WAR, Windows executables.
+
+
 See our `roadmap <https://scancode-toolkit.readthedocs.io/en/latest/contribute/roadmap.html>`_
 for upcoming features.
 
 
 Build and tests status
 ======================
 
-+-------+--------------+-----------------+--------------+
-|Branch | **Coverage** | **Linux/macOS** | **Windows**  |
-+=======+==============+=================+==============+
-|Master | |master-cov| | |master-posix|  | |master-win| |
-+-------+--------------+-----------------+--------------+
-|Develop| |devel-cov|  | |devel-posix|   | |devel-win|  |
-+-------+--------------+-----------------+--------------+
+We run tests on multiple CIs to ensure a good platform compabitility and on
+multiple version of Windows, Linux and macOS.
 
-Documentation Build
--------------------
-
-+--------+--------------+
-|Version | **RTD Build**|
-+========+==============+
-| Latest | |docs-rtd|   |
-+--------+--------------+
++--------------+--------------+--------------+
+| **Appveyor** | **Azure**    | **RTD Build**|
++==============+==============+==============+
+| |appveyor|   |    |azure|   | |docs-rtd|   |
++--------------+--------------+--------------+
 
 
 Documentation
@@ -110,7 +120,9 @@ Installation
 ============
 
 Before installing ScanCode make sure that you have installed the prerequisites
-properly. This means installing Python (Python 3.6+ is required).
+properly. This means installing Python (Python 3.6 or higher is required.
+When installing the app tarball or zip, only Python 3.6 is supported for now).
+
 See `prerequisites <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#prerequisites>`_
 for detailed information on the support platforms and Python versions.
 
@@ -215,19 +227,13 @@ See the NOTICE file and the .ABOUT files that document the origin and license of
 the third-party code used in ScanCode for more details.
 
 
-.. |master-posix| image:: https://api.travis-ci.org/nexB/scancode-toolkit.png?branch=master
+.. |appveyor| image:: https://ci.appveyor.com/api/projects/status/4webymu0l2ip8utr?svg=true
     :target: https://travis-ci.org/nexB/scancode-toolkit
-    :alt: Linux Master branch tests status
-.. |devel-posix| image:: https://api.travis-ci.org/nexB/scancode-toolkit.png?branch=develop
-    :target: https://travis-ci.org/nexB/scancode-toolkit
-    :alt: Linux Develop branch tests status
-
-.. |master-win| image:: https://ci.appveyor.com/api/projects/status/4webymu0l2ip8utr/branch/master?png=true
-    :target: https://ci.appveyor.com/project/nexB/scancode-toolkit
-    :alt: Windows Master branch tests status
-.. |devel-win| image:: https://ci.appveyor.com/api/projects/status/4webymu0l2ip8utr/branch/develop?png=true
-    :target: https://ci.appveyor.com/project/nexB/scancode-toolkit
-    :alt: Windows Develop branch tests status
+    :alt: Appveyor tests status (Windows)
+
+.. |azure| image:: https://dev.azure.com/nexB/scancode-toolkit/_apis/build/status/nexB.scancode-toolkit?branchName=develop
+    :target: https://dev.azure.com/nexB/scancode-toolkit/_build/latest?definitionId=1&branchName=develop
+    :alt: Azure tests status (Linux, macOS, Windows)
 
 .. |docs-rtd| image:: https://readthedocs.org/projects/scancode-toolkit/badge/?version=latest
     :target: https://scancode-toolkit.readthedocs.io/en/latest/?badge=latest

docs/source/getting-started/install.rst

@@ -286,13 +286,13 @@ Or you can run the following if you have `Git <https://git-scm.com/>`_ installed
 Now, by default the files are checked out to the develop branch, but you can jump to any checkpoint
 using the following command::
 
-    git checkout main
+    git checkout develop
 
-Here, ``main`` branch has the latest release of Scancode-Toolkit. You can also check out to any
-of the following:
+Here, ``develop`` branch has the latest release of Scancode-Toolkit. 
+You can also check out to any of the following:
 
-- Branches (Locally created or already present) [Example - ``main``]
-- Tags (essentially version numbers) [Example - ``v21.2.25``, ``v21.6.1``]
+- Branches (Locally created or already present) [Example - ``develop``]
+- Tags (essentially version numbers) [Example - ``v21.6.7``, ``v21.5.31``]
 - Commits (use the shortened commit hash) [Example - ``4502055``, ``f276398``]
 
 

etc/release/scancode-create-release.sh

@@ -6,7 +6,7 @@
 ################################################################################
 # ScanCode release build script
 # Create, test and publish release archives, wheels and sdists.
-# Use the --test to also run basic somke test of the built archives
+# Use the --test to also run basic smoke tests of the built archives
 #
 ################################################################################
 
@@ -142,6 +142,8 @@ function run_pypi_smoke_tests {
     echo "#### RELEASE TEST: Completed PyPI tests of $archive_to_test with Pythons: $python_dot_versions on OSses: $operating_systems"
 
 }
+
+
 if [ "$CLI_ARGS" == "--test" ]; then
     echo "##########################################################################"
     echo "### TESTING build for Python: $PYTHON_APP_VERSIONS on OS: $OPERATING_SYSTEMS"
@@ -161,14 +163,10 @@ if [ "$CLI_ARGS" == "--test" ]; then
 fi
 
 
-
-
 echo "##########################################################################"
 echo "### BUILDING App for Python: $PYTHON_APP_VERSIONS on OS: $OPERATING_SYSTEMS"
 
 
-
-
 ################################
 # Setup
 ################################

setup-mini.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = scancode-toolkit-mini
-version = 21.6.2
+version = 21.6.7
 license = Apache-2.0 AND CC-BY-4.0 AND LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft
 
 description = ScanCode is a tool to scan code for license, copyright, package and their documented dependencies and other interesting facts. scancode-toolkit-mini is a special build that does not come with pre-built binary dependencies by default. These are instead installed separately or with the extra_requires scancode-toolkit-mini[full]

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = scancode-toolkit
-version = 21.6.2
+version = 21.6.7
 license = Apache-2.0 AND CC-BY-4.0 AND LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft
 
 description = ScanCode is a tool to scan code for license, copyright, package and their documented dependencies and other interesting facts.

src/packagedcode/__init__.py

@@ -7,6 +7,8 @@
 # See https://aboutcode.org for more information about nexB OSS projects.
 #
 
+import attr
+
 from packagedcode import about
 from packagedcode import bower
 from packagedcode import build
@@ -126,20 +128,111 @@ def get_package_class(scan_data, default=models.Package):
     return ptype_class or default
 
 
-_props = frozenset([
-    'api_data_url',
-    'repository_download_url',
-    'purl',
-    'repository_homepage_url']
-)
+def get_package_instance(scan_data):
+    """
+    Return a Package instance re-built from a mapping of ``scan_data`` native
+    Python data that has the structure of a scan. Known attributes that store a
+    list of objects are also "rehydrated" (such as models.Party).
 
+    The Package instance will use the Package subclass that supports the
+    provided package "type" when possible or the base Package class otherwise.
 
-def get_package_instance(scan_data, properties=_props):
-    """
-    Given a `scan_data` native Python mapping representing a Package, return a
-    Package object instance.
+    Unknown attributes provided in ``scan_data`` that do not exist as fields in
+    the Package class are kept as items in the Package.extra_data mapping.
+    An Exception is raised if an "unknown attribute" name already exists as
+    a Package.extra_data key.
     """
-    # remove computed properties from attributes
-    scan_data = {k: v for k, v in scan_data.items() if k not in properties}
+    # TODO: consider using a proper library for this such as cattrs,
+    # marshmallow, etc. or use the field type that we declare.
+
+    # Each of these are lists of class instances tracked here, which are stored
+    # as a list of mappings in scanc_data
+    list_field_types_by_name = {
+        'parties': models.Party,
+        'dependencies': models.DependentPackage,
+        'installed_files': models.PackageFile,
+    }
+
+    # these are computed attributes serialized on a package
+    # that should not be recreated when serializing
+    computed_attributes = set([
+        'purl',
+        'repository_homepage_url',
+        'repository_download_url',
+        'api_data_url'
+    ])
+
+    # re-hydrate lists of typed objects
     klas = get_package_class(scan_data)
-    return klas(**scan_data)
+    existing_fields = attr.fields_dict(klas)
+
+    extra_data = scan_data.get('extra_data')
+    package_data = {}
+
+    for key, value in scan_data.items():
+        if not value or key in computed_attributes:
+            continue
+
+        field = existing_fields.get(key)
+
+        if not field:
+            if key not in extra_data:
+                # keep unknown field as extra data
+                extra_data[key] = value
+                continue
+            else:
+                raise Exception(
+                    f'Invalid scan_data with duplicated key: {key}={value!r} '
+                    f'present both as attribute AND as extra_data: '
+                    f'{key}={extra_data[key]!r}'
+                )
+
+        list_field_type = list_field_types_by_name.get(key)
+        if not list_field_type:
+            # this is a plain known field
+            package_data[key] = value
+            continue
+
+        # Since we have a list_field_type, value must be a list of mappings:
+        # we transform it in a list of objects.
+
+        if not isinstance(value, list):
+            raise Exception(
+                f'Invalid scan_data with unknown data structure. '
+                f'Expected the value to be a list of dicts and not a '
+                f'{type(value)!r} for {key}={value!r}'
+            )
+
+        objects = list(_build_objects_list(values=value, klass=list_field_type))
+        package_data[key] = objects
+
+    return klas(**package_data)
+
+
+def _build_objects_list(values, klass):
+    """
+    Yield ``klass`` objects built from a ``values`` list of mappings.
+    """
+    # Since we have a list_field_type, value must be a list of mappings:
+    # we transform it in a list of objects.
+
+    if not isinstance(values, list):
+        raise Exception(
+            f'Invalid scan_data with unknown data structure. '
+            f'Expected the value to be a list of dicts and not a '
+            f'{type(values)!r} for {values!r}'
+        )
+
+    for val in values:
+        if not val:
+            continue
+
+        if not isinstance(val, dict):
+            raise Exception(
+                f'Invalid scan_data with unknown data structure. '
+                f'Expected the value to be a mapping for and not a '
+                f'{type(val)!r} for {values!r}'
+            )
+
+        yield klass.create(**val)
+

src/packagedcode/about.py

@@ -37,12 +37,8 @@ class AboutPackage(models.Package):
     def recognize(cls, location):
         yield parse(location)
 
-    @classmethod
-    def get_package_root(cls, manifest_resource, codebase):
-        # FIXME: this should have been already stored with the Package itself as extra_data
-        with io.open(manifest_resource.location, encoding='utf-8') as loc:
-            package_data = saneyaml.load(loc.read())
-        about_resource = package_data.get('about_resource')
+    def get_package_root(self, manifest_resource, codebase):
+        about_resource = self.extra_data.get('about_resource')
         if about_resource:
             manifest_resource_parent = manifest_resource.parent(codebase)
             for child in manifest_resource_parent.children(codebase):
@@ -89,7 +85,7 @@ def build_package(package_data):
         owner = repr(owner)
     parties = [models.Party(type=models.party_person, name=owner, role='owner')]
 
-    return AboutPackage(
+    about_package = AboutPackage(
         type='about',
         name=name,
         version=version,
@@ -99,3 +95,5 @@ def build_package(package_data):
         homepage_url=homepage_url,
         download_url=download_url,
     )
+    about_package.extra_data['about_resource'] = package_data.get('about_resource')
+    return about_package

src/packagedcode/alpine.py

@@ -106,7 +106,7 @@ def build_package(package_fields):
             converted_fields.update(converted)
 
     # construct the package: we ignore unknown as we added a few technical fields
-    package = AlpinePackage.create(ignore_unknown=True, **converted_fields)
+    package = AlpinePackage.create(**converted_fields)
     return package
 
 # Note handlers MUST accept **kwargs as they also receive the current data