Merge branch 'release/v4.3.0'

.appveyor.yml

@@ -7,13 +7,15 @@ environment:
   matrix:
     - TOXENV: "py27"
       PLATFORMIO_BUILD_CACHE_DIR: C:\Temp\PIO_Build_Cache_P2_{build}
+      PYTHON_DIRS: C:\Python27-x64;C:\Python27-x64\Scripts
 
     - TOXENV: "py36"
       PLATFORMIO_BUILD_CACHE_DIR: C:\Temp\PIO_Build_Cache_P3_{build}
+      PYTHON_DIRS: C:\Python36-x64;C:\Python36-x64\Scripts
 
 install:
   - cmd: git submodule update --init --recursive
-  - cmd: SET PATH=C:\MinGW\bin;%PATH%
+  - cmd: SET PATH=%PYTHON_DIRS%;C:\MinGW\bin;%PATH%
   - cmd: SET PLATFORMIO_CORE_DIR=C:\.pio
   - cmd: pip install --force-reinstall tox
 

HISTORY.rst

@@ -6,6 +6,38 @@ Release Notes
 PlatformIO Core 4
 -----------------
 
+4.3.0 (2020-03-19)
+~~~~~~~~~~~~~~~~~~
+
+* Initial support for an official `PlatformIO for CLion IDE <https://docs.platformio.org/page/integration/ide/clion.html>`__ plugin:
+
+  - Smart C and C++ editor
+  - Code refactoring
+  - On-the-fly code analysis
+  - "New PlatformIO Project" wizard
+  - Building, Uploading, Testing
+  - Integrated debugger (inline variable view, conditional breakpoints, expressions, watchpoints, peripheral registers, multi-thread support, etc.)
+
+* `Device Monitor 2.0 <https://docs.platformio.org/page/core/userguide/device/cmd_monitor.html>`__
+
+  - Added **PlatformIO Device Monitor Filter API** (dev-platforms can extend base device monitor with a custom functionality, such as exception decoding) (`pull #3383 <https://github.com/platformio/platformio-core/pull/3383>`_)
+  - Configure project device monitor with `monitor_filters <https://docs.platformio.org/page/projectconf/section_env_monitor.html#monitor-filters>`__ option
+  - `Capture device monitor output to a file <https://docs.platformio.org/page/core/userguide/device/cmd_monitor.html#capture-output-to-a-file>`__ with ``log2file`` filter (`issue #670 <https://github.com/platformio/platformio-core/issues/670>`_)
+  - Show a timestamp for each new line with ``time`` filter (`issue #981 <https://github.com/platformio/platformio-core/issues/981>`_)
+  - Send a text to device on ENTER with ``send_on_enter`` filter (`issue #926 <https://github.com/platformio/platformio-core/issues/926>`_)
+  - Show a hexadecimal representation of the data (code point of each character) with ``hexlify`` filter
+
+* New standalone (1-script) `PlatformIO Core Installer <https://github.com/platformio/platformio-core-installer>`_
+* Initial support for `Renode <https://docs.platformio.org/page/plus/debug-tools/qemu.html>`__ simulation framework (`issue #3401 <https://github.com/platformio/platformio-core/issues/3401>`_)
+* Added support for Arm Mbed "module.json" ``dependencies`` field (`issue #3400 <https://github.com/platformio/platformio-core/issues/3400>`_)
+* Improved support for Arduino "library.properties" ``depends`` field
+* Fixed an issue when quitting from PlatformIO IDE does not shutdown PIO Home server
+* Fixed an issue "the JSON object must be str, not 'bytes'" when PIO Home is used with Python 3.5 (`issue #3396 <https://github.com/platformio/platformio-core/issues/3396>`_)
+* Fixed an issue when Python 2 does not keep encoding when converting ".ino" (`issue #3393 <https://github.com/platformio/platformio-core/issues/3393>`_)
+* Fixed an issue when ``"libArchive": false`` in "library.json" does not work (`issue #3403 <https://github.com/platformio/platformio-core/issues/3403>`_)
+* Fixed an issue when not all commands in `compilation database "compile_commands.json" <https://docs.platformio.org/page/integration/compile_commands.html>`__ use absolute paths (`pull #3415 <https://github.com/platformio/platformio-core/pull/3415>`_)
+* Fixed an issue when unknown transport is used for `PIO Unit Testing <https://docs.platformio.org/page/plus/unit-testing.html>`__ engine (`issue #3422 <https://github.com/platformio/platformio-core/issues/3422>`_)
+
 4.2.1 (2020-02-17)
 ~~~~~~~~~~~~~~~~~~
 
@@ -34,7 +66,7 @@ PlatformIO Core 4
   - Show computed project configuration with a new `platformio project config <https://docs.platformio.org/page/userguide/project/cmd_config.html>`_ command or dump to JSON with ``platformio project config --json-output`` (`issue #3335 <https://github.com/platformio/platformio-core/issues/3335>`_)
   - Moved ``platformio init`` command to `platformio project init <https://docs.platformio.org/page/userguide/project/cmd_init.html>`_
 
-* Generate `compilation database "compile_commands.json" <https://docs.platformio.org/page/faq.html#compilation-database-compile-commands-json>`_ (`issue #2990 <https://github.com/platformio/platformio-core/issues/2990>`_)
+* Generate `compilation database "compile_commands.json" <https://docs.platformio.org/page/integration/compile_commands.html>`__ (`issue #2990 <https://github.com/platformio/platformio-core/issues/2990>`_)
 * Control debug flags and optimization level with a new `debug_build_flags <https://docs.platformio.org/page/projectconf/section_env_debug.html#debug-build-flags>`__ option
 * Install a dev-platform with ALL declared packages using a new ``--with-all-packages`` option for `pio platform install <https://docs.platformio.org/page/userguide/platforms/cmd_install.html>`__ command  (`issue #3345 <https://github.com/platformio/platformio-core/issues/3345>`_)
 * Added support for "pythonPackages" in `platform.json <https://docs.platformio.org/page/platforms/creating_platform.html#manifest-file-platform-json>`__ manifest (PlatformIO Package Manager will install dependent Python packages from PyPi registry automatically when dev-platform is installed)

README.rst

@@ -58,8 +58,8 @@ Instruments
 * `Continuous Integration <https://docs.platformio.org/page/ci/index.html?utm_source=github&utm_medium=core>`_
 * `Advanced Scripting API <https://docs.platformio.org/page/projectconf/advanced_scripting.html?utm_source=github&utm_medium=core>`_
 
-PIO Plus
---------
+Professional
+------------
 
 * `PIO Check <https://docs.platformio.org/page/plus/pio-check.html?utm_source=github&utm_medium=core>`_
 * `PIO Remote <https://docs.platformio.org/page/plus/pio-remote.html?utm_source=github&utm_medium=core>`_

platformio/__init__.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-VERSION = (4, 2, 1)
+VERSION = (4, 3, 0)
 __version__ = ".".join([str(s) for s in VERSION])
 
 __title__ = "platformio"

platformio/builder/tools/compilation_db.py

@@ -76,6 +76,16 @@ def EmitCompilationDbEntry(target, source, env):
         :return: target(s), source(s)
         """
 
+        # Resolve absolute path of toolchain
+        for cmd in ("CC", "CXX", "AS"):
+            if cmd not in env:
+                continue
+            if os.path.isabs(env[cmd]):
+                continue
+            env[cmd] = where_is_program(
+                env.subst("$%s" % cmd), env.subst("${ENV['PATH']}")
+            )
+
         dbtarget = __CompilationDbNode(source)
 
         entry = env.__COMPILATIONDB_Entry(
@@ -195,14 +205,6 @@ def generate(env, **kwargs):
     )
 
     def CompilationDatabase(env, target):
-        # Resolve absolute path of toolchain
-        for cmd in ("CC", "CXX", "AS"):
-            if cmd not in env:
-                continue
-            env[cmd] = where_is_program(
-                env.subst("$%s" % cmd), env.subst("${ENV['PATH']}")
-            )
-
         result = env.__COMPILATIONDB_Database(target=target, source=[])
 
         env.AlwaysBuild(result)

platformio/builder/tools/piolib.py

@@ -18,6 +18,7 @@
 from __future__ import absolute_import
 
 import hashlib
+import io
 import os
 import re
 import sys
@@ -82,7 +83,8 @@ def get_used_frameworks(env, path):
                     fname, piotool.SRC_BUILD_EXT + piotool.SRC_HEADER_EXT
                 ):
                     continue
-                content = fs.get_file_contents(join(root, fname))
+                with io.open(join(root, fname), errors="ignore") as fp:
+                    content = fp.read()
                 if not content:
                     continue
                 if "Arduino.h" in content and include_re.search(content):
@@ -716,9 +718,11 @@ def extra_script(self):
 
     @property
     def lib_archive(self):
-        unique_value = "_not_declared_%s" % id(self)
-        global_value = self.env.GetProjectOption("lib_archive", unique_value)
-        if global_value != unique_value:
+        missing = object()
+        global_value = self.env.GetProjectConfig().getraw(
+            "env:" + self.env["PIOENV"], "lib_archive", missing
+        )
+        if global_value != missing:
             return global_value
         return self._manifest.get("build", {}).get(
             "libArchive", LibBuilderBase.lib_archive.fget(self)

platformio/builder/tools/piomaxlen.py

@@ -18,7 +18,6 @@
 from os import makedirs
 from os.path import isdir, isfile, join
 
-from platformio import fs
 from platformio.compat import WINDOWS, hashlib_encode_data
 
 # Windows CLI has limit with command length to 8192
@@ -67,7 +66,8 @@ def _file_long_data(env, data):
     )
     if isfile(tmp_file):
         return tmp_file
-    fs.write_file_contents(tmp_file, data)
+    with open(tmp_file, "w") as fp:
+        fp.write(data)
     return tmp_file
 
 

platformio/builder/tools/piomisc.py

@@ -15,17 +15,19 @@
 from __future__ import absolute_import
 
 import atexit
+import io
 import re
 import sys
 from os import environ, remove, walk
 from os.path import basename, isdir, isfile, join, realpath, relpath, sep
 from tempfile import mkstemp
 
+import click
 from SCons.Action import Action  # pylint: disable=import-error
 from SCons.Script import ARGUMENTS  # pylint: disable=import-error
 
 from platformio import fs, util
-from platformio.compat import glob_escape
+from platformio.compat import get_filesystem_encoding, get_locale_encoding, glob_escape
 from platformio.managers.core import get_core_package_dir
 from platformio.proc import exec_command
 
@@ -48,6 +50,39 @@ class InoToCPPConverter(object):
     def __init__(self, env):
         self.env = env
         self._main_ino = None
+        self._safe_encoding = None
+
+    def read_safe_contents(self, path):
+        error_reported = False
+        for encoding in (
+            "utf-8",
+            None,
+            get_filesystem_encoding(),
+            get_locale_encoding(),
+            "latin-1",
+        ):
+            try:
+                with io.open(path, encoding=encoding) as fp:
+                    contents = fp.read()
+                    self._safe_encoding = encoding
+                    return contents
+            except UnicodeDecodeError:
+                if not error_reported:
+                    error_reported = True
+                    click.secho(
+                        "Unicode decode error has occurred, please remove invalid "
+                        "(non-ASCII or non-UTF8) characters from %s file or convert it to UTF-8"
+                        % path,
+                        fg="yellow",
+                        err=True,
+                    )
+        return ""
+
+    def write_safe_contents(self, path, contents):
+        with io.open(
+            path, "w", encoding=self._safe_encoding, errors="backslashreplace"
+        ) as fp:
+            return fp.write(contents)
 
     def is_main_node(self, contents):
         return self.DETECTMAIN_RE.search(contents)
@@ -62,7 +97,7 @@ def merge(self, nodes):
         assert nodes
         lines = []
         for node in nodes:
-            contents = fs.get_file_contents(node.get_path())
+            contents = self.read_safe_contents(node.get_path())
             _lines = ['# 1 "%s"' % node.get_path().replace("\\", "/"), contents]
             if self.is_main_node(contents):
                 lines = _lines + lines
@@ -78,16 +113,14 @@ def merge(self, nodes):
     def process(self, contents):
         out_file = self._main_ino + ".cpp"
         assert self._gcc_preprocess(contents, out_file)
-        contents = fs.get_file_contents(out_file)
+        contents = self.read_safe_contents(out_file)
         contents = self._join_multiline_strings(contents)
-        fs.write_file_contents(
-            out_file, self.append_prototypes(contents), errors="backslashreplace"
-        )
+        self.write_safe_contents(out_file, self.append_prototypes(contents))
         return out_file
 
     def _gcc_preprocess(self, contents, out_file):
         tmp_path = mkstemp()[1]
-        fs.write_file_contents(tmp_path, contents, errors="backslashreplace")
+        self.write_safe_contents(tmp_path, contents)
         self.env.Execute(
             self.env.VerboseAction(
                 '$CXX -o "{0}" -x c++ -fpreprocessed -dD -E "{1}"'.format(

platformio/builder/tools/pioproject.py

@@ -14,7 +14,7 @@
 
 from __future__ import absolute_import
 
-from platformio.project.config import ProjectConfig, ProjectOptions
+from platformio.project.config import MISSING, ProjectConfig, ProjectOptions
 
 
 def GetProjectConfig(env):
@@ -25,7 +25,7 @@ def GetProjectOptions(env, as_dict=False):
     return env.GetProjectConfig().items(env=env["PIOENV"], as_dict=as_dict)
 
 
-def GetProjectOption(env, option, default=None):
+def GetProjectOption(env, option, default=MISSING):
     return env.GetProjectConfig().get("env:" + env["PIOENV"], option, default)