This repository is an MCVE for the following issues:

• JonathonReinhart/scuba#77
• JonathonReinhart/staticx#22

FIXED: The solution was to set include_package_data = True. See below for more details.

---

Open issues

• pypa/setuptools#1064

Problem statement

When python setup.py sdist bdist_wheel is invoked from a clean starting point, data files (identified in package_data), which are generated during setup.py hooks, will not be included in the wheel.

• From a clean starting point (git clean -fdx) (primarily removes mkpkg/generated_data):
  • python setup.py bdist_wheel - OK
  • python setup.py sdist then python setup.py bdist_wheel - OK
  • python setup.py sdist bdist_wheel - BROKEN

Steps to reproduce

First:

• Run git clean -fdx -- remove any generated files
• Run python setup.py sdist bdist_wheel
• Observe only the following line in the output:
  • copying build/lib/mypkg/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg
• Run unzip -l dist/mypkg-1.2.3-py2-none-any.whl
  • Observe that mypkg/generated_data is missing from the wheel

Now that mypkg/generated_data is present:

• Run python setup.py sdist bdist_wheel again
• Observe the following lines in the output:
  • copying build/lib/mypkg/generated_data -> build/bdist.linux-x86_64/wheel/mypkg
  • copying build/lib/mypkg/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg
• Run unzip -l dist/mypkg-1.2.3-py2-none-any.whl
  • Observe that mypkg/generated_data is now present in the wheel

Clean the directory, and run other variations of sdist and bdist_wheel (as mentioned above) and observe that the problem does not manifest.

Workarounds that do not work

• Hooking build_py instead of build

Solution

Setting include_package_data = True caused setup.py to behave as intended.

Note that the setuptools documentation is not particularly clear on why this works:

include_package_data If set to True, this tells setuptools to automatically include any data files it finds inside your package directories that are specified by your MANIFEST.in file. For more information, see the section below on Including Data Files.

package_data A dictionary mapping package names to lists of glob patterns. For a complete description and examples, see the section below on Including Data Files. You do not need to use this option if you are using include_package_data, unless you need to add e.g. files that are generated by your setup script and build process. (And are therefore not in source control or are files that you don’t want to include in your source distribution.)

...

In summary, ...

include_package_data Accept all data files and directories matched by MANIFEST.in.

package_data Specify additional patterns to match files and directories that may or may not be matched by MANIFEST.in or found in source control.

Nothing in this documentation describes the behavior I was seeing, however. In my case, it seemed like the files discovered by sdist (using MANIFEST.in) somehow overrode those that bdist_wheel would normally find.

For whatever reason, specifying include_package_data seems to have two, somewhat inverted purposes.