feat: Development dependencies are never copied in any case

You don't need to ignore it explicitly anymore
electron-userland · Jun 8, 2016 · 6d4ab11 · 6d4ab11
1 parent fc1587f
commit 6d4ab11
Showing 9 changed files with 160 additions and 113 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -1,4 +1,4 @@
-osx_image: xcode7
+osx_image: xcode7.3
 
 matrix:
   include:
@@ -13,7 +13,6 @@ language: c
 cache:
   directories:
     - node_modules
-    - test/testApp/node_modules
     - $HOME/.electron
     - $HOME/.cache/fpm
 
@@ -23,6 +22,7 @@ before_install:
 
 install:
 - nvm install $NODE_VERSION
+- nvm use --delete-prefix $NODE_VERSION
 - if [[ "$TRAVIS_OS_NAME" == "osx" && "$NODE_VERSION" == "4" ]]; then npm install npm -g ; fi
 - npm install
 - npm prune

diff --git a/docs/Options.md b/docs/Options.md
@@ -54,7 +54,7 @@ Here documented only `electron-builder` specific options:
 | app-category-type | <a name="BuildMetadata-app-category-type"></a><p>*OS X-only.* The application category type, as shown in the Finder via *View -&gt; Arrange by Application Category* when viewing the Applications directory.</p> <p>For example, <code>app-category-type=public.app-category.developer-tools</code> will set the application category to *Developer Tools*.</p> <p>Valid values are listed in [Apple’s documentation](https://developer.apple.com/library/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/LaunchServicesKeys.html#//apple_ref/doc/uid/TP40009250-SW8).</p>
 | asar | <a name="BuildMetadata-asar"></a><p>Whether to package the application’s source code into an archive, using [Electron’s archive format](https://github.com/electron/asar). Defaults to <code>true</code>. Reasons why you may want to disable this feature are described in [an application packaging tutorial in Electron’s documentation](http://electron.atom.io/docs/latest/tutorial/application-packaging/#limitations-on-node-api/).</p> <p>Or you can pass object of any asar options.</p>
 | productName | <a name="BuildMetadata-productName"></a>See [AppMetadata.productName](#AppMetadata-productName).
-| files | <a name="BuildMetadata-files"></a><p>A [glob patterns](https://www.npmjs.com/package/glob#glob-primer) relative to the [app directory](#MetadataDirectories-app), which specifies which files to include when copying files to create the package. Defaults to <code>\*\*\/\*</code> (i.e. [hidden files are ignored by default](https://www.npmjs.com/package/glob#dots)).</p> <p>[Multiple patterns](#multiple-glob-patterns) are supported. You can use <code>${os}</code> (expanded to osx, linux or win according to current platform) and <code>${arch}</code> in the pattern.</p> <p>If directory matched, all contents are copied. So, you can just specify <code>foo</code> to copy <code>foo</code> directory.</p> <p>Remember that default pattern <code>\*\*\/\*</code> is not added to your custom, so, you have to add it explicitly — e.g. <code>[&quot;\*\*\/\*&quot;, &quot;!ignoreMe${/\*}&quot;]</code>.</p> <p>May be specified in the platform options (e.g. in the <code>build.osx</code>).</p>
+| files | <a name="BuildMetadata-files"></a><p>A [glob patterns](https://www.npmjs.com/package/glob#glob-primer) relative to the [app directory](#MetadataDirectories-app), which specifies which files to include when copying files to create the package. Defaults to <code>\*\*\/\*</code> (i.e. [hidden files are ignored by default](https://www.npmjs.com/package/glob#dots)).</p> <p>Development dependencies are never copied in any case. You don’t need to ignore it explicitly.</p> <p>[Multiple patterns](#multiple-glob-patterns) are supported. You can use <code>${os}</code> (expanded to osx, linux or win according to current platform) and <code>${arch}</code> in the pattern. If directory matched, all contents are copied. So, you can just specify <code>foo</code> to copy <code>foo</code> directory.</p> <p>Remember that default pattern <code>\*\*\/\*</code> is not added to your custom, so, you have to add it explicitly — e.g. <code>[&quot;\*\*\/\*&quot;, &quot;!ignoreMe${/\*}&quot;]</code>.</p> <p>May be specified in the platform options (e.g. in the <code>build.osx</code>).</p>
 | extraResources | <a name="BuildMetadata-extraResources"></a><p>A [glob patterns](https://www.npmjs.com/package/glob#glob-primer) relative to the project directory, when specified, copy the file or directory with matching names directly into the app’s resources directory (<code>Contents/Resources</code> for OS X, <code>resources</code> for Linux/Windows).</p> <p>Glob rules the same as for [files](#BuildMetadata-files).</p>
 | extraFiles | <a name="BuildMetadata-extraFiles"></a>The same as [extraResources](#BuildMetadata-extraResources) but copy into the app's content directory (`Contents` for OS X, root directory for Linux/Windows).
 | osx | <a name="BuildMetadata-osx"></a>See [.build.osx](#OsXBuildOptions).
@@ -63,8 +63,6 @@ Here documented only `electron-builder` specific options:
 | linux | <a name="BuildMetadata-linux"></a>See [.build.linux](#LinuxBuildOptions).
 | compression | <a name="BuildMetadata-compression"></a>The compression level, one of `store`, `normal`, `maximum` (default: `normal`). If you want to rapidly test build, `store` can reduce build time significantly.
 | afterPack | <a name="BuildMetadata-afterPack"></a>*programmatic API only* The function to be run after pack (but before pack into distributable format and sign). Promise must be returned.
-| npmPrune | <a name="BuildMetadata-npmPrune"></a><p>Whether to [prune](https://docs.npmjs.com/cli/prune) native dependencies (<code>npm prune --production</code>) before starting to package the app. Defaults to <code>true</code> if [two package.json structure](https://github.com/electron-userland/electron-builder#two-packagejson-structure) is not used.</p>
-| npmRebuild | <a name="BuildMetadata-npmRebuild"></a>Whether to [rebuild](https://docs.npmjs.com/cli/rebuild) native dependencies (`npm rebuild`) before starting to package the app. Defaults to `true`.
 
 <a name="OsXBuildOptions"></a>
 ### `.build.osx`

diff --git a/package.json b/package.json
@@ -107,7 +107,7 @@
     "pre-git": "^3.8.4",
     "semantic-release": "^6.3.0",
     "should": "^9.0.0",
-    "ts-babel": "^1.0.0",
+    "ts-babel": "^1.0.2",
     "tsconfig-glob": "^0.4.3",
     "tslint": "3.10.0-dev.2",
     "typescript": "1.9.0-dev.20160607-1.0",

diff --git a/src/index.ts b/src/index.ts
@@ -2,4 +2,4 @@ export { Packager } from "./packager"
 export { PackagerOptions, ArtifactCreated, DIR_TARGET, BuildInfo } from "./platformPackager"
 export { BuildOptions, build, createPublisher, CliOptions, createTargets } from "./builder"
 export { PublishOptions, Publisher } from "./gitHubPublisher"
-export { AppMetadata, DevMetadata, Platform, Arch, archFromString, getProductName, BuildMetadata, OsXBuildOptions, WinBuildOptions, LinuxBuildOptions } from "./metadata"
+export { AppMetadata, DevMetadata, Platform, Arch, archFromString, getProductName, BuildMetadata, OsXBuildOptions, WinBuildOptions, LinuxBuildOptions, CompressionLevel } from "./metadata"
diff --git a/src/metadata.ts b/src/metadata.ts
@@ -74,6 +74,8 @@ export interface AuthorMetadata {
   readonly email: string
 }
 
+export type CompressionLevel = "store" | "normal" | "maximum"
+
 /*
  ## `.build`
  */
@@ -156,7 +158,7 @@ export interface BuildMetadata {
   /*
    The compression level, one of `store`, `normal`, `maximum` (default: `normal`). If you want to rapidly test build, `store` can reduce build time significantly.
    */
-  readonly compression?: "store" | "normal" | "maximum" | null
+  readonly compression?: CompressionLevel | null
 
   readonly "build-version"?: string | null
 
@@ -171,7 +173,7 @@ export interface BuildMetadata {
   //  */
   // readonly npmPrune?: boolean
   // deprecated
-  readonly prune?: boolean
+  // readonly prune?: boolean
 
   /*
    Whether to [rebuild](https://docs.npmjs.com/cli/rebuild) native dependencies (`npm rebuild`) before starting to package the app. Defaults to `true`.

diff --git a/src/platformPackager.ts b/src/platformPackager.ts
@@ -4,30 +4,18 @@ import EventEmitter = NodeJS.EventEmitter
 import { Promise as BluebirdPromise } from "bluebird"
 import * as path from "path"
 import { pack, ElectronPackagerOptions, userIgnoreFilter } from "electron-packager-tf"
-import { readdir, copy, unlink, lstat, remove } from "fs-extra-p"
-import { statOrNull, use, spawn, debug7zArgs, debug, warn, log, spawnNpmProduction } from "./util"
+import { readdir, copy, unlink, lstat, remove, realpath } from "fs-extra-p"
+import { statOrNull, use, warn, log, exec } from "./util"
 import { Packager } from "./packager"
 import { listPackage, statFile, AsarFileMetadata, createPackageFromFiles, AsarOptions } from "asar"
-import { path7za } from "7zip-bin"
+import { archiveApp } from "./targets/archive"
 import { Glob } from "glob"
 import { Minimatch } from "minimatch"
 import deepAssign = require("deep-assign")
 
 //noinspection JSUnusedLocalSymbols
 const __awaiter = require("./awaiter")
 
-class CompressionDescriptor {
-  constructor(public flag: string, public env: string, public minLevel: string, public maxLevel: string = "-9") {
-  }
-}
-
-const extToCompressionDescriptor: { [key: string]: CompressionDescriptor; } = {
-  "tar.xz": new CompressionDescriptor("--xz", "XZ_OPT", "-0", "-9e"),
-  "tar.lz": new CompressionDescriptor("--lzip", "LZOP", "-0"),
-  "tar.gz": new CompressionDescriptor("--gz", "GZIP", "-1"),
-  "tar.bz2": new CompressionDescriptor("--bzip2", "BZIP2", "-1"),
-}
-
 export const commonTargets = ["dir", "zip", "7z", "tar.xz", "tar.lz", "tar.gz", "tar.bz2"]
 
 export const DIR_TARGET = "dir"
@@ -168,18 +156,32 @@ export abstract class PlatformPackager<DC extends PlatformSpecificBuildOptions>
         promise = copy(this.info.appDir, appPath, {filter: userIgnoreFilter(opts), dereference: true})
       }
       else {
+        const ignoreFiles = new Set([path.relative(this.info.appDir, opts.out!), path.relative(this.info.appDir, this.buildResourcesDir)])
+        if (!this.info.isTwoPackageJsonProjectLayoutUsed) {
+          const result = await BluebirdPromise.all([listDependencies(this.info.appDir, false), listDependencies(this.info.appDir, true)])
+          const productionDepsSet = new Set(result[1])
+
+          // npm returns real path, so, we should use relative path to avoid any mismatch
+          const realAppDirPath = await realpath(this.info.appDir)
+
+          for (let it of result[0]) {
+            if (!productionDepsSet.has(it)) {
+              if (it.startsWith(realAppDirPath)) {
+                it = it.substring(realAppDirPath.length + 1)
+              }
+              else if (it.startsWith(this.info.appDir)) {
+                it = it.substring(this.info.appDir.length + 1)
+              }
+              ignoreFiles.add(it)
+            }
+          }
+        }
+
         let patterns = this.getFilePatterns("files", customBuildOptions)
         if (patterns == null || patterns.length === 0) {
           patterns = ["**/*"]
         }
-
-        const parsedPatterns = this.getParsedPatterns(patterns, arch)
-        if (!this.info.isTwoPackageJsonProjectLayoutUsed) {
-          const dotOptions = {dot: true}
-          parsedPatterns.push(new Minimatch("!node_modules/@(appdmg|electron-download|electron-builder|electron-prebuilt|electron-packager-tf|electron-winstaller-fixed|electron-osx-sign-tf|electron-osx-sign){,/**/*}", dotOptions))
-          parsedPatterns.push(new Minimatch(`!@(${path.relative(this.info.appDir, this.buildResourcesDir)}|${path.relative(this.info.appDir, opts.out!)}){,/**/*}`, dotOptions))
-        }
-        promise = copyFiltered(this.info.appDir, appPath, parsedPatterns, true)
+        promise = copyFiltered(this.info.appDir, appPath, this.getParsedPatterns(patterns, arch), true, ignoreFiles)
       }
 
       const promises = [promise]
@@ -193,24 +195,8 @@ export abstract class PlatformPackager<DC extends PlatformSpecificBuildOptions>
 
       await BluebirdPromise.all(promises)
 
-      let npmPrune = this.devMetadata.build.npmPrune
-      if (npmPrune == null) {
-        npmPrune = this.devMetadata.build.prune
-        if (npmPrune != null) {
-          warn("prune is deprecated and renamed to npmPrune, please specify as npmPrune")
-        }
-      }
-
-      if (npmPrune == null) {
-        npmPrune = !this.info.isTwoPackageJsonProjectLayoutUsed
-      }
-      else if (typeof npmPrune !== "boolean") {
-        throw new Error(`npmPrune expected to be boolean value, but string '"${npmPrune}"' was specified`)
-      }
-
-      if (npmPrune) {
-        log("Pruning app dependencies")
-        await spawnNpmProduction("prune", appPath)
+      if (opts.prune != null) {
+        warn("prune is deprecated — development dependencies are never copied in any case")
       }
 
       if (asarOptions != null) {
@@ -469,66 +455,7 @@ export abstract class PlatformPackager<DC extends PlatformSpecificBuildOptions>
   }
 
   protected async archiveApp(format: string, appOutDir: string, outFile: string): Promise<any> {
-    const compression = this.devMetadata.build.compression
-    const storeOnly = compression === "store"
-
-    const dirToArchive = this.platform === Platform.OSX ? path.join(appOutDir, `${this.appName}.app`) : appOutDir
-    if (format.startsWith("tar.")) {
-      // we don't use 7z here - develar: I spent a lot of time making pipe working - but it works on OS X and often hangs on Linux (even if use pipe-io lib)
-      // and in any case it is better to use system tools (in the light of docker - it is not problem for user because we provide complete docker image).
-      const info = extToCompressionDescriptor[format]
-      let tarEnv = process.env
-      if (compression != null && compression !== "normal") {
-        tarEnv = Object.assign({}, process.env)
-        tarEnv[info.env] = storeOnly ? info.minLevel : info.maxLevel
-      }
-
-      await spawn(process.platform === "darwin" || process.platform === "freebsd" ? "gtar" : "tar", [info.flag, "--transform", `s,^\.,${path.basename(outFile, "." + format)},`, "-cf", outFile, "."], {
-        cwd: dirToArchive,
-        stdio: ["ignore", debug.enabled ? "inherit" : "ignore", "inherit"],
-        env: tarEnv
-      })
-      return
-    }
-
-    const args = debug7zArgs("a")
-    if (compression === "maximum") {
-      if (format === "7z" || format.endsWith(".7z")) {
-        args.push("-mx=9", "-mfb=64", "-md=32m", "-ms=on")
-      }
-      else if (format === "zip") {
-        // http://superuser.com/a/742034
-        //noinspection SpellCheckingInspection
-        args.push("-mfb=258", "-mpass=15")
-      }
-      else {
-        args.push("-mx=9")
-      }
-    }
-    else if (storeOnly) {
-      if (format !== "zip") {
-        args.push("-mx=1")
-      }
-    }
-
-    // remove file before - 7z doesn't overwrite file, but update
-    try {
-      await unlink(outFile)
-    }
-    catch (e) {
-      // ignore
-    }
-
-    if (format === "zip" || storeOnly) {
-      args.push("-mm=" + (storeOnly ? "Copy" : "Deflate"))
-    }
-
-    args.push(outFile, dirToArchive)
-
-    await spawn(path7za, args, {
-      cwd: path.dirname(dirToArchive),
-      stdio: ["ignore", debug.enabled ? "inherit" : "ignore", "inherit"],
-    })
+    return archiveApp(this.devMetadata.build.compression, format, outFile, this.platform === Platform.OSX ? path.join(appOutDir, `${this.appName}.app`) : appOutDir)
   }
 }
 
@@ -592,15 +519,21 @@ function minimatchAll(path: string, patterns: Array<Minimatch>): boolean {
   return match
 }
 
-function copyFiltered(src: string, destination: string, patterns: Array<Minimatch>, dereference: boolean = false): Promise<any> {
+// we use relative path to avoid canonical path issue - e.g. /tmp vs /private/tmp
+function copyFiltered(src: string, destination: string, patterns: Array<Minimatch>, dereference: boolean = false, ignoreFiles?: Set<string>): Promise<any> {
   return copy(src, destination, {
     dereference: dereference,
     filter: it => {
       if (src === it) {
         return true
       }
-
       let relative = it.substring(src.length + 1)
+
+      // yes, check before path sep normalization
+      if (ignoreFiles != null && ignoreFiles.has(relative)) {
+        return false
+      }
+
       if (path.sep === "\\") {
         relative = relative.replace(/\\/g, "/")
       }
@@ -612,4 +545,29 @@ function copyFiltered(src: string, destination: string, patterns: Array<Minimatc
 export function computeEffectiveTargets(rawList: Array<string>, targetsFromMetadata: Array<string> | n): Array<string> {
   let targets = normalizeTargets(rawList.length === 0 ? targetsFromMetadata : rawList)
   return targets == null ? ["default"] : targets
+}
+
+async function listDependencies(appDir: string, production: boolean): Promise<Array<string>> {
+  let npmExecPath = process.env.npm_execpath || process.env.NPM_CLI_JS
+  const npmExecArgs = ["ls", production ? "--production" : "--dev", "--parseable"]
+  if (npmExecPath == null) {
+    npmExecPath = process.platform === "win32" ? "npm.cmd" : "npm"
+  }
+  else {
+    npmExecArgs.unshift(npmExecPath)
+    npmExecPath = process.env.npm_node_execpath || process.env.NODE_EXE || "node"
+  }
+
+  const result = (await exec(npmExecPath, npmExecArgs, {
+    cwd: appDir,
+    stdio: "inherit",
+    maxBuffer: 1024 * 1024,
+  })).trim().split("\n")
+  if (result.length > 0 && !result[0].includes("/node_modules/")) {
+    // first line is a project dir
+    const lastIndex = result.length - 1
+    result[0] = result[lastIndex]
+    result.length = result.length - 1
+  }
+  return result
 }
diff --git a/src/targets/archive.ts b/src/targets/archive.ts
@@ -0,0 +1,81 @@
+import { spawn, debug, debug7zArgs } from "../util"
+import { CompressionLevel } from "../metadata"
+import * as path from "path"
+import { unlink } from "fs-extra-p"
+import { path7za } from "7zip-bin"
+
+//noinspection JSUnusedLocalSymbols
+const __awaiter = require("../awaiter")
+
+class CompressionDescriptor {
+  constructor(public flag: string, public env: string, public minLevel: string, public maxLevel: string = "-9") {
+  }
+}
+
+const extToCompressionDescriptor: { [key: string]: CompressionDescriptor; } = {
+  "tar.xz": new CompressionDescriptor("--xz", "XZ_OPT", "-0", "-9e"),
+  "tar.lz": new CompressionDescriptor("--lzip", "LZOP", "-0"),
+  "tar.gz": new CompressionDescriptor("--gz", "GZIP", "-1"),
+  "tar.bz2": new CompressionDescriptor("--bzip2", "BZIP2", "-1"),
+}
+
+export async function archiveApp(compression: CompressionLevel | n, format: string, outFile: string, dirToArchive: string): Promise<any> {
+  const storeOnly = compression === "store"
+
+  if (format.startsWith("tar.")) {
+    // we don't use 7z here - develar: I spent a lot of time making pipe working - but it works on OS X and often hangs on Linux (even if use pipe-io lib)
+    // and in any case it is better to use system tools (in the light of docker - it is not problem for user because we provide complete docker image).
+    const info = extToCompressionDescriptor[format]
+    let tarEnv = process.env
+    if (compression != null && compression !== "normal") {
+      tarEnv = Object.assign({}, process.env)
+      tarEnv[info.env] = storeOnly ? info.minLevel : info.maxLevel
+    }
+
+    await spawn(process.platform === "darwin" || process.platform === "freebsd" ? "gtar" : "tar", [info.flag, "--transform", `s,^\.,${path.basename(outFile, "." + format)},`, "-cf", outFile, "."], {
+      cwd: dirToArchive,
+      stdio: ["ignore", debug.enabled ? "inherit" : "ignore", "inherit"],
+      env: tarEnv
+    })
+    return
+  }
+
+  const args = debug7zArgs("a")
+  if (compression === "maximum") {
+    if (format === "7z" || format.endsWith(".7z")) {
+      args.push("-mx=9", "-mfb=64", "-md=32m", "-ms=on")
+    }
+    else if (format === "zip") {
+      // http://superuser.com/a/742034
+      //noinspection SpellCheckingInspection
+      args.push("-mfb=258", "-mpass=15")
+    }
+    else {
+      args.push("-mx=9")
+    }
+  }
+  else if (storeOnly) {
+    if (format !== "zip") {
+      args.push("-mx=1")
+    }
+  }
+
+  // remove file before - 7z doesn't overwrite file, but update
+  try {
+    await unlink(outFile)
+  }
+  catch (e) {
+    // ignore
+  }
+
+  if (format === "zip" || storeOnly) {
+    args.push("-mm=" + (storeOnly ? "Copy" : "Deflate"))
+  }
+
+  args.push(outFile, dirToArchive)
+
+  await spawn(path7za, args, {
+    cwd: path.dirname(dirToArchive),
+    stdio: ["ignore", debug.enabled ? "inherit" : "ignore", "inherit"],
+  })
+}
diff --git a/test/src/globTest.ts b/test/src/globTest.ts
@@ -58,7 +58,14 @@ test.ifDevOrLinuxCi("ignore node_modules known dev dep", () => {
     }
   }, {
     tempDirCreated: projectDir => {
-      return outputFile(path.join(projectDir, "node_modules", "electron-osx-sign", "foo.js"), "")
+      return BluebirdPromise.all([
+        modifyPackageJson(projectDir, data => {
+          data.devDependencies = Object.assign({
+              "electron-osx-sign": "*",
+            }, data.devDependencies)
+        }),
+        outputFile(path.join(projectDir, "node_modules", "electron-osx-sign", "package.json"), "{}"),
+      ])
     },
     packed: projectDir => {
       return assertThat(path.join(projectDir, outDirName, "linux", "resources", "app", "node_modules", "electron-osx-sign")).doesNotExist()

diff --git a/tsconfig.json b/tsconfig.json
@@ -77,6 +77,7 @@
     "src/platformPackager.ts",
     "src/promise.ts",
     "src/repositoryInfo.ts",
+    "src/targets/archive.ts",
     "src/targets/squirrelWindows.ts",
     "src/util.ts",
     "src/winPackager.ts"