diff --git a/README.md b/README.md deleted file mode 100644 index 618c6e0c..00000000 --- a/README.md +++ /dev/null @@ -1,861 +0,0 @@ -# Apify command-line interface (Apify CLI) - -npm version -Build Status - -Apify command-line interface (Apify CLI) helps you create, develop, build and run -[Apify Actors](https://www.apify.com/actors), -and manage the Apify cloud platform from any computer. - -Apify Actors are cloud programs that can perform arbitrary web scraping, automation or data processing job. -They accept input, perform their job and generate output. -While you can develop Actors in an online IDE directly in the [Apify web application](https://console.apify.com/), -for complex projects it is more convenient to develop Actors locally on your computer -using Apify SDK -and only push the Actors to the Apify cloud during deployment. -This is where the Apify CLI comes in. - -Note that Actors running on the Apify platform are executed in Docker containers, so with an appropriate `Dockerfile` -you can build your Actors in any programming language. -However, we recommend using JavaScript / Node.js, for which we provide most libraries and support. - -## Installation - -### Via Homebrew - -On macOS (or Linux), you can install the Apify CLI via the [Homebrew package manager](https://brew.sh). - -```bash -brew install apify-cli -``` - -### Via NPM - -First, make sure you have [Node.js](https://nodejs.org) version 16 or higher with NPM installed on your computer: - -```bash -node --version -npm --version -``` - -Install or upgrade Apify CLI by running: - -```bash -npm -g install apify-cli -``` - -If you receive an `EACCES` error, you might need to run the command as root: - -```bash -sudo npm -g install apify-cli -``` - -Alternatively, you can use [Node Version Manager (nvm)](https://github.com/nvm-sh/nvm) and install Apify CLI only into a selected user-level Node version without requiring root privileges: - -``` -nvm install 16 -nvm use 16 -npm -g install apify-cli -``` - -Finally, verify that Apify CLI was installed correctly by running: - -```bash -apify --version -``` - -which should print something like: - -``` -apify-cli/0.10.0 darwin-x64 node-v16.14.2 -``` - -> You can also skip the manual global installation altogether and use `npx apify-cli` with all the following commands instead. - -## Basic usage - -The following examples demonstrate the basic usage of Apify CLI. - -### Create a new Actor from scratch - -```bash -apify create my-hello-world -``` - -First, you will be prompted to select a template with the boilerplate for the Actor, to help you get started quickly. -The command will create a directory called `my-hello-world` that contains a Node.js project -for the Actor and a few configuration files. - -> If you decided to skip the installation and go with `npx`, the command will be `npx apify-cli create my-hello-world`. - -### Create a new Actor from existing project - -```bash -cd ./my/awesome/project -apify init -``` - -This command will only set up local Actor development environment in an existing directory, -i.e. it will create the `.actor/actor.json` file and `apify_storage` directory. - -Before you can run your project locally using `apify run`, you have to set up the right start command in `package.json` under scripts.start. For example: - -```text -{ - ... - "scripts": { - "start": "node your_main_file.js", - }, - ... -} -``` - -You can find more information about by running `apify help run`. - -### Create a new Actor from Scrapy project - -If you want to run a Scrapy project on Apify platform, follow the Scrapy integration guide [here](https://docs.apify.com/cli/docs/integrating-scrapy). - -### Run the Actor locally - -```bash -cd my-hello-world -apify run -``` - -This command runs the Actor on your local machine. -Now's your chance to develop the logic - or magic :smirk: - -### Login with your Apify account - -```bash -apify login -``` - -Before you can interact with the Apify cloud, you need to [create an Apify account](https://console.apify.com/) -and log in to it using the above command. You will be prompted for -your [Apify API token](https://console.apify.com/settings/integrations). -Note that the command will store the API token and other sensitive information to `~/.apify`. - -### Push the Actor to the Apify cloud - -```bash -apify push -``` - -This command uploads your project to the Apify cloud and builds an Actor from it. On the platform, Actor needs to be built before it can be run. - -### Run an Actor on the Apify cloud - -```bash -apify call -``` - -Runs the Actor corresponding to the current directory on the Apify platform. - -This command can also be used to run other Actors, for example: - -```bash -apify call apify/hello-world -``` - -### So what's in this .actor/actor.json file? - -This file associates your local development project with an Actor on the Apify platform. -It contains information such as Actor name, version, build tag and environment variables. -Make sure you commit this file to the Git repository. - -For example, `.actor/actor.json` file can look as follows: - -```json -{ - "actorSpecification": 1, - "name": "name-of-my-scraper", - "version": "0.0", - "buildTag": "latest", - "environmentVariables": { - "MYSQL_USER": "my_username", - "MYSQL_PASSWORD": "@mySecretPassword" - }, - "dockerfile": "./Dockerfile", - "readme": "./ACTOR.md", - "input": "./input_schema.json", - "storages": { - "dataset": "./dataset_schema.json" - } -} -``` - -**`Dockerfile` field**\ -If you specify the path to your Docker file under the `dockerfile` field, this file will be used for Actor builds on the platform. If not specified, the system will look for Docker files at `.actor/Dockerfile` and `Dockerfile` in this order of preference. - -**`Readme` field** \ -If you specify the path to your readme file under the `readme` field, the readme at this path will be used on the platform. If not specified, readme at `.actor/README.md` and `README.md` will be used in this order of preference. - -**`Input` field**\ -You can embed your [input schema](https://docs.apify.com/actors/development/input-schema#specification-version-1) object directly in `actor.json` under `input` field. Alternatively, you can provide a path to a custom input schema. If not provided, the input schema at `.actor/INPUT_SCHEMA.json` and `INPUT_SCHEMA.json` is used in this order of preference. - -**`Storages.dataset` field**\ -You can define the schema of the items in your dataset under the `storages.dataset` field. This can be either an embedded object or a path to a JSON schema file. You can read more about the schema of your Actor output [here](https://docs.apify.com/actors/development/output-schema#specification-version-1). - -**Note on migration from deprecated config "apify.json"**\ -_Note that previously, Actor config was stored in the `apify.json` file that has been deprecated. You can find the (very slight) differences and migration info in [migration guidelines](https://github.com/apify/apify-cli/blob/master/MIGRATIONS.md)._ - -## Environment variables - -There are two options how you can set up environment variables for Actors. - -### Set up environment variables in .actor/actor.json - -All keys from `env` will be set as environment variables into Apify platform after you push Actor to Apify. Current values on Apify will be overridden. - -```json -{ - "actorSpecification": 1, - "name": "dataset-to-mysql", - "version": "0.1", - "buildTag": "latest", - "environmentVariables": { - "MYSQL_USER": "my_username", - "MYSQL_PASSWORD": "@mySecretPassword" - } -} -``` - -### Set up environment variables in Apify Console - -In [Apify Console](https://console.apify.com/actors) select your Actor, you can set up variables into Source tab. -After setting up variables in the app, remove the `environmentVariables` from `.actor/actor.json`. Otherwise, variables from `.actor/actor.json` will override variables in the app. - -```json -{ - "actorSpecification": 1, - "name": "dataset-to-mysql", - "version": "0.1", - "buildTag": "latest" -} -``` - -#### How to set secret environment variables in .actor/actor.json - -CLI provides commands to manage secrets environment variables. Secrets are stored to the `~/.apify` directory. -You can add a new secret using the command: - -```bash -apify secrets:add mySecretPassword pwd1234 -``` - -After adding a new secret you can use the secret in `.actor/actor.json`. - -```text -{ - "actorSpecification": 1, - "name": "dataset-to-mysql", - ... - "environmentVariables": { - "MYSQL_PASSWORD": "@mySecretPassword" - }, - ... -} -``` - -### Need help? - -To see all CLI commands simply run: - -```bash -apify help -``` - -To get information about a specific command run: - -```bash -apify help COMMAND -``` - -Still haven't found what you were looking for? Please go to [Apify Help center](https://www.apify.com/help) -or [contact us](https://www.apify.com/contact). - -## Command reference - -This section contains printouts of `apify help` for all commands. - - - -* [`apify actor`](#apify-actor) -* [`apify actor get-input`](#apify-actor-get-input) -* [`apify actor get-value KEY`](#apify-actor-get-value-key) -* [`apify actor push-data [ITEM]`](#apify-actor-push-data-item) -* [`apify actor set-value KEY [VALUE]`](#apify-actor-set-value-key-value) -* [`apify actors`](#apify-actors) -* [`apify call [ACTORID]`](#apify-call-actorid) -* [`apify create [ACTORNAME]`](#apify-create-actorname) -* [`apify datasets`](#apify-datasets) -* [`apify help [COMMAND]`](#apify-help-command) -* [`apify info`](#apify-info) -* [`apify init [ACTORNAME]`](#apify-init-actorname) -* [`apify key-value-stores`](#apify-key-value-stores) -* [`apify login`](#apify-login) -* [`apify logout`](#apify-logout) -* [`apify pull [ACTORID]`](#apify-pull-actorid) -* [`apify push [ACTORID]`](#apify-push-actorid) -* [`apify request-queues`](#apify-request-queues) -* [`apify run`](#apify-run) -* [`apify runs`](#apify-runs) -* [`apify secrets`](#apify-secrets) -* [`apify secrets add NAME VALUE`](#apify-secrets-add-name-value) -* [`apify secrets rm NAME`](#apify-secrets-rm-name) -* [`apify task`](#apify-task) -* [`apify task run TASKID`](#apify-task-run-taskid) -* [`apify validate-schema [PATH]`](#apify-validate-schema-path) - -## `apify actor` - -Commands are designed to be used in Actor runs. All commands are in PoC state, do not use in production environments. - -``` -USAGE - $ apify actor - -DESCRIPTION - Commands are designed to be used in Actor runs. All commands are in PoC state, do not use in production environments. -``` - -_See code: [src/commands/actor/index.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/actor/index.ts)_ - -## `apify actor get-input` - -Gets the Actor input value from the default key-value store associated with the Actor run. - -``` -USAGE - $ apify actor get-input - -DESCRIPTION - Gets the Actor input value from the default key-value store associated with the Actor run. -``` - -_See code: [src/commands/actor/get-input.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/actor/get-input.ts)_ - -## `apify actor get-value KEY` - -Gets a value from the default key-value store associated with the Actor run. - -``` -USAGE - $ apify actor get-value KEY - -ARGUMENTS - KEY Key of the record in key-value store - -DESCRIPTION - Gets a value from the default key-value store associated with the Actor run. -``` - -_See code: [src/commands/actor/get-value.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/actor/get-value.ts)_ - -## `apify actor push-data [ITEM]` - -Stores an object or an array of objects to the default dataset of the Actor run. - -``` -USAGE - $ apify actor push-data [ITEM] - -ARGUMENTS - ITEM JSON string with one object or array of objects containing data to be stored in the default dataset. - -DESCRIPTION - Stores an object or an array of objects to the default dataset of the Actor run. - It is possible to pass data using item argument or stdin. - Passing data using argument: - $ apify actor push-data {"foo": "bar"} - Passing data using stdin with pipe: - $ cat ./test.json | apify actor push-data -``` - -_See code: [src/commands/actor/push-data.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/actor/push-data.ts)_ - -## `apify actor set-value KEY [VALUE]` - -Sets or removes record into the default KeyValueStore associated with the Actor run. - -``` -USAGE - $ apify actor set-value KEY [VALUE] [-c ] - -ARGUMENTS - KEY Key of the record in key-value store. - VALUE Record data, which can be one of the following values: - - If empty, the record in the key-value store is deleted. - - If no `contentType` flag is specified, value is expected to be any JSON string value. - - If options.contentType is set, value is taken as is. - -FLAGS - -c, --contentType= Specifies a custom MIME content type of the record. By default "application/json" is used. - -DESCRIPTION - Sets or removes record into the default KeyValueStore associated with the Actor run. - It is possible to pass data using argument or stdin. - Passing data using argument: - $ apify actor set-value KEY my-value - Passing data using stdin with pipe: - $ cat ./my-text-file.txt | apify actor set-value KEY --contentType text/plain -``` - -_See code: [src/commands/actor/set-value.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/actor/set-value.ts)_ - -## `apify actors` - -Commands are designed to be used with Actors. - -``` -USAGE - $ apify actors - -DESCRIPTION - Commands are designed to be used with Actors. -``` - -_See code: [src/commands/actors/index.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/actors/index.ts)_ - -## `apify call [ACTORID]` - -Runs a specific Actor remotely on the Apify cloud platform. - -``` -USAGE - $ apify call [ACTORID] [-b ] [-t ] [-m ] [-w ] [-i | --input-file - ] [-s] [-o] - -ARGUMENTS - ACTORID Name or ID of the Actor to run (e.g. "my-actor", "apify/hello-world" or "E2jjCZBezvAZnX8Rb"). If not - provided, the command runs the remote Actor specified in the ".actor/actor.json" file. - -FLAGS - -b, --build= Tag or number of the build to run (e.g. "latest" or "1.2.34"). - -i, --input= Optional JSON input to be given to the Actor. - -m, --memory= Amount of memory allocated for the Actor run, in megabytes. - -o, --output-dataset Prints out the entire default dataset on successful run of the Actor. - -s, --silent Prevents printing the logs of the Actor run to the console. - -t, --timeout= Timeout for the Actor run in seconds. Zero value means there is no timeout. - -w, --wait-for-finish= Seconds for waiting to run to finish, if no value passed, it waits forever. - --input-file= Optional path to a file with JSON input to be given to the Actor. The file must be a - valid JSON file. You can also specify `-` to read from standard input. - -DESCRIPTION - Runs a specific Actor remotely on the Apify cloud platform. - The Actor is run under your current Apify account. Therefore you need to be logged in by calling "apify login". It - takes input for the Actor from the default local key-value store by default. -``` - -_See code: [src/commands/call.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/call.ts)_ - -## `apify create [ACTORNAME]` - -Creates a new Actor project directory from a selected boilerplate template. - -``` -USAGE - $ apify create [ACTORNAME] [-t ] [--skip-dependency-install] [--omit-optional-deps] - -ARGUMENTS - ACTORNAME Name of the Actor and its directory - -FLAGS - -t, --template= Template for the Actor. If not provided, the command will prompt for it. - Visit - https://raw.githubusercontent.com/apify/actor-templates/master/templates/manifest.json - to find available template names. - --omit-optional-deps Skip installing optional dependencies. - --skip-dependency-install Skip installing Actor dependencies. - -DESCRIPTION - Creates a new Actor project directory from a selected boilerplate template. -``` - -_See code: [src/commands/create.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/create.ts)_ - -## `apify datasets` - -Commands are designed to be used with Datasets. - -``` -USAGE - $ apify datasets - -DESCRIPTION - Commands are designed to be used with Datasets. -``` - -_See code: [src/commands/datasets/index.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/datasets/index.ts)_ - -## `apify help [COMMAND]` - -Display help for apify. - -``` -USAGE - $ apify help [COMMAND...] [-n] - -ARGUMENTS - COMMAND... Command to show help for. - -FLAGS - -n, --nested-commands Include all nested commands in the output. - -DESCRIPTION - Display help for apify. -``` - -_See code: [@oclif/plugin-help](https://github.com/oclif/plugin-help/blob/v6.2.10/src/commands/help.ts)_ - -## `apify info` - -Displays information about the currently active Apify account. - -``` -USAGE - $ apify info - -DESCRIPTION - Displays information about the currently active Apify account. - The information is printed to the console. -``` - -_See code: [src/commands/info.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/info.ts)_ - -## `apify init [ACTORNAME]` - -Initializes a new Actor project in an existing directory. - -``` -USAGE - $ apify init [ACTORNAME] [-y] - -ARGUMENTS - ACTORNAME Name of the Actor. If not provided, you will be prompted for it. - -FLAGS - -y, --yes Automatic yes to prompts; assume "yes" as answer to all prompts. Note that in some cases, the command may - still ask for confirmation. - -DESCRIPTION - Initializes a new Actor project in an existing directory. - If the directory contains a Scrapy project in Python, the command automatically creates wrappers so that you can run - your scrapers without changes. - - The command creates the ".actor/actor.json" file and the "storage" directory in the current directory, but does not - touch any other existing files or directories. - - WARNING: The directory at "storage" will be overwritten if it already exists. -``` - -_See code: [src/commands/init.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/init.ts)_ - -## `apify key-value-stores` - -Commands are designed to be used with Key Value Stores. - -``` -USAGE - $ apify key-value-stores - -DESCRIPTION - Commands are designed to be used with Key Value Stores. -``` - -_See code: [src/commands/key-value-stores/index.ts](https://github.com/apify/apify-cli/blob/v0.20.7/src/commands/key-value-stores/index.ts)_ - -## `apify login` - -Logs in to your Apify account. - -``` -USAGE - $ apify login [-t ] [-m console|manual] - -FLAGS - -m, --method=