# Managing Deployments

{% hint style="info" %}
View the full CLI/SDK Docs for **Deployments** here [https://paperspace.github.io/gradient-cli/gradient.cli.html#gradient-deployments](https://paperspace.github.io/gradient-cli/gradient.cli.html#gradient-deployments)
{% endhint %}

## Create a Deployment

{% tabs %}
{% tab title="Web UI" %}
### The Create Deployment flow

To create a Deployment via the UI, there are two pathways to start the Create Deployment flow: **a)** from a Model on the **Models** page; **b)** from the **Deployments** page. Both of these pathways are shown below, after which we will step through the Create Deployment flow.

#### a) From the Models page:

1. Navigate to your list of trained Models by clicking **Models** in the side nav.
2. Find the model you want to deploy, and click **Deploy Model**.

![A Model shown in the "cards" view on the Models page](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSB9ooGt-Z7hsxW7k8%2Fdeploy%20model%20from%20model.png?alt=media\&token=a9b90fe2-fa6c-4e2d-b5f9-a52761855983)

#### b) From the Deployments page:

1. Navigate to your list of Deployments by clicking **Deployments** in the side nav.
2. Click **Create Deployment +**.

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSGC0TAwx4PAWzJOWb%2Fcreate%20deployment%20from%20deployments.png?alt=media\&token=9335dbd1-d5ff-4d39-bb24-9ab5cd30c6e3)

### Using the Create Deployment flow

Now that you've started the Create Deployment flow, let's walk through the various options and deploy your Model!

#### Choose a Model

If you started the flow via the Models page (pathway **a** above), you'll skip this step since you've already chosen a Model to deploy.

If you started the flow via the Deployments page (pathway **b** above), you have the option to choose a Model by clicking the Model selector dropdown and selecting the Model you want to deploy.&#x20;

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSZjDXdZ2qV814wH7N%2Fcreate%20deployment%20header.png?alt=media\&token=72208b09-61d7-45f0-a9b0-fd139f1e38f7)

#### Choose a Container

Select the base container that will support your trained model to run it as a continuous web service. As both CPU & GPU serving are available, be sure to select the container corresponding to your selected machine type and what your Model was optimized for.

![Recommended container options shown with Tensorflow/serving selected](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSJuUfbLSi-wZC0rvf%2FScreen%20Shot%202019-12-31%20at%201.59.11%20PM.png?alt=media\&token=5341cb0a-9383-4082-ae47-7b111210844f)

#### Choose an instance type

Select the GPU or CPU machine type to run your Deployment.

![If you are using Gradient Enterprise, you'll want to select from among My Private Clusters](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSKERzWdyGcS-rDBPW%2FScreen%20Shot%202019-12-31%20at%201.59.02%20PM.png?alt=media\&token=6a1d008d-cae3-49e5-a1cf-a23a4ce08ada)

### Additional Input Parameters

#### Name

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSLu_x2gJ-KNVjh1r9%2FScreen%20Shot%202019-12-31%20at%202.06.14%20PM.png?alt=media\&token=5a14ec56-5c6f-46b4-8e83-5087068dfe1c)

#### **Instance Count**

Select the number of instances to run the Deployment on.  Below we chose 3, meaning there will be 3x K80 GPU instances backing this Deployment. Automatic load balancing is provided for all multi-instance deployments.

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSMRbpO0jNEl-Naqoe%2FScreen%20Shot%202019-12-31%20at%202.10.15%20PM.png?alt=media\&token=8d1e6d0f-cb88-4751-9d1f-7383262c8f13)

#### **Command**

If applicable, choose a command to run at container launch.

{% hint style="info" %}
**Note:** for the Tensorflow/serving base container used here, the command to run the Deployment is unnecessary and thus disabled. This option can be changed when choosing a different base image to deploy o&#x6E;_._
{% endhint %}

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSMmDvhxe_WxTRtkpy%2FScreen%20Shot%202019-12-31%20at%202.06.32%20PM.png?alt=media\&token=e1e7a73c-584b-4e44-b6f1-e43034bc277e)

#### Creative Active Deployment

**Create Active Deployment** (selected by default) means that the Deployment will be created and then automatically run:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSNkauj6B2MNpk2Yoz%2FScreen%20Shot%202019-12-31%20at%202.15.49%20PM.png?alt=media\&token=a147ce0c-a4f4-4fa7-847d-6ac356d0428c)

Alternately, if you don't want your Deployment to run automatically after it is created, you can click toggle **Create Inactive Deployment**:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LiB3KwyizkHpmBaP02o%2F-LiAzrpVNjM8Dv8tRD02%2FScreen%20Shot%202019-06-24%20at%208.17.48%20PM.png?alt=media\&token=b662a45f-30ed-4003-a6fe-f2b861e9be09)

{% hint style="info" %}
**Note:** You are only charged for Deployments when they are running.
{% endhint %}

#### Enable Basic Authentication

Since your Deployment will run as a continuous web service on the public internet, you may wish to require basic authentication on any requests to it. If so, be sure that **Enable Basic Authentication** is toggled _on_ and then enter a **username** and **password**:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxST5cx5dZQNuIzCxOf%2FScreen%20Shot%202019-12-31%20at%202.37.18%20PM.png?alt=media\&token=2c1f0c0b-b550-4d0d-a334-af7be9bf3722)

Finally, now that your Deployment is configured, click **Create Deployment** to create it:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSTjTiQ0pvV_9MsYzH%2Fcreate.png?alt=media\&token=e0130ece-bbb4-4af5-b0fd-d96d42367c61)
{% endtab %}

{% tab title="CLI" %}
Use the `create` subcommand and specify all of the following parameters you need to deploy your model.  Here's a sample command to create the same Deployment as you could do from the UI:

```bash
gradient deployments create \
    --deploymentType TFServing \
    --modelId <your-model-id> \
    --name "Sample Model"
    --machineType K80
    --imageUrl tensorflow/serving:latest-gpu
    --instanceCount 2
```

To obtain your Model ID, you can use the command `gradient models list` and copy the target Model ID from your available Models.\
\
The full set of options for deployments:&#x20;

```
  --deploymentType [TFServing|ONNX|Custom|Flask|TensorRT]
                                  Model deployment type  [required]
  --projectId TEXT                Project ID
  --modelId TEXT                  ID of a trained model
  --name TEXT                     Human-friendly name for new model deployment
                                  [required]

  --machineType TEXT              Type of machine for new deployment
                                  [required]

  --imageUrl TEXT                 Docker image for model serving
                                  [required]

  --instanceCount INTEGER         Number of machine instances
                                  [required]

  --command TEXT                  Deployment command
  --containerModelPath TEXT       Container model path
  --imageUsername TEXT            Username used to access docker image
  --imagePassword TEXT            Password used to access docker image
  --imageServer TEXT              Docker image server
  --containerUrlPath TEXT         Container URL path
  --method TEXT                   Method prefix - default to '/' - If TFServing REST method prefix is ':'
  --dockerArgs JSON_STRING        JSON-style list of docker args
  --env JSON_STRING               JSON-style environmental variables map
  --apiType TEXT                  Type of API - REST or GRPC, defaults to REST
  --ports TEXT                    Expose ports: Takes a single value eg 80
  --clusterId TEXT                Cluster ID
  --authUsername TEXT             Username
  --authPassword TEXT             Password
  --auth                          Generate username and password. Mutually
                                  exclusive with --authUsername and
                                  --authPassword

  --tag TEXT                      One or many tags that you want to add to
                                  model deployment job

  --tags TEXT                     Separated by comma tags that you want add to
                                  model deployment job

  --workspace TEXT                Path to workspace directory, archive, S3 or
                                  git repository

  --workspaceRef TEXT             Git commit hash, branch name or tag
  --workspaceUsername TEXT        Workspace username
  --workspacePassword TEXT        Workspace password
  --minInstanceCount TEXT         Minimal instance count
  --maxInstanceCount TEXT         Maximal instance count
  --scaleCooldownPeriod TEXT      Scale cooldown period
  --metric TEXT                   Autoscaling metrics. Example:
                                  my_metric/targetAverage:21.37

  --resource TEXT                 Autoscaling resources. Example:
                                  cpu/target:60
```

Sometimes, a user will want to do inference with a custom model using a custom container. An example of this is building a Flask or Streamlit container which will use the model trained on Gradient to do inference but also expose a custom dashboard for the team to use to consume the predictions for making business decisions. This is accomplished by specifying the modelType to be "Custom" & passing in the information for the custom container.&#x20;
{% endtab %}
{% endtabs %}

## View, Start, & Stop a Deployment

{% tabs %}
{% tab title="Web UI" %}
Since Deployments are continuous web services, they can be in multiple states, including _Provisioning_, _Provisioned_, _Running_, _Stopped_, and _Error_.

Navigate to the **Deployments** page in the side nav to see your list of Deployments:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSf1qGxLCcQkxHK6ip%2F-LxShthWE2FPO-jbcrXX%2FScreen%20Shot%202019-12-31%20at%203.48.12%20PM.png?alt=media\&token=17b2c24c-83bc-498b-b440-6756c57370b0)

Each Deployment has: a **Name** and a unique **ID**; links to its associated **Experiment** and **Model** (by ID) that it was created from; its **Container Type; Date Created; Status;** and **Actions** you can perform.

To start a _Stopped_ Deployment, click **Start** from among that Deployment's **Actions**. The Status will change to _Provisioning_ and, if all goes smoothly, will soon say _Running_. Learn more about [Deployment States](managing-deployments/deployment-states).

Click the Deployment's row to show its details:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LiB3KwyizkHpmBaP02o%2F-LiB3WinXj2TWRHI9L2l%2FScreen%20Shot%202019-06-24%20at%208.37.24%20PM.png?alt=media\&token=483a5bbc-566c-4800-b570-bf3829349501)

Each Deployment has its own unique RESTful API. Inference can be performed via the shown endpoint: `https://services.paperspace.io/model-serving/<your-model-id>:predict`.  The number of running instances and the instance count are visible as well.

Congrats, you've created a Deployment and can perform inference!

[Learn more about a Deployment's RESTful API here](managing-deployments/a-deployed-models-api-endpoint/deployment-restful-api-tfserving).
{% endtab %}

{% tab title="CLI" %}
## List Deployments

To list your Deployments with optional filtering, use the `list` subcommand:

```
gradient deployments list 
```

**Options:**

```bash
  --state [BUILDING|PROVISIONING|STARTING|RUNNING|STOPPING|STOPPED|ERROR]
                                  Filter by deployment state
  --projectId TEXT                Use to filter by project ID
  --modelId TEXT                  Use to filter by model ID
  --apiKey TEXT                   API key to use this time only
  --optionsFile PATH              Path to YAML file with predefined options
  --createOptionsFile PATH        Generate template options file
  --help                          Show this message and exit.
```

For example, to view all running Deployments in your team, run:

```bash
gradient list --state RUNNING
```

## Start a Deployment

To start a previously created but Stopped deployment by ID, use the `start` subcommand:

```bash
gradient deployments start --id <your-deployment-id>
```

## Stop a Deployment

To stop a _Running_ Deployment by ID, use the `stop` subcommand:

```bash
gradient deployments stop --id <your-deployment-id>

```
{% endtab %}
{% endtabs %}

## Edit a Deployment

{% tabs %}
{% tab title="Web UI" %}
You can edit a Deployment's attributes, such as the underlying model, the Deployment's name, instance count, etc.

To edit a Deployment, navigate to the **Deployments** page, find the Deployment you want to edit, and click **Edit** in the Actions column:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSU_nC7olCg880Tktp%2FScreen%20Shot%202019-12-31%20at%202.44.20%20PM.png?alt=media\&token=f52ed93b-7841-4d42-889a-ae1f1e5c2d9d)

This will launch the Edit Deployment flow, which is nearly the same as the Create Deployment flow. The differences are that the Edit Deployment flow will display the **Deployment ID**, the **Deployment Endpoint**, and will always allow you to **Choose a Model**; and it will _not_ display the **Create Active Deployment** toggle. (If you want to edit and start a stopped Deployment, save your changes and then click **Start** back on the Deployments page.)

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSZsdRRvnuRMQ9T7WG%2FScreen%20Shot%202019-12-31%20at%203.08.58%20PM.png?alt=media\&token=b81b76f2-9a70-4d9d-95d8-8f5522f377f0)

Besides those differences, you can edit any of the other values of your Deployment just like you did in the Create Deployment flow.

When you are done and want to save your changes, click **Edit Deployment** at the bottom:

![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSaXWtWcNkBXqiP4xk%2Fedit.png?alt=media\&token=96558720-b601-4c73-b35f-1efbfc31fdb5)


{% endtab %}

{% tab title="CLI" %}
You can edit a Deployment's attributes, such as the underlying model, the Deployment's name, instance count, etc.

To update a Deployment, use the `update` subcommand:

```bash
gradient deployments update --id <your-deployment-id>
```

All Deployment options can be updated with this command:&#x20;

```bash
Usage: gradient deployments update [OPTIONS]

  Modify existing deployment

Options:
  --id TEXT                       ID of existing deployment
                                  [required]

  --deploymentType [TFServing|ONNX|Custom|Flask|TensorRT]
                                  Model deployment type
  --projectId TEXT                Project ID
  --modelId TEXT                  ID of a trained model
  --name TEXT                     Human-friendly name for new model deployment
  --machineType TEXT              Type of machine for new deployment
  --imageUrl TEXT                 Docker image for model serving
  --instanceCount INTEGER         Number of machine instances
  --command TEXT                  Deployment command
  --containerModelPath TEXT       Container model path
  --imageUsername TEXT            Username used to access docker image
  --imagePassword TEXT            Password used to access docker image
  --imageServer TEXT              Docker image server
  --containerUrlPath TEXT         Container URL path
  --method TEXT                   Method
  --dockerArgs JSON_STRING        JSON-style list of docker args
  --env JSON_STRING               JSON-style environmental variables map
  --apiType TEXT                  Type of API
  --ports TEXT                    Ports
  --authUsername TEXT             Username
  --authPassword TEXT             Password
  --clusterId TEXT                Cluster ID
  --workspace TEXT                Path to workspace directory, archive, S3 or
                                  git repository

  --workspaceRef TEXT             Git commit hash, branch name or tag
  --workspaceUsername <username>  Workspace username
  --workspacePassword TEXT        Workspace password
  --minInstanceCount TEXT         Minimal instance count
  --maxInstanceCount TEXT         Maximal instance count
  --scaleCooldownPeriod TEXT      Scale cooldown period
  --metric TEXT                   Autoscaling metrics. Example:
                                  my_metric/targetAverage:21.37

  --resource TEXT                 Autoscaling resources. Example:
                                  cpu/target:60

  --apiKey TEXT                   API key to use this time only
  --optionsFile PATH              Path to YAML file with predefined options
  --createOptionsFile PATH        Generate template options file
  --help                          Show this message and exit.

```
{% endtab %}
{% endtabs %}

## Secure Deployment with Basic HTTPS authentication

The client sends the user name and password as unencrypted base64 encoded text. &#x20;

Most web browsers will display a login dialog when this response is received, allowing the user to enter a username and password.&#x20;

{% tabs %}
{% tab title="GUI" %}
![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-M0x98h9ogIFBXpDZ-ah%2F-M0xBmxdTmJBQLQ0KD7l%2FScreenshot%202020-02-25%20at%2017.34.11.png?alt=media\&token=40022513-ab45-4032-a32f-f855207eb3ef)

To add HTTP authentication to deployment in GUI - at the bottom of the creation page you have to check _"Enable Basic Authentication"_
{% endtab %}

{% tab title="CLI" %}
To secure a deployment with HTTP authentication you have to pass&#x20;

```
--authUsername 'testuser' 
--authPassword 'test' 
```

Example command:

```
gradient deployments create 
--deploymentType TFServing 
--name "authtest" 
--modelId <model id> 
--authUsername 'testuser' 
--authPassword 'test' 
--machineType c5.xlarge 
--clusterId clu7jqg9j 
--imageUrl 'tensorflow/serving:latest' 
--instanceCount 1
```
{% endtab %}
{% endtabs %}







