# Managing Deployments {% hint style="info" %} View the full CLI/SDK Docs for **Deployments** here {% endhint %} ## Create a Deployment {% tabs %} {% tab title="Web UI" %} ### The Create Deployment flow To create a Deployment via the UI, there are two pathways to start the Create Deployment flow: **a)** from a Model on the **Models** page; **b)** from the **Deployments** page. Both of these pathways are shown below, after which we will step through the Create Deployment flow. #### a) From the Models page: 1. Navigate to your list of trained Models by clicking **Models** in the side nav. 2. Find the model you want to deploy, and click **Deploy Model**. ![A Model shown in the "cards" view on the Models page](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSB9ooGt-Z7hsxW7k8%2Fdeploy%20model%20from%20model.png?alt=media\&token=a9b90fe2-fa6c-4e2d-b5f9-a52761855983) #### b) From the Deployments page: 1. Navigate to your list of Deployments by clicking **Deployments** in the side nav. 2. Click **Create Deployment +**. ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSGC0TAwx4PAWzJOWb%2Fcreate%20deployment%20from%20deployments.png?alt=media\&token=9335dbd1-d5ff-4d39-bb24-9ab5cd30c6e3) ### Using the Create Deployment flow Now that you've started the Create Deployment flow, let's walk through the various options and deploy your Model! #### Choose a Model If you started the flow via the Models page (pathway **a** above), you'll skip this step since you've already chosen a Model to deploy. If you started the flow via the Deployments page (pathway **b** above), you have the option to choose a Model by clicking the Model selector dropdown and selecting the Model you want to deploy. ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSZjDXdZ2qV814wH7N%2Fcreate%20deployment%20header.png?alt=media\&token=72208b09-61d7-45f0-a9b0-fd139f1e38f7) #### Choose a Container Select the base container that will support your trained model to run it as a continuous web service. As both CPU & GPU serving are available, be sure to select the container corresponding to your selected machine type and what your Model was optimized for. ![Recommended container options shown with Tensorflow/serving selected](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSJuUfbLSi-wZC0rvf%2FScreen%20Shot%202019-12-31%20at%201.59.11%20PM.png?alt=media\&token=5341cb0a-9383-4082-ae47-7b111210844f) #### Choose an instance type Select the GPU or CPU machine type to run your Deployment. ![If you are using Gradient Enterprise, you'll want to select from among My Private Clusters](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSKERzWdyGcS-rDBPW%2FScreen%20Shot%202019-12-31%20at%201.59.02%20PM.png?alt=media\&token=6a1d008d-cae3-49e5-a1cf-a23a4ce08ada) ### Additional Input Parameters #### Name ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSLu_x2gJ-KNVjh1r9%2FScreen%20Shot%202019-12-31%20at%202.06.14%20PM.png?alt=media\&token=5a14ec56-5c6f-46b4-8e83-5087068dfe1c) #### **Instance Count** Select the number of instances to run the Deployment on. Below we chose 3, meaning there will be 3x K80 GPU instances backing this Deployment. Automatic load balancing is provided for all multi-instance deployments. ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSMRbpO0jNEl-Naqoe%2FScreen%20Shot%202019-12-31%20at%202.10.15%20PM.png?alt=media\&token=8d1e6d0f-cb88-4751-9d1f-7383262c8f13) #### **Command** If applicable, choose a command to run at container launch. {% hint style="info" %} **Note:** for the Tensorflow/serving base container used here, the command to run the Deployment is unnecessary and thus disabled. This option can be changed when choosing a different base image to deploy on*.* {% endhint %} ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSMmDvhxe_WxTRtkpy%2FScreen%20Shot%202019-12-31%20at%202.06.32%20PM.png?alt=media\&token=e1e7a73c-584b-4e44-b6f1-e43034bc277e) #### Creative Active Deployment **Create Active Deployment** (selected by default) means that the Deployment will be created and then automatically run: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxNK3k-frz4FIXXEBOP%2F-LxSNkauj6B2MNpk2Yoz%2FScreen%20Shot%202019-12-31%20at%202.15.49%20PM.png?alt=media\&token=a147ce0c-a4f4-4fa7-847d-6ac356d0428c) Alternately, if you don't want your Deployment to run automatically after it is created, you can click toggle **Create Inactive Deployment**: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LiB3KwyizkHpmBaP02o%2F-LiAzrpVNjM8Dv8tRD02%2FScreen%20Shot%202019-06-24%20at%208.17.48%20PM.png?alt=media\&token=b662a45f-30ed-4003-a6fe-f2b861e9be09) {% hint style="info" %} **Note:** You are only charged for Deployments when they are running. {% endhint %} #### Enable Basic Authentication Since your Deployment will run as a continuous web service on the public internet, you may wish to require basic authentication on any requests to it. If so, be sure that **Enable Basic Authentication** is toggled *on* and then enter a **username** and **password**: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxST5cx5dZQNuIzCxOf%2FScreen%20Shot%202019-12-31%20at%202.37.18%20PM.png?alt=media\&token=2c1f0c0b-b550-4d0d-a334-af7be9bf3722) Finally, now that your Deployment is configured, click **Create Deployment** to create it: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSTjTiQ0pvV_9MsYzH%2Fcreate.png?alt=media\&token=e0130ece-bbb4-4af5-b0fd-d96d42367c61) {% endtab %} {% tab title="CLI" %} Use the `create` subcommand and specify all of the following parameters you need to deploy your model. Here's a sample command to create the same Deployment as you could do from the UI: ```bash gradient deployments create \ --deploymentType TFServing \ --modelId \ --name "Sample Model" --machineType K80 --imageUrl tensorflow/serving:latest-gpu --instanceCount 2 ``` To obtain your Model ID, you can use the command `gradient models list` and copy the target Model ID from your available Models.\ \ The full set of options for deployments: ``` --deploymentType [TFServing|ONNX|Custom|Flask|TensorRT] Model deployment type [required] --projectId TEXT Project ID --modelId TEXT ID of a trained model --name TEXT Human-friendly name for new model deployment [required] --machineType TEXT Type of machine for new deployment [required] --imageUrl TEXT Docker image for model serving [required] --instanceCount INTEGER Number of machine instances [required] --command TEXT Deployment command --containerModelPath TEXT Container model path --imageUsername TEXT Username used to access docker image --imagePassword TEXT Password used to access docker image --imageServer TEXT Docker image server --containerUrlPath TEXT Container URL path --method TEXT Method prefix - default to '/' - If TFServing REST method prefix is ':' --dockerArgs JSON_STRING JSON-style list of docker args --env JSON_STRING JSON-style environmental variables map --apiType TEXT Type of API - REST or GRPC, defaults to REST --ports TEXT Expose ports: Takes a single value eg 80 --clusterId TEXT Cluster ID --authUsername TEXT Username --authPassword TEXT Password --auth Generate username and password. Mutually exclusive with --authUsername and --authPassword --tag TEXT One or many tags that you want to add to model deployment job --tags TEXT Separated by comma tags that you want add to model deployment job --workspace TEXT Path to workspace directory, archive, S3 or git repository --workspaceRef TEXT Git commit hash, branch name or tag --workspaceUsername TEXT Workspace username --workspacePassword TEXT Workspace password --minInstanceCount TEXT Minimal instance count --maxInstanceCount TEXT Maximal instance count --scaleCooldownPeriod TEXT Scale cooldown period --metric TEXT Autoscaling metrics. Example: my_metric/targetAverage:21.37 --resource TEXT Autoscaling resources. Example: cpu/target:60 ``` Sometimes, a user will want to do inference with a custom model using a custom container. An example of this is building a Flask or Streamlit container which will use the model trained on Gradient to do inference but also expose a custom dashboard for the team to use to consume the predictions for making business decisions. This is accomplished by specifying the modelType to be "Custom" & passing in the information for the custom container. {% endtab %} {% endtabs %} ## View, Start, & Stop a Deployment {% tabs %} {% tab title="Web UI" %} Since Deployments are continuous web services, they can be in multiple states, including *Provisioning*, *Provisioned*, *Running*, *Stopped*, and *Error*. Navigate to the **Deployments** page in the side nav to see your list of Deployments: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSf1qGxLCcQkxHK6ip%2F-LxShthWE2FPO-jbcrXX%2FScreen%20Shot%202019-12-31%20at%203.48.12%20PM.png?alt=media\&token=17b2c24c-83bc-498b-b440-6756c57370b0) Each Deployment has: a **Name** and a unique **ID**; links to its associated **Experiment** and **Model** (by ID) that it was created from; its **Container Type; Date Created; Status;** and **Actions** you can perform. To start a *Stopped* Deployment, click **Start** from among that Deployment's **Actions**. The Status will change to *Provisioning* and, if all goes smoothly, will soon say *Running*. Learn more about [Deployment States](https://paperspace.gitbook.io/gradient/master/deployments/managing-deployments/deployment-states). Click the Deployment's row to show its details: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LiB3KwyizkHpmBaP02o%2F-LiB3WinXj2TWRHI9L2l%2FScreen%20Shot%202019-06-24%20at%208.37.24%20PM.png?alt=media\&token=483a5bbc-566c-4800-b570-bf3829349501) Each Deployment has its own unique RESTful API. Inference can be performed via the shown endpoint: `https://services.paperspace.io/model-serving/:predict`. The number of running instances and the instance count are visible as well. Congrats, you've created a Deployment and can perform inference! [Learn more about a Deployment's RESTful API here](https://paperspace.gitbook.io/gradient/master/deployments/managing-deployments/a-deployed-models-api-endpoint/deployment-restful-api-tfserving). {% endtab %} {% tab title="CLI" %} ## List Deployments To list your Deployments with optional filtering, use the `list` subcommand: ``` gradient deployments list ``` **Options:** ```bash --state [BUILDING|PROVISIONING|STARTING|RUNNING|STOPPING|STOPPED|ERROR] Filter by deployment state --projectId TEXT Use to filter by project ID --modelId TEXT Use to filter by model ID --apiKey TEXT API key to use this time only --optionsFile PATH Path to YAML file with predefined options --createOptionsFile PATH Generate template options file --help Show this message and exit. ``` For example, to view all running Deployments in your team, run: ```bash gradient list --state RUNNING ``` ## Start a Deployment To start a previously created but Stopped deployment by ID, use the `start` subcommand: ```bash gradient deployments start --id ``` ## Stop a Deployment To stop a *Running* Deployment by ID, use the `stop` subcommand: ```bash gradient deployments stop --id ``` {% endtab %} {% endtabs %} ## Edit a Deployment {% tabs %} {% tab title="Web UI" %} You can edit a Deployment's attributes, such as the underlying model, the Deployment's name, instance count, etc. To edit a Deployment, navigate to the **Deployments** page, find the Deployment you want to edit, and click **Edit** in the Actions column: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSU_nC7olCg880Tktp%2FScreen%20Shot%202019-12-31%20at%202.44.20%20PM.png?alt=media\&token=f52ed93b-7841-4d42-889a-ae1f1e5c2d9d) This will launch the Edit Deployment flow, which is nearly the same as the Create Deployment flow. The differences are that the Edit Deployment flow will display the **Deployment ID**, the **Deployment Endpoint**, and will always allow you to **Choose a Model**; and it will *not* display the **Create Active Deployment** toggle. (If you want to edit and start a stopped Deployment, save your changes and then click **Start** back on the Deployments page.) ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSZsdRRvnuRMQ9T7WG%2FScreen%20Shot%202019-12-31%20at%203.08.58%20PM.png?alt=media\&token=b81b76f2-9a70-4d9d-95d8-8f5522f377f0) Besides those differences, you can edit any of the other values of your Deployment just like you did in the Create Deployment flow. When you are done and want to save your changes, click **Edit Deployment** at the bottom: ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-LxSQ8735DNxjUOsphY8%2F-LxSaXWtWcNkBXqiP4xk%2Fedit.png?alt=media\&token=96558720-b601-4c73-b35f-1efbfc31fdb5) {% endtab %} {% tab title="CLI" %} You can edit a Deployment's attributes, such as the underlying model, the Deployment's name, instance count, etc. To update a Deployment, use the `update` subcommand: ```bash gradient deployments update --id ``` All Deployment options can be updated with this command: ```bash Usage: gradient deployments update [OPTIONS] Modify existing deployment Options: --id TEXT ID of existing deployment [required] --deploymentType [TFServing|ONNX|Custom|Flask|TensorRT] Model deployment type --projectId TEXT Project ID --modelId TEXT ID of a trained model --name TEXT Human-friendly name for new model deployment --machineType TEXT Type of machine for new deployment --imageUrl TEXT Docker image for model serving --instanceCount INTEGER Number of machine instances --command TEXT Deployment command --containerModelPath TEXT Container model path --imageUsername TEXT Username used to access docker image --imagePassword TEXT Password used to access docker image --imageServer TEXT Docker image server --containerUrlPath TEXT Container URL path --method TEXT Method --dockerArgs JSON_STRING JSON-style list of docker args --env JSON_STRING JSON-style environmental variables map --apiType TEXT Type of API --ports TEXT Ports --authUsername TEXT Username --authPassword TEXT Password --clusterId TEXT Cluster ID --workspace TEXT Path to workspace directory, archive, S3 or git repository --workspaceRef TEXT Git commit hash, branch name or tag --workspaceUsername Workspace username --workspacePassword TEXT Workspace password --minInstanceCount TEXT Minimal instance count --maxInstanceCount TEXT Maximal instance count --scaleCooldownPeriod TEXT Scale cooldown period --metric TEXT Autoscaling metrics. Example: my_metric/targetAverage:21.37 --resource TEXT Autoscaling resources. Example: cpu/target:60 --apiKey TEXT API key to use this time only --optionsFile PATH Path to YAML file with predefined options --createOptionsFile PATH Generate template options file --help Show this message and exit. ``` {% endtab %} {% endtabs %} ## Secure Deployment with Basic HTTPS authentication The client sends the user name and password as unencrypted base64 encoded text. Most web browsers will display a login dialog when this response is received, allowing the user to enter a username and password. {% tabs %} {% tab title="GUI" %} ![](https://1320806315-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHZRFUkajubOAmgu6Rd%2F-M0x98h9ogIFBXpDZ-ah%2F-M0xBmxdTmJBQLQ0KD7l%2FScreenshot%202020-02-25%20at%2017.34.11.png?alt=media\&token=40022513-ab45-4032-a32f-f855207eb3ef) To add HTTP authentication to deployment in GUI - at the bottom of the creation page you have to check *"Enable Basic Authentication"* {% endtab %} {% tab title="CLI" %} To secure a deployment with HTTP authentication you have to pass ``` --authUsername 'testuser' --authPassword 'test' ``` Example command: ``` gradient deployments create --deploymentType TFServing --name "authtest" --modelId --authUsername 'testuser' --authPassword 'test' --machineType c5.xlarge --clusterId clu7jqg9j --imageUrl 'tensorflow/serving:latest' --instanceCount 1 ``` {% endtab %} {% endtabs %}