Using Custom Containers
Objectives
Build a custom container and push it to a registry
Launch a Jupyter Notebook based on the custom container
Train and test a model
Introduction
The jobs submitted to Gradient run in the context of a container. Though the platform comes with mainstream container images, there may be a need to run a job within a custom container.
In this tutorial, we will learn how to build a custom container for training a model based on Apache MXNet. The container encapsulates all the dependencies needed for the job and Jupyter Notebooks for interactive development.
You may want to clone the Github repository that has the Dockerfile and Jupyter Notebook.
Building a Custom Container
Custom containers are Docker images that can be easily built on any machine that has Docker CE installed.
Make sure that Docker is running by executing the following command:
$ docker version
Client: Docker Engine - Community
Version: 18.09.2
API version: 1.39
Go version: go1.10.8
Git commit: 6247962
Built: Sun Feb 10 04:12:39 2019
OS/Arch: darwin/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:06 2019
OS/Arch: linux/amd64
Experimental: true
We will now create a Dockerfile that has the commands to install Apache MXNet and Jupyter Notebooks environment.
The first line of Dockerfile instructs the build process to use mxnet/python as the base image. The next three lines simply use pip to install dependencies such as jupyterlab and matplotlib.
If you have cloned the Github Repo, Dockerfile is available in the Container directory.
Let’s go ahead and build the custom container. Replace the USERNAME environment variable with your Docker Hub username.
We can verify the image creation by running docker images command.
Before we can access the custom container image from Gradient, we need to push it a container registry such as Docker Hub.
Login to Docker Hub, and then push the image.
In the next step, we will launch a Jupyter Notebook based on the container image we just built.
Launching a Notebook based on a Custom Container
Access the Notebooks from the left navigation bar of Gradient. Choose Custom Container to pull the Apache MXNet image that we built.
Under the container name, type the full container image name including the tag. If you are accessing a private registry, populate the username and password sections.
Provide the command that you would want to run along with the user context. These settings are highly dependent on the way the container is built. Since we didn’t include a default run command within the image, we are instructing Gradient to launch a Jupyter Notebook.
Follow the remaining steps involved in launching a Gradient Notebook. Refer to the Notebook tutorial for a step-by-step guide.
After the Notebook instance is launched successfully, click on the URL to access it.
You can upload the Notebook, MNIST.ipynb, cloned from the Github repo.
You can now train and test the CNN model based Lenet architecture trained with the popular MNIST dataset.
Summary
The custom containers feature of Gradient allows developers to add additional runtimes, frameworks, and tools to the platform.
Last updated