Using Custom Containers
Last updated
Last updated
Build a custom container and push it to a registry
Launch a Jupyter Notebook based on the custom container
Train and test a model
The jobs submitted to Gradient run in the context of a container. Though the platform comes with mainstream container images, there may be a need to run a job within a custom container.
In this tutorial, we will learn how to build a custom container for training a model based on Apache MXNet. The container encapsulates all the dependencies needed for the job and Jupyter Notebooks for interactive development.
You may want to clone the Github repository that has the Dockerfile and Jupyter Notebook.
Custom containers are Docker images that can be easily built on any machine that has Docker CE installed.
Make sure that Docker is running by executing the following command:
$ docker version
Client: Docker Engine - Community
Version: 18.09.2
API version: 1.39
Go version: go1.10.8
Git
We will now create a Dockerfile that has the commands to install Apache MXNet and Jupyter Notebooks environment.
FROM mxnet/python:latest
RUN pip3 install requests
RUN pip3 install jupyterlab
RUN pip3 install matplotlibThe first line of Dockerfile instructs the build process to use mxnet/python as the base image. The next three lines simply use pip to install dependencies such as jupyterlab and matplotlib.
If you have cloned the Github Repo, Dockerfile is available in the Container directory.
Let’s go ahead and build the custom container. Replace the USERNAME environment variable with your Docker Hub username.
$ docker build -t $USERNAME/mxnet .
Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM mxnet/python:latest
latest: Pulling from mxnet/python
7e6591854262: Pull complete
....
6a614067a2c4: Pull complete
107776115e90: Pull complete
Digest: sha256:628b8e5d6d375d595128e11c8088edae37534524aa1ca2c5086355c89a8d2649
Status: Downloaded newer image for mxnet/python:latest
---> 121567f56cd1
Step 2/4 : RUN pip3 install requests
---> Running in 0127fabe8f91
Requirement already satisfied: requests in /usr/local/lib/python3.5/dist-packages (2.21.0)
....
Removing intermediate container 0127fabe8f91
---> d839e172a78e
Step 3/4 : RUN pip3 install jupyterlab
---> Running in 7b9ddfe995a3
Collecting jupyterlab
Downloading https://files.pythonhosted.org/packages/dd/7f/18b4ecfa055243f1eccdb1d7a1cdc0ae529f3df4c1098cee442ad177511a/jupyterlab-0.35.6-py3-none-any.whl (14.8MB)
Collecting jupyterlab-server<0.3.0,>=0.2.0 (from jupyterlab)
Downloading https://files.pythonhosted.org/packages/78/77/e8a9c300afbe24aa46abaf1091d9e7b82328559e99cf2d601e858bcb3e1a/jupyterlab_server-0.2.0-py3-none-any.whl
...
Removing intermediate container 7b9ddfe995a3
---> 1bb94afa4112
Step 4/4 : RUN pip3 install matplotlib
---> Running in a57b4eaa39f6
Collecting matplotlib
Downloading https://files.pythonhosted.org/packages/89/61/465fb3bfba684b0f53b5c4829c3c89e86e6fe9fdcdfda93e38f1788090f0/matplotlib-3.0.3-cp35-cp35m-manylinux1_x86_64.whl (13.0MB)
Collecting cycler>=0.10 (from matplotlib)
Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
...
Removing intermediate container a57b4eaa39f6
---> d34a43f406cf
Successfully built d34a43f406cf
Successfully tagged janakiramm/mxnet:latestWe can verify the image creation by running docker images command.
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
janakiramm/mxnet latest d34a43f406cf 2 minutes ago 742MB
mxnet/python latest 121567f56cd1 6 days ago 551MBBefore we can access the custom container image from Gradient, we need to push it a container registry such as Docker Hub.
Login to Docker Hub, and then push the image.
$ docker login
Authenticating with existing credentials...
Login Succeeded$ docker push $USERNAME/mxnet:latest
The push refers to repository [docker.io/janakiramm/mxnet]
8b622d1d381a: Pushed
4569983d4e90: Pushed
4af26e82c9e0: Pushed
5d5ab40659c9: Layer already exists
1bd27f114801: Layer already exists
f5ea869d942f: Layer already exists
26d13f8f529c: Layer already exists
d908c11b1a82: Layer already exists
7ccfaa7554e3: Layer already exists
89ec57aea3bf: Layer already exists
a0c1e01578b7: Layer already exists
latest: digest: sha256:8e525774d076a84935348539740418bf4726ce349685733d8787e49c990851e2 size: 2623In the next step, we will launch a Jupyter Notebook based on the container image we just built.
Access the Notebooks from the left navigation bar of Gradient. Choose Custom Container to pull the Apache MXNet image that we built.
Under the container name, type the full container image name including the tag. If you are accessing a private registry, populate the username and password sections.
Provide the command that you would want to run along with the user context. These settings are highly dependent on the way the container is built. Since we didn’t include a default run command within the image, we are instructing Gradient to launch a Jupyter Notebook.
Follow the remaining steps involved in launching a Gradient Notebook. Refer to the Notebook tutorial for a step-by-step guide.
After the Notebook instance is launched successfully, click on the URL to access it.
You can upload the Notebook, MNIST.ipynb, cloned from the Github repo.
You can now train and test the CNN model based Lenet architecture trained with the popular MNIST dataset.
The custom containers feature of Gradient allows developers to add additional runtimes, frameworks, and tools to the platform.
