Registering Models in Gradient
Objectives
Understand the workflow involved in registering models
Passing environment variables through Gradient CLI
Persisting model files in Gradient Storage
Registering Tensorflow Models in Gradient
Introduction
Experiments in Gradient can generate machine learning models, which can be interpreted and stored in your Project's Models list. This list holds references to the model and checkpoint files generated during the training period as well as summary metrics associated with the model's performance, such as accuracy and loss.
In this tutorial, we will create an experiment to generate a Keras model based on the Fashion MNIST dataset. We will learn techniques such as passing environment variables to jobs, specifying the right container image, and mentioning the path to store the model artifacts.
The model is trained in Keras but it is finally exported as a TensorFlow model through tf.saved_model.simple_savemethod. This approach seralizes Keras session into a TensorFlow .pb file.
Start by cloning the repo https://github.com/janakiramm/fashionmnist that contains the code for training and inferencing the model.
Create a Project for Fashion MNIST
We will start by creating a project that can contain multiple experiments we may run during the training.
gradient projects create --name FashionCreate an Experiment to Train the Model
We will now start an experiment within the project created above. Make a note of the project id before proceeding further.
Switch to the train directory of the cloned Github repo.
cd traingradient experiments run singlenode \
--name fmnist \
--projectId prioax2c4 \
--experimentEnv "{\"EPOCHS\":5}" \
--container tensorflow/tensorflow:1.9.0-gpu \
--machineType K80 \
--command "python train.py --modelPath /storage/model --version 1" \
--modelType Tensorflow \
--modelPath "/storage/model"
--workspace .The above command has multiple switches that are important to the job. Let’s understand each of them.
The singlenode parameter runs the job on a single host.
--name assigns a friendly name to the experiment.
--projectId associates the experiment with an existing project.
--experimentEnv passes environment variables to the script. In our code, we decide the number of epochs based on the value defined in the EPOCHS environment variable.
--container parameter points the job to a container image used for the training job. Notice that we are passing an image that can advantage of a GPU-based machine.
--machineType schedules the job in one of the preferred instances. In our case, we are using K80 machine type that comes with an NVIDIA K80 GPU. Since the container and machine type are based on GPU, the job exploits the CUDA and cuDNN for accelerated training.
--command instructs the job to execute the script along with the passed parameters. The script expects the path to store the final model artifacts along with the version number. Since we are using a sub-directory under the /storage directory, the files stored are persisted across experiments. The model files stored here are used to register the TensorFlow model with Gradient. Feel free to explore train.py to understand how environment variables and command line parameters can be used to target Gradient specific features while keeping the code independent.
--modelType Tensorflowswitch indicates that the job generates a valid TensorFlow model which can be managed and served by Gradient. Frameworks other than TensorFlow will be supported in the near future, such as ONNX and Custom.
--modelPath tells Gradient where to look for the model artifacts. This is typically /artifacts or /storage location. We are passing /storage/model directory which was used within the code.
--workspace . tells Gradient to upload your current directory (.) to the experiment. The files in this directory will be the working directory of your experiment.
Within a few seconds of running the command, you should see the logs displayed on the screen.
Archiving your working directory for upload as your experiment workspace...(See https://docs.paperspace.com/gradient/experiments/run-experiments for more information.)
Removing existing archive
Creating zip archive: train.zip
100% (1 of 1) |########################################| Elapsed Time: 0:00:00 Time: 0:00:00
Finished creating archive: train.zip
Uploading zipped workspace to S3
100% (3108 of 3108) |##################################| Elapsed Time: 0:00:00 ETA: 00:00:00
100% (3108 of 3108) |##################################| Elapsed Time: 0:00:00 Time: 0:00:00
Uploading completed
New experiment created and started with ID: e720893n7f5vx
Awaiting logs...
js3v54dfgz1zcu 1 Downloading data from http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
32768/29515 [=================================] - 0s 4us/step
40960/29515 [=========================================] - 0s 3us/step
js3v54dfgz1zcu 4 Downloading data from http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
26427392/26421880 [==============================] - 2s 0us/step
26435584/26421880 [==============================] - 2s 0us/step
js3v54dfgz1zcu 7 Downloading data from http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
16384/5148 [===============================================================================================] - 0s 0us/step
js3v54dfgz1zcu 9 Downloading data from http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
4423680/4422102 [==============================] - 1s 0us/step
4431872/4422102 [==============================] - 1s 0us/step
js3v54dfgz1zcu 12 2019-06-29 06:30:41.922354: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
js3v54dfgz1zcu 13 2019-06-29 06:30:42.014405: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
js3v54dfgz1zcu 14 2019-06-29 06:30:42.014841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
js3v54dfgz1zcu 15 name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
js3v54dfgz1zcu 16 pciBusID: 0000:00:04.0
js3v54dfgz1zcu 17 totalMemory: 11.17GiB freeMemory: 11.09GiB
js3v54dfgz1zcu 18 2019-06-29 06:30:42.014881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
js3v54dfgz1zcu 19 2019-06-29 06:30:42.345929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
js3v54dfgz1zcu 20 2019-06-29 06:30:42.345995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
js3v54dfgz1zcu 21 2019-06-29 06:30:42.346006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
js3v54dfgz1zcu 22 2019-06-29 06:30:42.346329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10748 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
js3v54dfgz1zcu 23 _________________________________________________________________
js3v54dfgz1zcu 24 Layer (type) Output Shape Param #
js3v54dfgz1zcu 25 =================================================================
js3v54dfgz1zcu 26 Conv1 (Conv2D) (None, 13, 13, 8) 80
js3v54dfgz1zcu 27 _________________________________________________________________
js3v54dfgz1zcu 28 flatten (Flatten) (None, 1352) 0
js3v54dfgz1zcu 29 _________________________________________________________________
js3v54dfgz1zcu 30 Softmax (Dense) (None, 10) 13530
js3v54dfgz1zcu 31 =================================================================
js3v54dfgz1zcu 32 Total params: 13,610
js3v54dfgz1zcu 33 Trainable params: 13,610
js3v54dfgz1zcu 34 Non-trainable params: 0
js3v54dfgz1zcu 35 _________________________________________________________________
js3v54dfgz1zcu 36 Epoch 1/5
58080/60000 [============================>.] - ETA: 0s - loss: 0.5437 - acc: 0.8103
60000/60000 [==============================] - 7s 112us/step - loss: 0.5406 - acc: 0.8113
js3v54dfgz1zcu 39 Epoch 2/5
60000/60000 [==============================] - 5s 82us/step - loss: 0.4034 - acc: 0.8597
js3v54dfgz1zcu 41 Epoch 3/5
57536/60000 [===========================>..] - ETA: 0s - loss: 0.3718 - acc: 0.8697
60000/60000 [==============================] - 5s 88us/step - loss: 0.3715 - acc: 0.8698.8698
js3v54dfgz1zcu 44 Epoch 4/5
54944/60000 [==========================>...] - ETA: 0s - loss: 0.3508 - acc: 0.8760
60000/60000 [==============================] - 6s 92us/step - loss: 0.3514 - acc: 0.876059
js3v54dfgz1zcu 47 Epoch 5/5
59488/60000 [============================>.] - ETA: 0s - loss: 0.3391 - acc: 0.8794
60000/60000 [==============================] - 5s 85us/step - loss: 0.3392 - acc: 0.8795.8794
10000/10000 [==============================] - 0s 45us/step
js3v54dfgz1zcu 51
js3v54dfgz1zcu 52 Model accuracy: 0.8657
js3v54dfgz1zcu 53
js3v54dfgz1zcu 54 Model saved to /storage/model
js3v54dfgz1zcu 55
js3v54dfgz1zcu 56 PSEOFVerifying the Creation of Model
We can check if the output of the job is registered as a valid TensorFlow model with the following command.
gradient models list+------+-----------------+------------+------------+----------------+
| Name | ID | Model Type | Project ID | Experiment ID |
+------+-----------------+------------+------------+----------------+
| None | mosdnkkv1o1xuem | Tensorflow | prioax2c4 | e720893n7f5vx |
+------+-----------------+------------+------------+----------------+
The project id prioax2c4 and experiment id e720893n7f5vx confirm that it is the model associated with the latest experiment.
You can also visit the Models section of Gradient UI to see a list of registered models.
Summary
After registering the model, we can turn that into a deployment to perform inferencing.
Last updated