Hyperparameter Tuning
Last updated
Last updated
This feature is currently only available to our Gradient hosted offering. Contact Sales to learn more.
Paperspace Hyperparameter Tuning based on Hyperopt
Hyperopt is a method for searching through a hyperparameter space. For example, it can use the Tree-structured Parzen Estimator (TPE) algorithm, which intelligently explores the search space while narrowing down to the best estimated parameters.
It is thus a good method for meta-optimizing a neural network. Whereas a neural network is an optimization problem that is tuned using gradient descent methods, hyperparameters cannot be tuned using gradient descent methods. That's where Hyperopt comes in and shines: it's useful not only for tuning hyperparameters like learning rate, but also for tuning more sophisticated parameters, and in a flexible way: it can change the number of layers of different types; the number of neurons in one layer or another; or even the type of layer to use at a certain place in the network given an array of choices, each themselves with nested, tunable hyperparameters.
This kind of Oriented Random Search is Hyperopt's strength, as opposed to a simpler Grid Search where hyperparameters are pre-established with fixed-step increases. Random Search for Hyperparameter Optimization has proven to be an effective search technique. The paper about this technique sits among the most cited deep learning papers. In summary, it is more efficient to randomly search through values and intelligently narrow the search space, rather than looping on fixed sets of hyperparameter values.
If you want to learn more about Hyperopt, you'll probably want to watch the video below, made by the creator of Hyperopt
A parameter is defined with either a certain uniform range or a probability distribution, such as:
There are also a few quantized versions of those functions, which round the generated values at each step of "q":
It is also possible to use a "choice" that can lead to hyperparameter nesting:
Every time you create a hyperparameter tuning experiment we create at least 3 jobs:
Hyperparameter Server that will run your tuning command
n amount of workers that will run jobs
mongoDb server to store job results
Assuming that you have configured an API Key for the paperspace cli enter:
On successfully creating a hyperparameter experiment it should return something like this:
For workers and parameter server you can easily access the logs and outputs of the various tasks.
hyper_tune()
Function to run hyperparameter tuning.
It accepts the following arguments:
train_model User model to tune.
hparam_def User definition (scope) of search space. To set this value, refer to hyperopt documentation.
algo Search algorithm. Default: tpe.suggest (from hyperopt).
max_ecals Maximum number of function evaluations to allow before returning. Default: 25.
func Function to be run by hyper tune. Default: fmin (from hyperopt). Do not change this value if you do not know what you are doing!
It returns a dict with information about the tuning process.
It can raise a ConfigError exception with message if there's no connection to MongoDB.
Note: You do not need to worry about setting your MongoDB version; it will be set within Paperspace infrastructure for hyperparameter tuning.
Usage example:
This feature is coming soon to our platform! In the meantime, we recommend using a tool like TensorBoard to compare the results of your hyperparameter search.
