Gotcha! TF Saver Subset Initialization

While working on the first part of my style-transfer project, I dealt with two main variable groups:

• The input variable which was the image I was optimizing.
• The VGG-19 network, whose weights were frozen.

To get both of these interacting properly and functioning correctly within a Tensorflow (TF) environment, the input variable would need to be initialized to an initial value while the vgg19 network would need to be initialized with pre-trained weights.

In the past, I have either initialized the entire network with pre-trained weights or with new, initial values but not a combination. I knew that a TF Saver object would be used to restored the weights like so:

saver.restore(sess, 'vgg_19.ckpt')

Where sess refers to a TF Session and the filepath points to the checkpoint file. Since I had a new input variable that didn’t correspond to one of the pre-trained weights, I assumed that the Saver would skip restoring this and rely on me to initialize it myself later. I was wrong:

NotFoundError: Tensor name "input_var" not found in checkpoint files vgg_19.ckpt

The Saver noted down all the variables present in the graph and expected to find all those variables in the checkpoint. When it didn’t, it complained. I found some guidance using this link which helped me better understand the var_list parameter of the Saver initialization method. This parameter specificied which variables would be saved and restored. The former was a lot easier to grasp than the latter. It struck me as a little counterintutive that in initializing the Saver object, I would be specifying up-front which variables I would be restoring. Having thought about it for awhile now and tinkering with it, it makes a little more sense to me. Or perhaps I’ve just got used to this quirk.

So given my new knowledge, I knew that I had to specify which variables would be loaded from the checkpoint (all the vgg19 variables) and which would not be the responsiblity of the Saver (the new input variable). To get the names of all the variables present in the graph, I used the following:

all_variables = tf.get_collection_ref(tf.GraphKeys.GLOBAL_VARIABLES)

Printing that out, I could see that the first variable was the new input variable and so all_variables[1:] provided me with all the vgg19 variables. So the Saver would be created as such:

saver = tf.train.Saver(var_list=all_variables[1:])

Running the restore method of this object caused no errors!

In my graph, the input variable was at the very start so I could specify the var_list the way I did. Perhaps in yours, the new variable might exist somewhere else in the list or maybe you even have several variables you want the Saver to ignore. In that case, just have a look at all_variables and extract the necessary variables. Maybe you could write a for loop to go through the list and ignore variables with certain names.

So when mixing around new and pre-trained variables, be sure create the Saver object appropriately. If you’d like to play around more, here’s a notebook for you to tinker with!

• 2020 3
• 2019 3
• 2018 11