Core ML 3 Create Updatable Models

Core ML got a big boost this year with the Core ML 3 update during WWDC 2019. Among the many improvements, we got, On-Device Learning stands out. The goal of this article is to show you the process to create your own Core ML 3 models which can be updated on your iPhone or iPad.

Though it’s fancy to do an on-device model training, but it isn’t feasible given the long duration it’ll take for the model to train on iOS Devices. At best, it is recommended to retrain a model on the device, specifically for a user.

Before we go down the implementation road, let’s admire the other updates Core ML 3 has in store for us.

What’s New In Core ML 3

  • 70 new neural network layers – New layers allow more complex neural network models to be converted to Core ML without the need of writing custom layers.
  • Variety of new models – Models such as kNN classifier, ItemSimilarityRecommender, SoundAnalysisPreprocessing, WordEmbedding, Linked Models will just help to solve more machine learning problems.
  • Core ML API has more abstraction – You need not convert images to CVPixelBuffer for CNN models anymore. MLFeatureValue does that for you automatically.
  • ML Model Configuration – We can now configure the ML Model as per our needs. Example constraining MLComputeUnits to cpu, gpu and/or neural engine.
  • On Device Learning – Now you can easily train/retrain your models on the device itself. No need to recompile it from your machine!

Linked Models allow reusability across models. So if two models are using another model, that model would now be loaded just once.

Let’s deep dive into the code now.
We’ll be creating a Keras Cat vs Dog Image Classifier CNN Model using Categorical Cross-Entropy loss function. If you directly want to jump onto the Core ML part skip the next section.

Creating a Keras Image Classifier Model

The model is compiled with the categorical cross-entropy function since at the time of writing this, Core ML 3 supports it as the only updatable CNN loss layer.

You can view the full Notebook here:

We ran the above model for 5 epochs and got an accuracy of around 73% which is pretty decent.

Now that we’ve got our Keras model, all that is left is converting it into an updatable Core ML Model

Creating an Updatable Core ML Model

To create an updatable Model, we need to set an isUpdatable flag on the model and some individual layers. Besides that, we need to define the training inputs, loss function, optimizers and other parameters like epochs and batch size on our core ml model.

We need to update the coremltools to 3.0. At the time of writing the beta 6 release was the latest. Just use the pip command to install it:

pip install coremltools==3.0b6

Converting Keras to Core ML Model

import coremltools

output_labels = ['Cat', 'Dog']
coreml_model = coremltools.converters.keras.convert('new_model_category.h5', input_names=['image'], output_names=['output'],
                                                   class_labels=output_labels,
                                                   image_input_names='image')

coreml_model.author = 'Anupam Chugh'
coreml_model.short_description = 'Cat Dog Classifier converted from a Keras model'
coreml_model.input_description['image'] = 'Takes as input an image'
coreml_model.output_description['output'] = 'Prediction as cat or dog'
coreml_model.output_description['classLabel'] = 'Returns Cat Or Dog as class label'


coreml_model.save('crossenteropymodel.mlmodel')

Generate an updatable Core ML Model

First, load the specs from the existing Core ML model into a NeuralNetworkBuilder.

coreml_model_path = "./crossenteropymodel.mlmodel"

spec = coremltools.utils.load_spec(coreml_model_path)
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)
builder.inspect_layers(last=3)
builder.inspect_input_features()

By inspecting the layers, we can determine the dense_layers that need to made updatable.
Along with that, we need to set the input parameters for the training input for on-device training.

neuralnetwork_spec = builder.spec

neuralnetwork_spec.description.input[0].type.imageType.width = 150
neuralnetwork_spec.description.input[0].type.imageType.height = 150

neuralnetwork_spec.description.metadata.author = 'Anupam Chugh'
neuralnetwork_spec.description.metadata.license = 'MIT'
neuralnetwork_spec.description.metadata.shortDescription = (
        'Cat Dog Classifier converted from a Keras model')

Now, we need to make the last two dense layers as updatable and set the loss function input. The loss function input is the output value from the softmax activation layer:

ios-coreml3-updatable-model-activation-layer

model_spec = builder.spec
builder.make_updatable(['dense_5', 'dense_6'])
builder.set_categorical_cross_entropy_loss(name='lossLayer', input='output')

from coremltools.models.neural_network import SgdParams
builder.set_sgd_optimizer(SgdParams(lr=0.01, batch=5))
builder.set_epochs(2)

make_updatable method is used to make a layer updatable. It requires a list of layer names. The layers are dense_5 and dense_6 in our case, available from neural network builder properties.

Now we just need to set the training input description and save our specifications in a new model as shown below:

model_spec.isUpdatable = True
model_spec.specificationVersion = coremltools._MINIMUM_UPDATABLE_SPEC_VERSION

model_spec.description.trainingInput[0].shortDescription = 'Image for training and updating the model'
model_spec.description.trainingInput[1].shortDescription = 'Set the value as Cat or Dog and update the model'

coremltools.utils.save_spec(model_spec, "CatDogUpdatable.mlmodel")

specificationVersion property is important since it ensures that the correct version number is used for the mlmodel. Core ML 3 is v4 so the above updatable model works on v4 and above only.

Our Updatable Core ML Model is now ready to use!
Let’s import it into an XCode project. You shall see a model description similar to the one in the image below:

ios-coreml3-model-description

It’s clear from the image above that two new sections, Update and Properties are added with Core ML 3.
Update section contains the input description for training the input.

In the next part of this article, we’ll actually train a batch of images on the iOS device using the above generated updatable model in Core ML 3.

One Reply to “Core ML 3 Create Updatable Models”

  1. I think that is one of the so much important info for me. And i am glad studying your article. But want to statement on some basic issues, The web site taste is great, the articles is actually nice : D. Good task, cheers

Leave a Reply

Your email address will not be published. Required fields are marked *