Core ML got a big boost this year with the Core ML 3 update during WWDC 2019. Among the many improvements, we got, On-Device Learning stands out. The goal of this article is to show you the process to create your own Core ML 3 models which can be updated on your iPhone or iPad.
Though it’s fancy to do an on-device model training, but it isn’t feasible given the long duration it’ll take for the model to train on iOS Devices. At best, it is recommended to retrain a model on the device, specifically for a user.
Before we go down the implementation road, let’s admire the other updates Core ML 3 has in store for us.
What’s New In Core ML 3
- 70 new neural network layers – New layers allow more complex neural network models to be converted to Core ML without the need of writing custom layers.
- Variety of new models – Models such as
Linked Modelswill just help to solve more machine learning problems.
- Core ML API has more abstraction – You need not convert images to
CVPixelBufferfor CNN models anymore.
MLFeatureValuedoes that for you automatically.
- ML Model Configuration – We can now configure the ML Model as per our needs. Example constraining
MLComputeUnitsto cpu, gpu and/or neural engine.
- On Device Learning – Now you can easily train/retrain your models on the device itself. No need to recompile it from your machine!
Linked Models allow reusability across models. So if two models are using another model, that model would now be loaded just once.
Let’s deep dive into the code now.
We’ll be creating a Keras Cat vs Dog Image Classifier CNN Model using Categorical Cross-Entropy loss function. If you directly want to jump onto the Core ML part skip the next section.
Creating a Keras Image Classifier Model
The model is compiled with the categorical cross-entropy function since at the time of writing this, Core ML 3 supports it as the only updatable CNN loss layer.
You can view the full Notebook here:
We ran the above model for 5 epochs and got an accuracy of around 73% which is pretty decent.
Now that we’ve got our Keras model, all that is left is converting it into an updatable Core ML Model
Creating an Updatable Core ML Model
To create an updatable Model, we need to set an
isUpdatable flag on the model and some individual layers. Besides that, we need to define the training inputs, loss function, optimizers and other parameters like epochs and batch size on our core ml model.
We need to update the
coremltools to 3.0. At the time of writing the beta 6 release was the latest. Just use the pip command to install it:
pip install coremltools==3.0b6
Converting Keras to Core ML Model
import coremltools output_labels = ['Cat', 'Dog'] coreml_model = coremltools.converters.keras.convert('new_model_category.h5', input_names=['image'], output_names=['output'], class_labels=output_labels, image_input_names='image') coreml_model.author = 'Anupam Chugh' coreml_model.short_description = 'Cat Dog Classifier converted from a Keras model' coreml_model.input_description['image'] = 'Takes as input an image' coreml_model.output_description['output'] = 'Prediction as cat or dog' coreml_model.output_description['classLabel'] = 'Returns Cat Or Dog as class label' coreml_model.save('crossenteropymodel.mlmodel')
Generate an updatable Core ML Model
First, load the specs from the existing Core ML model into a NeuralNetworkBuilder.
coreml_model_path = "./crossenteropymodel.mlmodel" spec = coremltools.utils.load_spec(coreml_model_path) builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec) builder.inspect_layers(last=3) builder.inspect_input_features()
By inspecting the layers, we can determine the dense_layers that need to made updatable.
Along with that, we need to set the input parameters for the training input for on-device training.
neuralnetwork_spec = builder.spec neuralnetwork_spec.description.input.type.imageType.width = 150 neuralnetwork_spec.description.input.type.imageType.height = 150 neuralnetwork_spec.description.metadata.author = 'Anupam Chugh' neuralnetwork_spec.description.metadata.license = 'MIT' neuralnetwork_spec.description.metadata.shortDescription = ( 'Cat Dog Classifier converted from a Keras model')
Now, we need to make the last two dense layers as updatable and set the loss function input. The loss function input is the output value from the softmax activation layer:
model_spec = builder.spec builder.make_updatable(['dense_5', 'dense_6']) builder.set_categorical_cross_entropy_loss(name='lossLayer', input='output') from coremltools.models.neural_network import SgdParams builder.set_sgd_optimizer(SgdParams(lr=0.01, batch=5)) builder.set_epochs(2)
make_updatable method is used to make a layer updatable. It requires a list of layer names. The layers are
dense_6 in our case, available from neural network builder properties.
Now we just need to set the training input description and save our specifications in a new model as shown below:
model_spec.isUpdatable = True model_spec.specificationVersion = coremltools._MINIMUM_UPDATABLE_SPEC_VERSION model_spec.description.trainingInput.shortDescription = 'Image for training and updating the model' model_spec.description.trainingInput.shortDescription = 'Set the value as Cat or Dog and update the model' coremltools.utils.save_spec(model_spec, "CatDogUpdatable.mlmodel")
specificationVersion property is important since it ensures that the correct version number is used for the mlmodel. Core ML 3 is v4 so the above updatable model works on v4 and above only.
Our Updatable Core ML Model is now ready to use!
Let’s import it into an XCode project. You shall see a model description similar to the one in the image below:
It’s clear from the image above that two new sections, Update and Properties are added with Core ML 3.
Update section contains the input description for training the input.
In the next part of this article, we’ll actually train a batch of images on the iOS device using the above generated updatable model in Core ML 3.