Training and testing models locally
You can test out your model type by making use of Google Cloud Platform's command-line tools. You can download them for free from Google's website. You should be making use of the command
gcloud ml-engine local train to run the training and testing scripts in your custom model on your local machine. Learn more about that command by following this link.
You will need to create a model based on your new model type to get the details you need to run the command for training locally. Navigate to the model zoo and find your custom model type.
Click on it and click on the button Create model from this type. Follow the wizard to create the new model. When the model has been created, you can simulate the training by clicking on Options in the top left-hand corner and then on Simulate training.
Clicking on Simulate training will force Criterion AI to reset the model (if it has already been trained once before) and set the start time to the point in time when you clicked the button. It will essentially do everything that will also be done when initiating the training of the model in the cloud, except for actually running the gcloud command to submit the training job to Google Cloud ML Engine. This way, you can interact with your model locally via your training script (by running the
gcloud ml-engine local train command) and test if your integration with Criterion AI's APIs work the way it should.
When clicking on Simulate training, the
info.json file generated for the model will automatically be opened in a new window or downloaded as a JSON file (depending on your browser settings). Use this
info.json file when running your training script locally.
By default, you will not be able to access the datasets in the info.json file from your local machine. Thus, you must download your data to your own computer if you wish to run your training script. When you are ready to test your model type with online data, please submit a ticket to our support team to request access to the online datasets from your local environment.
Testing your testing script
Just like you can simulate training in Criterion AI, you can simulate the creation of a test report (i.e., your testing script). Create a test report like you normally would (after the model has completed training) and make sure to check the box with the label Simulate testing? This will create a new test report in Criterion AI without actually submitting the testing job to Google Cloud ML Engine. Using the
gcloud ml-engine local train command, you can run your testing script locally to test your integration with Criterion AI's APIs.
Similar to running your training script locally, your testing script will not be able to access online datasets. Submit a ticket to our support to get access to your online datasets for testing purposes.
Getting your model type approved
When you have tested both your training and testing scripts thoroughly, you can request that your model type be approved in the platform so that users in your organization can initiate training jobs for models based on your model type in the cloud. Please contact our support to initiate the approval process.