Providing updates on the model training progress

You can provide updates on the model training progress in real time, which makes for a great user experience. The updates will be presented on an interactive chart on the model details page and will update automatically whenever a new updated is being posted. See an example of such a chart below.

You, as a model developer, can post back any type of numerical information to Criterion AI via the RESTful endpoint included in the info.json file (i.e., the model description file) located in the root of the job bucket. The remote_url key in the model description file will hold the value of the relative path along with the necessary query parameters, which you need in order to submit the POST requests containing the model updates. Combine this value with the value of the host_name key to get the entire URL you need to call.

Consider the sample info.json file below:

{
  "id": "cc2e041d-737e-428e-92e3-a981a291e45b",
  "name": "Sample Model",
  "datasets": [
    {
      "id": "8f1731e4-6606-4382-90a6-8308066ed498",
      "bucket": "gs://8f1731e4-6606-4382-90a6-8308066ed498.datasets.criterion.ai",
      "name": "Sample Dataset"
    }
  ],
  "settings": {
    "imageWidth": 224,
    "imageHeight": 224,
    "type": "rgb",
    "rotationRange": 0,
    "horizontalSymmetry": false,
    "verticalSymmetry": false,
    "classWeights": "",
    "ignoreClasses": "",
    "img_format": "jpeg"
  },
  "api_key": "2O57R13JR6hfV9k17X0zT59E",
  "host_name": "https://app.criterion.ai",
  "charts_url": "/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/charts?key=2O57R13JR6hfV9k17X0zT59E",
  "complete_url": "/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/complete?key=2O57R13JR6hfV9k17X0zT59E&modelPath=",
  "remote_url": "/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/remotemonitor?key=2O57R13JR6hfV9k17X0zT59E"
}

The values of the host_name and remote_url keys, respectively, are https://app.criterion.ai and /api/models/cc2e041d-737e-428e-92e3-a981a291e45b/remotemonitor?key=2O57R13JR6hfV9k17X0zT59E. Combine these to get the complete URL: https://app.criterion.ai/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/remotemonitor?key=2O57R13JR6hfV9k17X0zT59E. You should use this URL in your POST requests.

The content type of the POST request should be application/x-www-form-urlencoded. The body of the request should contain a key named data with the value being a JSON object containing the model update information you wish to present to the user. An example of what such an update may look like (in JSON) is presented below.

{
  "epoch": 42,
  "acc": 0.99,
  "val_acc": 0.98,
  "loss": 0.1,
  "val_loss": 0.15
}

The JSON object must contain a key named epoch with the value of the current epoch. In addition, it may include any number of keys containing values that you think will be interesting for the user. In this example, we have included some accuracy and loss metrics for both the training and validation sets. The only rule that applies to the values of the additional keys is that the values must be numerical so that they can be plotted on a chart.

The entire POST request can be seen in its raw version below.

POST https://app.criterion.ai/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/remotemonitor?key=2O57R13JR6hfV9k17X0zT59E HTTP/1.1
Host: app.criterion.ai
Content-Type: application/x-www-form-urlencoded
Content-Length: 160

data=%7B%0A%09%22epoch%22%3A%2042%2C%0A%09%22acc%22%3A%200.99%2C%0A%09%22val_acc%22%3A%200.98%2C%0A%09%22loss%22%3A%200.1%2C%0A%09%22val_loss%22%3A%200.15%0A%7D

In Python, you can use the requests library to easily submit the updates. See the example below.

import json, requests
url = "https://app.criterion.ai/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/remotemonitor?key=2O57R13JR6hfV9k17X0zT59E"
update = {
  "epoch": 42,
  "acc": 0.99,
  "val_acc": 0.98,
  "loss": 0.1,
  "val_loss": 0.15
}
payload = { "data": json.dumps(update) }
r = requests.post(url, data=payload)

The server will respond with a 200 OK status code, which indicates that the model training update has been posted successfully. Other possible response codes are 401 Unauthorized, which means that the API key posted in the query parameter key is incorrect, and 400 Bad Request, which means that the format of the request is incorrect.

You can test this out for yourself by creating a toy model in Criterion AI and post updates to it. If you experience any issues with the API, please contact our support. We will be happy to help you out.

Still need help? Contact Us Contact Us