Generating and posting Vega-Lite charts

Vega-Lite is a high-level grammar of interactive graphics and, at Criterion AI, we make use of it to present charts generated by models in our web interface. It is a way for you, as a model developer, to present useful graphics to the users of your custom model and, thanks to its rich structure, you can generate really impressive visualizations. Learn more about Vega-Lite on the project's website.

To generate Vega-Lite charts, you can make use of the really great Python package called Altair. Altair is a declarative statistical visualization library for Python, based on Vega-Lite, and the source for it is available on GitHub. You can get inspired by some of the impressive visualizations you can make with Altair (and Vega-Lite) by checking out the project's example gallery.

Criterion AI makes use of Vega-Embed to render the charts generated by models using Altair in our web interface. This means that any chart that can be rendered with Vega-Embed can be submitted to Criterion AI. Charts can be submitted both during the training process of a model and during the creation of a test report on a model. Below, you can see some examples of Vega-Lite charts that have been generated during the training process and submitted to Criterion AI on the model details page.

To post charts to Criterion AI, first generate them using Altair from your training or testing script in Python. Then, serialize them to JSON (learn how to on Altair's documentation site). You can submit the JSON for the chart using one of Criterion AI's RESTful APIs, which you can extract from the model's info.json file.

The charts_url in the model description file will hold the value of the relative path along with the necessary query parameters, which you need in order to submit the POST requests containing the charts serialized as JSON. Combine this value with the value of the host_name key to get the entire URL you need to call.

Consider the sample info.json file below:

{
  "id": "cc2e041d-737e-428e-92e3-a981a291e45b",
  "name": "Sample Model",
  "datasets": [
    {
      "id": "8f1731e4-6606-4382-90a6-8308066ed498",
      "bucket": "gs://8f1731e4-6606-4382-90a6-8308066ed498.datasets.criterion.ai",
      "name": "Sample Dataset"
    }
  ],
  "settings": {
    "imageWidth": 224,
    "imageHeight": 224,
    "type": "rgb",
    "rotationRange": 0,
    "horizontalSymmetry": false,
    "verticalSymmetry": false,
    "classWeights": "",
    "ignoreClasses": "",
    "img_format": "jpeg"
  },
  "api_key": "2O57R13JR6hfV9k17X0zT59E",
  "host_name": "https://app.criterion.ai",
  "charts_url": "/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/charts?key=2O57R13JR6hfV9k17X0zT59E",
  "complete_url": "/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/complete?key=2O57R13JR6hfV9k17X0zT59E&modelPath=",
  "remote_url": "/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/remotemonitor?key=2O57R13JR6hfV9k17X0zT59E"
}

The values of the host_name and charts_url keys, respectively, are https://app.criterion.ai and /api/models/cc2e041d-737e-428e-92e3-a981a291e45b/charts?key=2O57R13JR6hfV9k17X0zT59E. Combine these to get the complete URL: https://app.criterion.ai/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/charts?key=2O57R13JR6hfV9k17X0zT59E. You should use this URL in your POST requests.

The content type of the POST request should be application/json. The body of the request should be the JSON object representing the chart. An example of a chart serialized to JSON is presented below.

{
  "$schema": "https://vega.github.io/schema/vega-lite/v2.json",
  "config": {
    "view": {
      "height": 300,
      "width": 400
    }
  },
  "data": {
    "url": "https://vega.github.io/vega-datasets/data/cars.json"
  },
  "encoding": {
    "color": {
      "field": "Origin",
      "type": "nominal"
    },
    "x": {
      "field": "Horsepower",
      "type": "quantitative"
    },
    "y": {
      "field": "Miles_per_Gallon",
      "type": "quantitative"
    }
  },
  "mark": "point"
}

The chart above is generated by following Altair code (taken from this example).

import altair as alt
from vega_datasets import data

chart = alt.Chart(data.cars.url).mark_point().encode(
    x='Horsepower:Q',
    y='Miles_per_Gallon:Q',
    color='Origin:N'
)

serialized = chart.to_json()

The entire POST request can be seen in its raw version below.

POST https://app.criterion.ai/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/charts?key=2O57R13JR6hfV9k17X0zT59E HTTP/1.1
Host: app.criterion.ai
Content-Type: application/json
Content-Length: 351

{"$schema":"https://vega.github.io/schema/vega-lite/v2.json","config":{"view":{"height":300,"width":400}},"data":{"url":"https://vega.github.io/vega-datasets/data/cars.json"},"encoding":{"color":{"field":"Origin","type":"nominal"},"x":{"field":"Horsepower","type":"quantitative"},"y":{"field":"Miles_per_Gallon","type":"quantitative"}},"mark":"point"}

In Python, you can use the requests library to easily submit the updates. See the example below (note that the variable serialized contains the chart serialized as JSON).

import json, requests
url = "https://app.criterion.ai/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/charts?key=2O57R13JR6hfV9k17X0zT59E"
headers = {'Content-Type': 'application/json'}
r = requests.post(url, data=serialized, headers=headers)

The server will respond with a 200 OK status code, which indicates that the chart has been posted successfully. Other possible response codes are 401 Unauthorized, which means that the API key posted in the query parameter key is incorrect, and 400 Bad Request, which means that the format of the request is incorrect.

Posting charts to test reports

Similar to posting charts to models, you can post charts to test reports. Consider the description file (JSON) for a sample test report below.

{
  "id": "ee4447ec-3719-4bb1-96c4-313335527a11",
  "dataset": {
    "id": "f01eda35-d537-4e5e-9298-6366ddb702f9",
    "bucket": "gs://f01eda35-d537-4e5e-9298-6366ddb702f9.datasets.criterion.ai",
    "name": "Test Dataset"
  },
  "api_key": "2O57R13JR6hfV9k17X0zT59E",
  "host_name": "https://app.criterion.ai",
  "charts_url": "/api/models/c0f4dcb9-7bee-4c5e-ae17-3d63aa43ac1b/testreports/ee4447ec-3719-4bb1-96c4-313335527a11/charts?key=2O57R13JR6hfV9k17X0zT59E",
  "complete_url": "/api/models/c0f4dcb9-7bee-4c5e-ae17-3d63aa43ac1b/testreports/ee4447ec-3719-4bb1-96c4-313335527a11/complete?key=2O57R13JR6hfV9k17X0zT59E"
}

Like the info.json file for a model training job, there is a charts_url key containing the relative path to the API endpoint that you need to use in your POST requests. Using this value instead, everything else works the same way for test reports.

Naming your charts

You can add a name to your chart simply by appending &name= followed by the name of the cart to the value of the charts_url key. To post a chart with the name My Demo Chart for the example model from above, use this URL in your POST requests instead:

  • https://app.criterion.ai/api/models/cc2e041d-737e-428e-92e3-a981a291e45b/charts?key=2O57R13JR6hfV9k17X0zT59E&name=My%20Demo%20Chart

Similarly, for the example test report above, submit POST requests to:

  • https://app.criterion.ai/api/models/c0f4dcb9-7bee-4c5e-ae17-3d63aa43ac1b/testreports/ee4447ec-3719-4bb1-96c4-313335527a11/charts?key=2O57R13JR6hfV9k17X0zT59E&name=My%20Demo%20Chart

Make sure to URL encode the name of your chart before appending it to the URL. You can learn more about URL encoding in Python on URLEncoder.io.

Testing it out

You can test this out for yourself by creating a toy model (or a toy test report) in Criterion AI and post charts to it. If you experience any issues with the API, please contact our support. We will be happy to help you out.

Still need help? Contact Us Contact Us