Version: Next

Use Pinned Models

In order to query a trained model for predictions, you have to pin the model. This allows you to access a snapshot of the model, even as the model is being continuously changed and updated by Re:infer users. You can have multiple pinned versions of a model, allowing you to develop against a newer version while using an older version in production.

Re:infer provides a number of ways of using a pinned model to get predictions. Using the Trigger API, which iterates through existing comments in a dataset, is a common choice for many simple use-cases including RPA-driven automation. Alternatively, you can use the Predict endpoints to either get predictions for specific comments by ID, or get predictions for emails or other pieces of text not stored in Re:infer.

Configuration

In order to query a pinned model for predictions, you need to specify the dataset and model version. In order to use the predictions in your application, you need to determine appropriate confidence thresholds for each label you want to predict. The table below provides more detail.

Please consult our knowledge base to learn how to verify that your model is performing well.

ItemDescription
Dataset nameThe dataset contains the training data used to create a model, and the label taxonomy describing available labels. A dataset together with a model version uniquely identifies the model you want to query for predictions.
Model versionThe model version is an integer that points to a particular snapshot of the model. This provides deterministic results for your application even as the model is being continuously improved. In order to use a model version in the API, you need to "pin" it to ensure it will not be deleted when new models are trained for the same dataset. You can do so in the "Models" page of your dataset, where you can also see all available pinned model versions.

When using Predict endpoints, the model version is included in each API request. When using Trigger endpoints, the model version is specified when creating a Trigger.
Mapping between label names and thresholdsThe model will predict zero, one, or multiple labels for each comment. Each predicted label will have an associated confidence score. To convert this prediction into a "Yes/No" answer, the confidence level needs to be checked against a threshold: if the confidence score is above the threshold, the label should be counted as predicted, if the confidence score is below the threshold, the label should be discarded.

The threshold needs to be picked according to the requirements of your specific use case and will be different for each label. Selecting a threshold should be done by inspecting the validation page and choosing a value that strikes the appropriate balance between false positives and false negatives.

If a label's validation scores are not reliable (this is flagged on the Validation page) then the predictions can't be reliably thresholded. In such a case, for the purposes of testing the threshold can be set to an arbitrary value (e.g. 0.15), however such a label should be excluded entirely in a production scenario.

For convenience, the thresholds can be specified in the API request so that the API response won't contain labels with confidence scores below the threshold. If you want to receive labels with confidence scores below the threshold (eg. for debugging purposes), you can instead apply the thresholds in your application.

Change Management

In production, the model is expected to be continuously improved by labelling exceptions and accounting for data drift. Your application should support upgrading the model by making the model version and confidence thresholds configurable.