Supademo Logo
Supademo Logo

Made with Supademo

Demo Image 1
Demo Image 2
Demo Image 3

Creating a Python Evaluator - Blog

Author Image

Dhruv Singh

Updated: Mar 18, 2024

Description

Click through a step-by-step, interactive demo walkthrough of Honeyhive, powered by Supademo.

Steps

1
Select which event type you want to run your evaluator on. Click on "Model".
2
Let's setup an evaluator for the LLM step. Click on "Model".
3
Now, let's specific the model event to run your evaluator on. Click on "All Completions".
4
Select "Model Completion" as the event.
5
Next, click on "Show Schema" to review what fields are available.
6
Now, click on "content" under `outputs` to add that field to your evaluator.
7
Following that, we can paste the reference to that field in the "Console" code.
8
Next up, we'll add a simple `len(model_response)` to get the length of response
9
Click on "Get datapoints" to test your new evaluator against production logs.
10
Video step
11
After that, click on "Run all".
12
Let's add a simple passing range for what the minimum & maximum length should be.
13
Hit the toggle under "Enable In Production" to enable it in production
14
Click on "Advanced Settings".
15
Set a sampling percentage based on how expensive the evaluator might be to run.
16
Click on "Create" to create your evaluator!
17
Your evaluator is now enabled over production logs!
18
Enjoyed the guided demo?
Creating a Python Evaluator - Blog