Skip to main content

​
Cleanlab Integration

Cleanlab provides tools for evaluating and improving the quality of LLM outputs. The Cleanlab TLM (Trustworthy Language Model) library helps you assess the trustworthiness of LLM responses and provides explanations for its evaluations. This guide shows you how to integrate Cleanlab TLM with Langtrace to monitor and trace your Cleanlab interactions.

​
Installation

First, install the required packages: 
pip install cleanlab_tlm langtrace-python-sdk python-dotenv

​
Integration Example

Here’s a complete example showing how to integrate Cleanlab TLM with Langtrace: 
import os

from dotenv import find_dotenv, load_dotenv
from langtrace_python_sdk import langtrace
from langtrace_python_sdk.utils.with_root_span import with_langtrace_root_span

_ = load_dotenv(find_dotenv())

langtrace.init()

from cleanlab_tlm import TLM
from openai import OpenAI

openai_client = OpenAI()

tlm = TLM(
    api_key=os.getenv("TLM_API_KEY"),
    options={"log": ["explanation"], "model": "gpt-4o-mini"},
)


def inference(prompt: str):
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": prompt},
        ],
        stream=False,
    )
    response_text = response.choices[0].message.content
    return response_text


@with_langtrace_root_span("Get Trustworthiness Score")
def inference_get_trustworthiness_score(prompt: str):
    response = inference(prompt)
    return tlm.get_trustworthiness_score(prompt, response)


print(inference_get_trustworthiness_score("How many r's are in strawberry?"))

​
What Gets Traced

When you integrate Cleanlab TLM with Langtrace, the following information is captured in your traces:
  • Service Name: Cleanlab
  • Service Type: framework
  • Inputs: The prompt or question sent to the model
  • Response: The model’s response
  • Trustworthiness Score: The score calculated by Cleanlab TLM
  • Explanation: The reasoning behind the trustworthiness assessment (when enabled)

​
Viewing Traces

After running your application with the Cleanlab TLM integration, you can view the traces in the Langtrace dashboard. The traces will show the Cleanlab TLM operations, including:
  1. The prompt operation
  2. The trustworthiness score calculation
Each trace will include the inputs, outputs, and metadata from your Cleanlab TLM interactions.

​
Example Traces

Here’s how Cleanlab TLM traces appear in the Langtrace dashboard: Cleanlab TLM Trace Overview You can see detailed information about each trace, including the trustworthiness score and explanation: Cleanlab TLM Trace Details

​
Additional Configuration

You can customize your Cleanlab TLM configuration by adjusting the options parameter: 
tlm = TLM(
    api_key=api_key,
    options={
        "log": ["explanation", "prompt"],  # Log explanation and prompt
        "model": "gpt-4o-mini",            # Specify the model
        # Add other Cleanlab TLM options as needed
    }
)
For more information about Cleanlab and its capabilities, visit cleanlab.ai.
⌘I