Pythia API can be used to detect hallucinations in NLP applications such as chatbots, RAG systems, and text summarization models.
Prerequisites
Check out the quickstart guide for getting started with Pythia. It outlines the steps for requesting an API key, installing Wisecube SDK, and authenticating your API key.
The ask_pythia API endpoint takes in a reference, response, and question to compare claims and detect hallucinations. Each API call serves a different purpose, enabling advanced hallucination detection and claim verification in LLMs.
The input/ output validators in Pythia run with each API request to ensure inputs meet quality standards and outputs are factually correct.
Example
reference = ["2-D ECHOCARDIOGRAM,Multiple views of the heart and great vessels reveal normal intracardiac and great vessel relationships. Cardiac function is normal. There is no significant chamber enlargement or hypertrophy. There is no pericardial effusion or vegetations seen. Doppler interrogation, including color flow imaging, reveals systemic venous return to the right atrium with normal tricuspid inflow. Pulmonary outflow is normal at the valve. Pulmonary venous return is to the left atrium. The interatrial septum is intact. Mitral inflow and ascending aorta flow are normal. The aortic valve is trileaflet. The coronary arteries appear to be normal in their origins. The aortic arch is left-sided and patent with normal descending aorta pulsatility."]
response = "65 year old female presented with pT3 pN1a moderately differentiated adenocarcinoma of rectum on 9/15/2021."
question = "What was theTNM stage of the cancer?"
client.ask_pythia(reference,response,question)
Output
{
"data": {
"askPythia": {
"claims": [
{
"claim": [
"65 year old female",
"presented with",
"pT3 pN1a moderately differentiated adenocarcinoma of rectum"
],
"class": "entailment"
},
{
"claim": [
"Presentation of pT3 pN1a moderately differentiated adenocarcinoma of rectum",
"occurred on",
"9/15/2021"
],
"class": "entailment"
},
{
"claim": [
"pT3 pN1a moderately differentiated adenocarcinoma of rectum",
"is",
"TNM stage of the cancer"
],
"class": "entailment"
}
],
"metrics": {
"accuracy": 1.0,
"contradiction": 0.0,
"entailment": 1.0,
"neutral": 0.0
},
"validatorsResults": [
{
"validatedField": "input_response",
"validator": {
"id": 1,
"name": "detect_pii",
"description": "The Anonymize Scanner acts as your digital guardian, ensuring your user prompts remain confidential and free from sensitive data exposure.",
"source": "llm Guard",
"alwaysOn": true,
"input": "input_reference",
"output": "input_response"
},
"valid": true,
"isValid": true,
"riskScore": 0.0,
"errorMessage": null
},
{
"validatedField": "input_reference",
"validator": {
"id": 1,
"name": "detect_pii",
"description": "The Anonymize Scanner acts as your digital guardian, ensuring your user prompts remain confidential and free from sensitive data exposure.",
"source": "llm Guard",
"alwaysOn": true,
"input": "input_reference",
"output": "input_response"
},
"valid": false,
"isValid": false,
"riskScore": 1.0,
"errorMessage": "Error, the text sent has sensitive data -> \nWJ Health Care\nAnatomic Pathology\n$21-14349\nOne Hospital Drive, Columbia, MO 65212\nPhone: [REDACTED_PHONE_NUMBER_1] Fax. [REDACTED_PHONE_NUMBER_2]\nSURGICAL PATHOLOGY REPORT\n** FINAL **\nPatient:\nOperation Date:\n9/15/2021\nPatient ID: (am,\nAccession Date:\n9/16/2021\nService:\nGeneral Surgery Oncology\nSignout Date:\n9/21/2021\nLocation:\nOncology\nFacility:\nUniversity of MO Hospital (a\nAttending Physician:\n[REDACTED_PERSON_1], M.D.\nDOB: ey\nOther Physician(s):\n{None Given}\nSex: GEE\nFINAL DIAGNOSIS:\n1. COL..."
}
]
}
}
}
Empty Result Error
Calling ask_pythia() with random text throws an error due to empty results.
Random Text Call Example
{
"reference": ["All new patients with likely advanced lung cancer were offered prehabilitation at respiratory clinic, if fit for further investigation. Prehabilitation included assessment and appropriate intervention from a consultant in palliative medicine, registered dietitian and rehabilitation physiotherapist. Four objective endpoints were identified, namely admissions to hospital, time spent in the hospital, treatment rates and overall survival. Outcomes were to be compared with 178 prehab eligible historical controls diagnosed from 2019 to 2021."],
"response": "From July 2021 to June 2023, 65 patients underwent prehabilitation and 72% of patients underwent all 3 interventions. 54 patients had a stage 3 or 4 lung cancer. In the prehab group, fewer patients attended Accident and Emergency (31.5 vs 37.4 attendances per 100 patients) and fewer were admitted (51.9 vs 67.9) when compared with historical controls. Those receiving prehab spent a lot less time in the hospital (129.7 vs 543.5 days per 100 patients) with shorter admissions (2.5 vs 8 days). Systemic anticancer treatment rates increased in the short term but were broadly similar overall. Median survival was higher in the prehabilitation group (0.73 vs 0.41 years, p=0.046)."
}
Output Error
{
"errors": [
{
"message": "java.lang.reflect.UndeclaredThrowableException: null",
"locations": [],
"path": [
"askPythia"
],
"extensions": {
"errorType": "INTERNAL"
}
}
],
"data": {
"askPythia": null
}
}