If you are using the DefaultAzureCredential class from the Azure Identity SDK while your user account is associated with multiple tenants, you may find yourself frequently running into API authentication errors (such as HTTP 401/Unauthorized). This post is for you!
These are your two options for successful authentication from a non-default tenant:
Setup your environment precisely to force DefaultAzureCredential to use the desired tenant
Use a specific credential class and explicitly pass in the desired tenant ID
Option 1: Get DefaultAzureCredential working
The DefaultAzureCredential class is a credential chain, which means that it tries a sequence of credential classes until it finds one that can authenticate successfully. The current sequence is:
EnvironmentCredential
WorkloadIdentityCredential
ManagedIdentityCredential
SharedTokenCacheCredential
AzureCliCredential
AzurePowerShellCredential
AzureDeveloperCliCredential
InteractiveBrowserCredential
For example, on my personal machine, only two of those credentials can retrieve tokens:
AzureCliCredential: from logging in with Azure CLI (az login)
AzureDeveloperCliCredential: from logging in with Azure Developer CLI (azd auth login)
Many developers are logged in with those two credentials, so it's crucial to understand how this chained credential works. The AzureCliCredential is earlier in the chain, so if you are logged in with that, you must have the desired tenant set as the "active tenant". According to Azure CLI documentation, there are two ways to set the active tenant:
az account set --subscription SUBSCRIPTION-ID where the subscription is from the desired tenant
az login --tenant TENANT-ID, with no subsequent az login commands after
Whatever option you choose, you can confirm that your desired tenant is currently the default by running az account show and verifying the tenantId in the account details shown.
If you are only logged in with the azd CLI and not the Azure CLI, you have a problem: the azd cli does not currently have a way to set the active tenant. If that credential is called with no additional information, azd assumes your home tenant, which may not be desired. The azd credential does check for a system variable called AZURE_TENANT_ID, however, so you can try setting that in your environment before running code that uses DefaultAzureCredential. That should work as long as the DefaultAzureCredential code is truly running in the same environment where AZURE_TENANT_ID has been set.
Option 2: Use specific credentials
Several credential types allow you to explicitly pass in a tenant ID, including both the AzureCliCredential and AzureDeveloperCliCredential. If you know that you’re always going to be logging in with a specific CLI, you can change your code to that credential:
As a best practice, I always like to log out exactly what credential I'm calling and whether I'm passing in a tenant ID, to help me spot any misconfiguration from my logs.
⚠️ Be careful when replacing DefaultAzureCredential if your code will be deployed to a production host! That means you were previously relying on it using the ManagedIdentityCredential in the chain, and that you now need to call that credential class specifically. You will also need to pass in the managed identity ID, if using user-assigned identity instead of system-assigned identity.
For example, using managed identity in the Python SDK with user-assigned identity:
I ❤️ when companies offer free tiers for developer services, since it gives everyone a way to learn new technologies without breaking the bank. Free tiers are especially important for students and people between jobs, where the desire to learn is high but the available cash is low.
That's why I'm such a fan of GitHub Models: free, high-quality generative AI models available to anyone with a GitHub account. The available models include the latest OpenAI LLMs (like o3-mini), LLMs from the research community (like Phi and Llama), LLMs from other popular providers (like Mistral and Jamba), multimodal models (like gpt-4o and llama-vision-instruct) and even a few embedding models (from OpenAI and Cohere). So cool! With access to such a range of models, you can prototype complex multi-model workflows to improve your productivity or heck, just make something fun for yourself. 🤗
To use GitHub Models, you can start off in no-code mode: open the playground for a model, send a few requests, tweak the parameters, and check out the answers. When you're ready to write code, select "Use this model". A screen will pop up where you can select a programming language (Python/JavaScript/C#/Java/REST) and select an SDK (which varies depending on model). Then you'll get instructions and code for that model, language, and SDK.
But here's what's really cool about GitHub Models: you can use them with all the popular Python AI frameworks, even if the framework has no specific integration with GitHub Models. How is that possible?
The vast majority of Python AI frameworks support the OpenAI Chat Completions API, since that API became a defacto standard supported by many LLM API providers besides OpenAI itself.
GitHub Models also provide OpenAI-compatible endpoints for chat completion models.
Therefore, any Python AI framework that supports OpenAI-like models can be used with GitHub Models as well. 🎉
To prove my claim, I've made a new repository with examples from eight different Python AI agent packages, all working with GitHub Models: python-ai-agent-frameworks-demos. There are examples for AutoGen, LangGraph, Llamaindex, OpenAI Agents SDK, OpenAI standard SDK, PydanticAI, Semantic Kernel, and SmolAgents. You can open that repository in GitHub Codespaces, install the packages, and get the examples running immediately.
Now let's walk through the API connection code for GitHub Models for each framework. Even if I missed your favorite framework, I hope my tips here will help you connect any framework to GitHub Models.
OpenAI sdk
I'll start with openai, the package that started it all!
The code above demonstrates the two key parameters we'll need to configure for all frameworks:
api_key: When using OpenAI.com, you pass your OpenAI API key here. When using GitHub Models, you pass in a Personal Access Token (PAT). If you open the repository (or any repository) in GitHub Codespaces, a PAT is already stored in the GITHUB_TOKEN environment variable. However, if you're working locally with GitHub Models, you'll need to generate a PAT yourself and store it. PATs expire after a while, so you need to generate new PATs every so often.
base_url: This parameter tells the OpenAI client to send all requests to "https://models.inference.ai.azure.com" instead of the OpenAI.com API servers. That's the domain that hosts the OpenAI-compatible endpoint for GitHub Models, so you'll always pass that domain as the base URL.
If we're working with the new openai-agents SDK, we use very similar code, but we must use the AsyncOpenAI client from openai instead. Lately, Python AI packages are defaulting to async, because it's so much better for performance.
Now let's look at all of the packages that make it really easy for us, by allowing us to directly bring in an instance of either OpenAI or AsyncOpenAI.
For PydanticAI, we configure an AsyncOpenAI client, then construct an OpenAIModel object from PydanticAI, and pass that model to the agent:
For Semantic Kernel, the code is very similar. We configure an AsyncOpenAI client, then construct an OpenAIChatCompletion object from Semantic Kernel, and add that object to the kernel.
I saved Llamaindex for last, as it is the most different. The Llamaindex Python package has a different constructor for OpenAI.com versus OpenAI-like servers, so I opted to use that OpenAILike constructor instead. However, I also needed an embeddings model for my example, and the package doesn't have an OpenAIEmbeddingsLike constructor, so I used the standard OpenAIEmbedding constructor.
In all of the examples above, I specified the "gpt-4o" model. The "gpt-4o" model is a great choice for agents because it supports function calling, and many agent frameworks only work (or work best) with models that natively support function calling.
Fortunately, GitHub Models includes multiple models that support function calling, at least in my basic experiments:
gpt-4o
gpt-4o-mini
o3-mini
AI21-Jamba-1.5-Large
AI21-Jamba-1.5-Mini
Codestral-2501
Cohere-command-r
Ministral-3B
Mistral-Large-2411
Mistral-Nemo
Mistral-small
You might find that some models work better than others, especially if you're using agents with multiple tools. With GitHub Models, it's very easy to experiment and see for yourself, by simply changing the model name and re-running the code.
So, have you started prototyping AI agents with GitHub Models yet?! Go on, experiment, it's fun!
This year, we're seeing the rise in "reasoning models", models that include an additional thinking process in order to generate their answer. Reasoning models can produce more accurate answers and can answer more complex questions. Some of those models, like o1 and o3, do the reasoning behind the scenes and only report how many tokens it took them (quite a few!).
The DeepSeek-R1 model is interesting because it reveals its reasoning process along the way. When we can see the "thoughts" of a model, we can see how we might approach the question ourself in the future, and we can also get a better idea for how to get better answers from that model. We learn both how to think with the model, and how to think without it.
So, if we want to build an app using a transparent reasoning model like DeepSeek-R1, we ideally want our app to have special handling for the thoughts, to make it clear to the user the difference between the reasoning and the answer itself. It's also very important for a user-facing app to stream the response, since otherwise a user will have to wait a very long time for both the reasoning and answer to come down the wire.
Here's an app with streamed, collapsible thoughts:
We first deploy a DeepSeek-R1 model on Azure, using Bicep files (infrastructure-as-code) that provision a new Azure AI Services resource with the DeepSeek-R1 deployment. This deployment is what's called a "serverless model", so we only pay for what we use (as opposed to dedicated endpoints, where the pay is by hour).
We give both our local developer account and our application backend role-based access to use the deployment, by assigning the "Cognitive Services User" role. That allows us to connect using keyless authentication, a much more secure approach than API keys.
Connecting to DeepSeek-R1 on Azure from Python
We have a few different options for making API requests to a DeepSeek-R1 serverless deployment on Azure:
OpenAI Python API library, which is focused on supporting OpenAI models but can also be used with any models that are compatible with the OpenAI HTTP API, which includes Azure AI models like DeepSeek-R1
Any of your favorite Python LLM packages that have support for OpenAI-compatible APIs, like Langchain, Litellm, etc.
I am using the openai package for this sample, since that's the most familiar amongst Python developers. As you'll see, it does require a bit of customization to point that package at an Azure AI inference endpoint. We need to change:
Base URL: Instead of pointing to openai.com server, we'll point to the deployed serverless endpoint which looks like "https://<resource-name>.services.ai.azure.com/models"
API version: The Azure AI Inference APIs require an API version string, which allows for versioning of API responses. You can see that API version in the API reference. In the REST API, it is passed as a query parameter, so we will need the openai package to send it along as a query parameter as well.
API authentication: Instead of providing an OpenAI key (or Azure AI services key, in this case), we're going to pass an OAuth2 token in the authorization headers of each request, and make sure that the token is refreshed before it expires.
Setting up the keyless API authentication can be a bit tricky! First, we need to acquire a token provider for our current credential, using the azure-identity package:
That code uses either ManagedIdentityCredential when it's running in production (on Azure Container Apps, with a user-assigned identity) or AzureDeveloperCliCredential when it's running locally. The token_provider function returns a token string every time we call it
For the next step, it helps to understand a bit about how the OpenAI package works. The OpenAI package sends all HTTP requests through httpx, a popular Python package that can make calls either synchronously or asynchronously, and it allows for customization of the httpx clients by developers that need more control of the HTTP requests.
In our case, we need to add the token in the "Authorization" header of each HTTP request, so we make a subclass of httpx.Auth that sets the header on each asynchronous request by calling the token provider function:
class TokenBasedAuth(httpx.Auth):
async def async_auth_flow(self, request):
token = await openai_token_provider()
request.headers["Authorization"] = f"Bearer {token}"
yield request
def sync_auth_flow(self, request):
raise RuntimeError("Cannot use a sync authentication class with httpx.AsyncClient")
Each time the token provider function is called, it will make sure that the token has not yet expired, and fetch a new one as necessary.
Now we can create a AsyncOpenAI client by passing in a custom httpx client using that TokenBasedAuth class, along with the correct base URL and API version:
You'll notice that instead of the typical model name that we send in when using OpenAI, we send in the deployment name. For convenience, I often name deployments the same as the model, so that they will match even if I mistakenly pass in the model name.
Streaming the response from the backend
As I've discussed previously on this blog, we should always use streaming responses when building user-facing chat applications, to reduce perceive latency and improve the user experience.
To receive a streamed response from the chat completions API, we specified stream=True in the call above. Then, as we receive each event from the server, we check whether the content is the special "<think>" start token or "</think>" end token. When we know the model is currently in a thinking mode, we pass down the content chunks in a "reasoning_content" field. Otherwise, we pass down the content chunks in the "content" field.
We send each event to our frontend using a common approach of JSON-lines over a streaming HTTP response (which has the "Transfer-encoding: chunked" header). That means the client receives a JSON separated by a new line for each event, and can easily parse them out. The other common approaches are server-sent events or websockets, but both are unnecessarily complex for this scenario.
To process the streaming JSON lines that are returned from the server, I brought in my tiny ndjson-readablestream package, which uses ReadableStream along with JSON.parse to make it easy to iterate over each JSON object as it comes in. When I see that the JSON is "reasoning_content", I display it in a special collapsible container.
let answer = "";
let thoughts = "";
for await (const event of readNDJSONStream(response.body)) {
if (!event.delta) {
continue;
}
if (event.delta.reasoning_content) {
thoughts += event.delta.reasoning_content;
if (thoughts.trim().length > 0) {
// Only show thoughts if they are more than just whitespace
messageDiv.querySelector(".loading-bar").style.display = "none";
messageDiv.querySelector(".thoughts").style.display = "block";
messageDiv.querySelector(".thoughts-content").innerHTML = converter.makeHtml(thoughts);
}
} else {
messageDiv.querySelector(".loading-bar").style.display = "none";
answer += event.delta.content;
messageDiv.querySelector(".answer-content").innerHTML = converter.makeHtml(answer);
}
messageDiv.scrollIntoView();
if (event.error) {
messageDiv.innerHTML = "Error: " + event.error;
}
}
The azure-search-openai-demo repository was first created in March 2023 and is now the most popular RAG sample solution for Azure. Since the world of generative AI changes so rapidly, we've made many upgrades to its underlying packages and technologies over the past two years. But we've never changed the default GPT model used for the RAG flow: gpt-35-turbo.
Why, when there are new models that are cheaper and reportedly better, such as gpt-4o-mini? Well, changing the model is one of the most significant changes you can make to impact RAG answer quality, and I did not want to make the change without thorough evaluation.
Good news! I have now run several bulk evaluations on different RAG knowledge bases, and I feel fairly confident that a switch to gpt-4o-mini is a positive overall change, with some caveats. In my evaluations, gpt-4o-mini generates answers with comparable groundedness and relevance. The time-per-token is slightly less, but the answers are 50% longer on average, thus they take 45% more time for generation. The additional answer length often provides additional details based off the context, especially for questions where the answer is a list or a sequential process. The gpt-4o-mini per-token pricing is about 1/3 of gpt-35-turbo pricing, which works out to a lower overall cost.
Let's dig into the results more in this post.
Evaluation results
I ran bulk evaluations on two knowledge bases, starting with the sample data that we include in the repository, a bunch of invented HR documents for a fictitious company. Then, since I always like to evaluate knowledge that I know deeply, I also ran evaluations on a search index composed entirely of my own blog posts from this very blog.
Here are the results for the HR documents, for 50 Q/A pairs:
metric
stat
gpt-35-turbo
gpt-4o-mini
gpt_groundedness
pass_rate
0.98
0.98
mean_rating
4.94
4.9
gpt_relevance
pass_rate
0.98
0.96
mean_rating
4.42
4.54
answer_length
mean
667.7
934.36
latency
mean
2.96
3.8
citations_matched
rate
0.45
0.53
any_citation
rate
1.0
1.0
For that evaluation, groundedness was essentially the same (and was already very high), relevance only increased in its average rating (but not pass rate, which is the percentage of 4/5 scores), but we do see an increase in the number of citations in the answer that match the citations from the ground truth. That metric is actually my favorite, since it's the only one that compares the app's new answer to the ground truth answer.
Here are the results for my blog, for 200 Q/A pairs:
metric
stat
gpt-35-turbo
gpt-4o-mini
gpt_groundedness
pass_rate
0.97
0.95
mean_rating
4.89
4.8
gpt_relevance
pass_rate
0.89
0.94
mean_rating
4.04
4.25
answer_length
mean
402.24
663.34
latency
mean
2.74
3.27
citations_matched
rate
0.8
0.8
any_citation
rate
1.0
0.96
For this evaluation, we actually see a slight decrease in groundedness, an increase in relevance (both the average rating and pass rate), and the same percentage of citations matched from the ground truth.
I was concerned to see the decrease in groundedness, so I reviewed all the gpt-4o-mini answers with low groundedness. Almost all of them were variations of "I don't know." The model didn't feel comfortable that it had the right information to answer the question, so it decided not to answer. As I've discussed here in a previous blog post, that's a good thing! We want our models to be able to admit a lack of confidence - that's much better than an overconfident model spreading misinformation. So even though the gpt-35-turbo answers weren't wrong, I'm okay with gpt-4o-mini opting out, since it means it will be more likely to opt out for other questions where it definitely lacks the necessary information.
Why are the answers wordier?
You can also see an increase in answer length and latency in both the evaluations, so it's clear that gpt-4o-mini has a tendency towards longer answers across the domains.
We don't want our RAG applications to start producing wordier answers without good reason. A wordier answer requires more tokens to generate, increasing our costs, and it takes longer to finish generation. Fortunately, our app has a streaming interface, so users can start reading the response as soon as the first token is available, but users still may not want to wait for unnecessarily verbose answers.
I went through the generated answers for my blog for both models to get a feel for how the extra tokens are being used. The gpt-4o-mini answers tend to be more comprehensive, including details that the older model left out, which is probably why they earned higher relevance scores. In addition, the gpt-4o-mini answers tend to use more Markdown formatting for lists and bolding, which makes the longer answers surprisingly easier to read than the shorter less-formatted answers.
I'll share a few examples here so you can see what I mean:
Question #1: What are the options for a Web Developer to combine coding and teaching?
gpt-35-turbo
gpt-4o-mini
Some options for a Web Developer to combine coding and teaching include:
Developer relations: Working with API companies to help teach their APIs [girldevelopit-sf-thanks-for-great-2012.html].
Engineering in EdTech: Creating online educational experiences in the field of educational technology [combining-coding-and-teaching-into.html].
Teaching online: Creating content on platforms like Youtube, Udemy, or Lynda, or applying to be a creator at companies like TreeHouse, CodeSchool, and Codecademy [combining-coding-and-teaching-into.html].
A Web Developer can combine coding and teaching in several ways:
Developer Relations: Work with API companies to help teach their APIs [combining-coding-and-teaching-into.html].
Engineering in EdTech: Consider creating online educational experiences by exploring job opportunities in EdTech companies [combining-coding-and-teaching-into.html].
Teaching Online: Create content on platforms like YouTube, Udemy, or Lynda, or apply to be a creator at companies like TreeHouse, CodeSchool, and Codecademy [combining-coding-and-teaching-into.html].
Starting a CS Education Company: For example, co-founding a company like CodeHS [combining-coding-and-teaching-into.html].
K-12 and College Teaching: Look for teaching positions in K-12 schools or colleges, as there is an increasing demand for computer science teachers [combining-coding-and-teaching-into.html].
Running Internal Training Classes: Teach internal training classes at large companies to help engineers with technology [combining-coding-and-teaching-into.html].
Teaching on the Side: Join clubs and organizations dedicated to teaching coding, such as GirlDevelopIt, CoderDojo, or GirlsWhoCode, or participate in programs like TEALS to teach AP CS classes [combining-coding-and-teaching-into.html].
Question #2: What does .editorconfig do?
gpt-35-turbo
gpt-4o-mini
The .editorconfig file is used to set coding style preferences for a project, such as indent style and size. It helps maintain consistent coding styles across different editors and IDEs. [porting-project-from-spaces-to-tabs.html]
An .editorconfig file is used to define and maintain consistent coding styles for multiple developers working on the same project. It allows you to specify settings such as the indentation style (tabs or spaces) and the size of the indentation (e.g., 4 spaces for tabs). This helps ensure that everyone adheres to the same formatting rules without needing to adjust their individual editor settings manually. In the context of GitHub, it can also influence how tabs are rendered in projects [porting-project-from-spaces-to-tabs.html].
Question #3: how does the az containerapp up command help in deploying a container app, and what does it do?
gpt-35-turbo
gpt-4o-mini
The az containerapp up command helps in deploying a container app by creating an Azure resource group, building the container image using the local Dockerfile, creating a registry in the Azure Container Registry and pushing the image to the registry, and finally creating a Container App that uses the pushed image and allows external ingress on a specified port [deploying-containerized-fastapi-app-to.html].
The az containerapp up command simplifies the deployment of a container app by automating several steps. When executed, it performs the following actions:
Creates an Azure resource group (e.g., "flask-aca-rg" or "fastapi-aca-rg") to organize resources.
Creates a Container App Environment and a Log Analytics workspace within that resource group.
Builds the container image using the local Dockerfile.
Creates a registry in the Azure Container Registry (or uses an existing one if specified) and pushes the image to that registry.
Creates a Container App (e.g., "flask-aca-app" or "fastapi-aca-app") that utilizes the pushed image and allows external ingress on a specified port (e.g., port 5000 for Flask or port 80 for FastAPI). Upon successful completion, the command provides a public URL to access the deployed container app [deploying-containerized-flask-app-to.html][deploying-containerized-fastapi-app-to.html].
Those answers are fairly representative of the differences. For short, simple questions, gpt-4o-mini may sometimes answer with slightly more details. For any questions where the answer is a list or a sequence, gpt-4o-mini is more likely to write a longer list with bolded list items for better readability.
Next steps
I will send a PR to azure-search-openai-demo to default the model to gpt-4o-mini, and once merged, I'll note in the release notes that developers may see longer response lengths with the new model. As always, developers can always override the default model, as many have been doing to use gpt-4, gpt-4o-mini, and gpt-4o, over the past year.
If you have any learnings based on your own evaluations of the various GPT models on RAG answer quality, please share them with me! I would love to see more evaluation results shared so that we can learn together about the differences between models.
When we build apps on top of Large Language Models, we need to evaluate the app responses for quality and safety.
When we evaluate the quality of an app, we're making sure that it provides answers that are coherent, clear, aligned to the user's needs, and in the case of many applications: factually accurate. I've written here about quality evaluations, plus gave a recent live stream on evaluating RAG answer quality.
When we evaluate the safety of an app, we're ensuring that it only provides answers that we're comfortable with our users receiving, and that a user cannot trick the app into providing unsafe answers. For example, we don't want answers to contain hateful sentiment towards groups of people or to include instructions about engaging in destructive behavior. See more examples of safety risks in this list from Azure AI Foundry documentation.
Thanks to the Azure AI Evaluation SDK, I have now added a safety evaluation flow to two open-source RAG solutions, RAG on Azure AI Search, and RAG on PostgreSQL, using very similar code. I'll step through the process in this blog post, to make it easier for all you to add safety evaluations to your own apps!
The overall steps for safety evaluation:
Provision an Azure AI Project
Configure the Azure AI Evaluation SDK
Simulate app responses with AdversarialSimulator
Evaluate the responses with ContentSafetyEvaluator
Provision an Azure AI Project
We must have an Azure AI Project in in order to use the safety-related functionality from the Azure AI Evaluation SDK, and that project must be in one of the regions that support the safety backed service.
Since a Project must be associated with an Azure AI Hub, you either need to create both a Project and Hub, or reuse existing ones. You can then use that project for other purposes, like model fine-tuning or the Azure AI Agents service.
You can create a Project from the Azure AI Foundry portal, or if you prefer to use infrastructure-as-code, you can use these Bicep files to configure the project. You don't need to deploy any models in that project, as the project's safety backend service uses its own safety-specific GPT deployment.
First we must either add the azure-ai-evaluation Python package to our requirements file, or install it directly into the environment:
pip install azure-ai-evaluation
Then we create a dict in our Python file with all the necessary details about the Azure AI project - the subscription ID, resource group, and project name. As a best practice, I store those values environment variables:
Next, we use the AdversarialSimulator class to simulate users interacting with the app in the ways most likely to produce unsafe responses.
We initialize the class with the project configuration and a valid credential. For my code, I used keyless authentication with the AzureDeveloperCliCredential class, but you could use other credentials as well, including AzureKeyCredential.
The SDK supports multiple scenarios. Since my code is evaluating a RAG question-asking app, I'm using AdversarialScenario.ADVERSARIAL_QA. My evaluation code would also benefit from simulating with AdversarialScenario.ADVERSARIAL_CONVERSATION since both RAG apps support multi-turn conversations. Use the scenario that matches your app.
For the AdversarialScenario.ADVERSARIAL_QA scenario, the simulated questions are based off of templates with placeholders, and the placeholders filled with randomized values, so hundreds of questions can be generated (up to the documented limits). Those templates are available in multiple languages, so you should specify a language code if you're evaluating a non-English app.
We use the max_simulation_results parameter to generate 200 simulations. I recommend starting with much less than that when you're testing out the system, and then discussing with your data science team or safety team how many simulations they require before deeming an app safe for production. If you don't have a team like that, then one approach is to run it for increasing numbers of simulations and track the resulting metrics as simulation size increases. If the metrics keep changing, then you likely need to go with the higher number of simulations until they stop changing.
The target parameter expects a local Python function that matches the documented signature: it must accept a particular set of arguments, and respond with messages in a particular format.
Whenever I run the safety evaluations, I send the simulated questions to the local development server, to avoid the latency and security issues of sending requests to a deployed endpoint. Here's what that looks like as a callback function:
While the simulator is running, you'll see the progress status in the terminal. This can take a significant amount of time (5 seconds per simulation, in my case), since it needs to generate the question and send it to your app for answering.
Once the simulations are done running, they're available in the returned list. If you want, you can easily save them to a local JSONL file, following the example from the documentation:
output_file = "grounding_simulation_output.jsonl"
with open(output_file, "w") as file:
for output in outputs:
file.write(output.to_eval_qr_json_lines())
I experimented with saving the outputs locally, but when I saw how disturbing some of the questions were, I opted to only keep the outputs in memory, so that fellow developers wouldn't accidentally wander into the questions file when working on the app. 😱
Evaluate the responses with ContentSafetyEvaluator
Now that we have stored responses from our app, we can use the evaluators from the SDK to rate the safety levels of the responses. Ideally, our app's response is some sort of refusal, like "I can't answer that question", which would be considered a very safe response. In my case, since these RAG apps use Azure OpenAI models that have a built-in Content Safety filter for both the prompt and response, a common response was "That question violated the Content Safety Filter", which is also a very safe response, and proves that the filter is working well.
The Azure AI Evaluation SDK includes multiple evaluators for safety violations: ViolenceEvaluator, SexualEvaluator, SelfHarmEvaluator, HateUnfairnessEvaluator, and a composite evaluator that covers all four: ContentSafetyEvaluator. For a more performant evaluation, I went for the composite evaluator.
I run the ContentSafety evaluator on each app response, and track the overall score statistics in a dictionary:
from azure.ai.evaluation import ContentSafetyEvaluator
evaluators = ["hate_unfairness", "sexual", "violence", "self_harm"]
summary = {evaluator: {"low_count": 0, "score_total": 0}
for evaluator in evaluators}
for output in outputs:
query = output["messages"][0]["content"]
answer = output["messages"][1]["content"]
safety_eval = ContentSafetyEvaluator(
credential=credential, azure_ai_project=azure_ai_project)
eval_score = safety_eval(query=query, response=answer)
for evaluator in evaluators:
if eval_score[evaluator] == "Very low" or severity_level == "Low":
summary[evaluator]["low_count"] += 1
summary[evaluator]["score_total"] += eval_score[f"{evaluator}_score"]
Each evaluator response includes three components: a numeric score (1-7), a corresponding word score ("Very low", "Low", etc), and a reason.
I keep track of the average score across all the simulations, as well as the fraction of times that a response earns either "Very low" or "Low". Once I've computed the overall statistics, I store the results in a file that looks like this:
As you can see, every evaluator had a 100% low rate, meaning every response earned either a "Very low" or "Low". The average score is slightly above zero, but that just means that some responses got "Low" instead of "Very low", so that does not concerned me. This is a great result to see, and gives me confidence that my app is outputting safe responses, especially in adversarial situations.
When should you run safety evaluations?
Running a full safety evaluation takes a good amount of time (~45 minutes for 200 questions) and uses cloud resources, so you don't want to be running evaluations on every little change to your application. However, you should definitely consider running it for prompt changes, model version changes, and model family changes.
For example, I ran the same evaluation for the RAG-on-PostgreSQL solution to compare two model choices: OpenAI gpt-4o (hosted on Azure) and Lllama3.1:8b (running locally in Ollama). The results:
Evaluator
gpt-4o-mini - % Low or Very low
llama3.1:8b - % Low or Very low
Hate/Unfairness
100%
97.5%
Sexual
100%
100%
Violence
100%
99%
Self-Harm
100%
100%
When we see that our app has failed to provide a safe answer for some questions, it helps to look at the actual response. For all the responses that failed in that run, the app answered by claiming it didn't know how to answer the question but still continue to recommend matching products (from its retrieval stage). That's problematic since it can be seen as the app condoning hateful sentiments or violent behavior. Now I know that to safely use that model with users, I would need to do additional prompt engineering or bring in an external safety service, like Azure AI Content Safety.
More resources
If you want to implement a safety evaluation flow in your own app, check out:
Everyone's going ga-ga for DeepSeek-R1, so I thought I'd try it out in a live stream today:
I'll summarize my experience in this post.
I tried Python through two different hosts, via the OpenAI Python SDK
GitHub Models: Open to anyone with a GitHub account, free up to a certain number of requests per day. Great for learning and experimenting with new models.
Ollama: Includes 1.5B all the way to 671B models, but my Mac M1 can only run the 8B.
It's also possible to deploy DeepSeek-R1 on Azure, but I used the hosts that were easy to setup quickly.
Connecting with the OpenAI SDK
The DeepSeek-R1 model provides an "OpenAI-compatible interface", so that you can use the OpenAI python SDK for making chat completion requests. The DeepSeek-R1 model is fairly limited in its compatibility - no temperature, no function calling, less attention paid to the "system" message - but it's still very usable.
Then I make the chat completion request, leaving off most parameters and system message.
It is possible to specify max_tokens, but the model might end its response in the middle of a thought,
so we need to be very careful when setting that parameter. It also supports the stop parameter.
response = client.chat.completions.create(
model=model_name,
messages=[
{
"role": "user",
"content": "You're an assistant that loves emojis. Write a haiku about a hungry cat who wants tuna"
},
],
)
Now you'll get a response like this:
<think>
The model's thought process, which can be VERY long.
</think>
The model's final answer.
You can choose to extract the thoughts using a regular expression for those tags, as shown in this article, and then render it differently to the user.
The thinking can take a very long time however, so my preference is to stream the response. That way I can start reading its thoughts as soon as they begin.
Handling streamed thoughts
To receive a streamed response, we first add stream=True to the chat completion call:
Then, in our stream processing code, we keep track of whether we've seen the start think tag or the end think tag, and display the thoughts differently to the user:
is_thinking = False
for event in completion:
if event.choices:
content = event.choices[0].delta.content
if content == "<think>":
is_thinking = True
print("🧠Thinking...", end="", flush=True)
elif content == "</think>":
is_thinking = False
print("🛑\n\n")
elif content:
print(content, end="", flush=True)
Then our output looks like this:
🧠Thinking...
The model's thought process, which can be VERY long.
🛑
The model's final answer.
We could use a similar approach when streaming down thoughts from the backend to the frontend, so that the frontend could visually distinguish between the thoughts and the answer itself.
Tip: There are some questions that are so easy for it to answer that the "thoughts" will simply be a new line- for example, if I simply say "hi" to the model. We may want to consider that edge case in how we render thoughts. The vast majority of questions will have thoughts, however - even a seemingly simple question like "who painted the Mona Lisa?" had a long thinking process to determine that, yes, it was definitely Leonardo DaVinci.
Using DeepSeek-R1 with RAG
Since I spend most of my time these days on applications that use RAG (Retrieval-Augmented Generation), I wanted to see how it would handle answering questions based on provided context.
Document chunks from PDFs, from the search index created by this Azure RAG solution, with ~500 tokens in each chunk, and three chunks retrieved per question.
I started off trying RAG with Ollama and the 8B model, and I was surprised at the results: the model completely hallucinated the sources, seemingly ignoring what I passed to it, in a very confident way. For example, here's the model describing a line of the CSV and its format:
"F-Typesport; engine: 3.0L turbocharged I6; horsepower: 505.0; torque: 295 lb-ft; acceleration: 2.8s; top speed: 155 mph." So, each car's description continues with multiple attributes separated by semicolons and some other punctuation.
However, there is no "F-Typesport" in the CSV, no mention of torque or horsepower. It made that all up!
So I moved back to GitHub Models, where I could RAG with the full model with its 671 billion parameters. I got very good results with the full model, along with a very long thought process. I asked it "what's the fastest hybrid car?", and it went line-by-line through the CSV, mentioning the acceleration of each car. Here's how it started:
Looking at the CSV headers: vehicle, year, msrp, acceleration, mpg, class. The acceleration column is the one to focus on. The task is to scan through all entries and find the minimum acceleration value.
Let me go through the data row by row and note the acceleration values. The first entry is the Prius (1st Gen) with 7.46 seconds. Then the Tino at 8.2, Prius (2nd Gen) at 7.97. The Insight from 2000 is 9.52, Civic (1st Gen) 2001 at 7.04. Wait, that's a low number. Continuing: Alphard 2003 is 8.33, Escape 2004 at 10.32, Lexus RX400h 2005 at 12.76. Then Highlander 2005 (12.76), Mercury Mariner 2006 (8.98), Camry 2006 (11.28), Lexus GS450h 2006 (18.65). That's pretty high. Estima 2006 (9.26), Altima 2006 (13.29).
At the end, it did some "double-checking" (like to make sure the fastest car was indeed a hybrid car), and then correctly reported the fastest car.
I tried the same query with gpt-4o and o1, and they both also correctly reported the fastest car. The difference, of course, is that DeepSeek-R1 shows its thought process. That can be helpful in some situations, and was actually helpful for me, as I'm not a car person and didn't know what column/value would properly answer that question.
I then tried RAG on the document chunks, and it came up with a correct answer, as well as providing the citations in the correct format (square brackets, page numbers, as requested in my prompt). Once again, it had a very lengthy thought process, where it looked at each source in turn, described what the source contributed to the answer. For example, here's how it started:
Okay, the user is asking what's included in the Northwind Health Plus plan that isn't in the standard plan. Let me go through the provided sources carefully.
From Benefit_Options.pdf#page=3, it mentions that Northwind Health Plus covers emergency services, mental health and substance abuse, and out-of-network services, which the Standard plan doesn't. Also, Health Plus has more comprehensive prescription drug coverage and emergency services both in and out of network. Another source, Northwind_Standard_Benefits_Details.pdf#page=45, confirms that Standard covers preventive care, professional visits, and some diagnostics but doesn't list those additional services. The same Benefit_Options.pdf#page=3 also compares costs, but the key point here is the coverage differences. So, putting it together, the Plus plan includes those additional services and broader drug coverage that the Standard lacks.
Generally, it seems like the thinking process scales linearly with the size of the data: i.e. for the CSV, it scales with the number of ros, and for the document chunks, it scales with the number of chunks. I have not scientifically verified this, of course, so it's possible that the model would take a different approach with vastly different amounts of context, but this is what it looks like for the kind of data I sent to it. If anyone does verify that with some more rigor, let me know!
The thought process also looked at each line in the instructions portion of my prompt that described how to cite the sources, so we can expect longer thought processes for each additional instruction requested of the model. For example, this is the model trying to adhere to one of the lines:
But the user's instruction says: "Don't combine sources, list each source separately, for example [info1.txt][info2.pdf]." However, if all benefits are from the same source, it's allowed to list the source once per claim. Wait, no, if multiple facts are from the same source, each fact should be followed by the source. For example, "[Benefit_Options.pdf#page=3]" after each item.
That would make me think very carefully about each line in the prompt, knowing how much the model is actually paying attention to them. It also seems like a good way to iterate on prompts to find the clearest wording for the desired behavior.
One of the amazing benefits of modern machine learning is that computers can reliably turn text into speech, or transcribe speech into text, across multiple languages and accents. We can then use those capabilities to make our web apps more accessible for anyone who has a situational, temporary, or chronic issue that makes typing difficult. That describes so many people - for example, a parent holding a squirmy toddler in their hands, an athlete with a broken arm, or an individual with Parkinson's disease.
There are two approaches we can use to add speech capabilites to our apps:
Which one to use? The great thing about the browser APIs is that they're free and available in most modern browsers and operating systems. The drawback of the APIs is that they're often not as powerful and flexible as cloud-based services, and the speech output often sounds much more robotic. There are also a few niche browser/OS combos where the built-in APIs don't work, like SpeechRecognition on Microsoft Edge on a Mac M1. That's why we decided to add both options to azure-search-openai-demo, to give developers the option to decide for themselves.
In this post, I'm going to show you how to add speech capabilities using the free built-in browser APIs, since free APIs are often easier to get started with, and it's important to do what we can to improve the accessibility of our apps. The GIF below shows the end result, a chat app with both speech input and output buttons:
All of the code described in this post is part of openai-chat-vision-quickstart, so you can grab the full code yourself after seeing how it works.
Speech input with SpeechRecognition API
To make it easier to add a speech input button to any app, I'm wrapping the functionality inside a custom HTML element, SpeechInputButton.
First I construct the speech input button element with an instance of the SpeechRecognition API, making sure to use the browser's preferred language if any are set:
Then I define the connectedCallback() method that will be called whenever this custom element has been added to the DOM. When that happens, I define the inner HTML to render a button and attach event listeners for both mouse and keyboard events. Since we want this to be fully accessible, keyboard support is important.
The majority of the code is in the startRecording function. It sets up a listener for the "result" event from the SpeechRecognition instance, which contains the transcribed text. It also sets up a listener for the "end" event, which is triggered either automatically after a few seconds of silence (in some browsers) or when the user ends the recording by clicking the button. Finally, it sets up a listener for any "error" events. Once all listeners are ready, it calls start() on the SpeechRecognition instance and styles the button to be in an active state.
startRecording() {
if (this.speechRecognition == null) {
this.dispatchEvent(
new CustomEvent("speech-input-error", {
detail: { error: "SpeechRecognition not supported" },
})
);
}
this.speechRecognition.onresult = (event) => {
let input = "";
for (const result of event.results) {
input += result[0].transcript;
}
this.dispatchEvent(
new CustomEvent("speech-input-result", {
detail: { transcript: input },
})
);
};
this.speechRecognition.onend = () => {
this.isRecording = false;
this.renderButtonOff();
this.dispatchEvent(new Event("speech-input-end"));
};
this.speechRecognition.onerror = (event) => {
if (this.speechRecognition) {
this.speechRecognition.stop();
if (event.error == "no-speech") {
this.dispatchEvent(
new CustomEvent("speech-input-error", {
detail: {error: "No speech was detected. Please check your system audio settings and try again."},
}));
} else if (event.error == "language-not-supported") {
this.dispatchEvent(
new CustomEvent("speech-input-error", {
detail: {error: "The selected language is not supported. Please try a different language.",
}}));
} else if (event.error != "aborted") {
this.dispatchEvent(
new CustomEvent("speech-input-error", {
detail: {error: "An error occurred while recording. Please try again: " + event.error},
}));
}
}
};
this.speechRecognition.start();
this.isRecording = true;
this.renderButtonOn();
}
If the user stops the recording using the keyboard shortcut or button click, we call stop() on the SpeechRecognition instance. At that point, anything the user had said will be transcribed and become available via the "result" event.
stopRecording() {
if (this.speechRecognition) {
this.speechRecognition.stop();
}
}
Alternatively, if the user presses the Escape keyboard shortcut, we instead call abort() on the SpeechRecognition instance, which stops the recording and does not send any previously untranscribed speech over.
abortRecording() {
if (this.speechRecognition) {
this.speechRecognition.abort();
}
}
Once the custom HTML element is fully defined, we register it with the desired tag name, speech-input-button:
You can see the full custom HTML element code in speech-input.js and the usage in index.html. There's also a fun pulsing animation for the button's active state in styles.css.
Speech output with SpeechSynthesis API
Once again, to make it easier to add a speech output button to any app, I'm wrapping the functionality inside a custom HTML element, SpeechOutputButton. When defining the custom element, we specify an observed attribute named "text", to store whatever text should be turned into speech when the button is clicked.
class SpeechOutputButton extends HTMLElement {
static observedAttributes = ["text"];
In the constructor, we check to make sure the SpeechSynthesis API is supported, and remember the browser's preferred language for later use.
The majority of the code is in the toggleSpeechOutput function. If the speech is not yet playing, it creates a new SpeechSynthesisUtterance instance, passes it the "text" attribute, and sets the language and audio properties. It attempts to use a voice that's optimal for the desired language, but falls back to "en-US" if none is found. It attaches event listeners for the start and end events, which will change the button's style to look either active or unactive. Finally, it tells the SpeechSynthesis API to speak the utterance.
toggleSpeechOutput() {
if (!this.isConnected) {
return;
}
const text = this.getAttribute("text");
if (this.synth != null) {
if (this.isPlaying || text === "") {
this.stopSpeech();
return;
}
// Create a new utterance and play it.
const utterance = new SpeechSynthesisUtterance(text);
utterance.lang = this.lngCode;
utterance.volume = 1;
utterance.rate = 1;
utterance.pitch = 1;
let voice = this.synth
.getVoices()
.filter((voice) => voice.lang === this.lngCode)[0];
if (!voice) {
voice = this.synth
.getVoices()
.filter((voice) => voice.lang === "en-US")[0];
}
utterance.voice = voice;
if (!utterance) {
return;
}
utterance.onstart = () => {
this.isPlaying = true;
this.renderButtonOn();
};
utterance.onend = () => {
this.isPlaying = false;
this.renderButtonOff();
};
this.synth.speak(utterance);
}
}
When the user no longer wants to hear the speech output, indicated either via another press of the button or by pressing the Escape key, we call cancel() from the SpeechSynthesis API.
To use this custom speech-output-button element in a chat application, we construct it dynamically each time that we've received a full response from an LLM, and call setAttribute to pass in the text to be spoken:
You can see the full custom HTML element code in speech-output.js and the usage in index.html. This button also uses the same pulsing animation for the active state, defined in styles.css.
Acknowledgments
I want to give a huge shout-out to John Aziz for his amazing work adding speech input and output to the azure-search-openai-demo, as that was the basis for the code I shared in this blog post.
