Generate Machine-Readable Output from OpenAIs ChatGPT in .NET – Christians Digital Grove of .NET and AI

OpenAI ChatGPT can be a customized API for your application by generating machine-readable output for your prompt. The model behind can be either the current GPT-4 model or a fine-tuned model with your own business data provided via an extended context or functions. This article shows you how to generate machine-readable JSON from OpenAI ChatGPT using OkGoDoIt OpenAI-API using C#/.NET.

Generating machine-readable output can be a challenge as ChatGPTs prompt closure is non-deterministic. I write this as fact, even it’s counter-intuitive: where does the randomness come in as GPT is in the end only a probability graph? But more about this after the example and the C# code.

For making the prompt closure machine-readable you should use two different approaches:
A. Lower the temperature parameter for the prompt completion. The temperature is a noise factor in ChatGPTs API (a value between 0 and 2). By lowering the noise factor you decrease the “creativity” ChatGPT shows in its output.
B. Define the expected machine-readable format in the context of the prompt and also provide examples for the expected output and for exceptions, e. g. in form of JSON, YAML, or XML context.

Lets start with the example! We want to create a ChatGPT-based service, which can provide Italian cooking recipes in JSON format to a consuming service, e. g. an website about Italian cucina. The prompt completion should be least creative as possible, as it should be in a machine-readable format. Therefore when instantiating a new chat the temperature parameter needs to be set to zero explicitly.

using System.Text;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using OpenAI_API;
using OpenAI_API.Models;

OpenAIAPI openAiApi = new OpenAIAPI(new APIAuthentication("your-secret-key"));

var chat = openAiApi.Chat.CreateConversation();
chat.Model = Model.ChatGPTTurbo;
chat.RequestParameters.Temperature = 0;

Thereafter you can set-up the system context by telling ChatGPT it will have the role of an API for cooking recipes from Italy. In the same context you can define that ChatGPT shall generate its answers in JSON format based on a given example. It is important telling the system, that this example represents a fix structure and not additional functional nodes shall be added and also no functional nodes shall be removed. In the end of this example the user prompt is given in form of a simple question.

// system context set-up
chat.AppendSystemMessage("You are an API for cooking recipes from Italy.");
// provide example for JSON output and make it clear, that no tags should be added or removed
chat.AppendSystemMessage("You answer every question for a recipe in JSON format. Use the following example provided as JSON template. Do not add additional JSON tags or remove any JSON tags present in the example.");

// add JSON example to context
chat.AppendExampleChatbotOutput("{\"ingredients\":[{\"pasta\", \"salt\", \"tomato\"]}, \"actions\":[{\"Boil the water.\"}, {\"Add a spoon of salt.\"}, {\"Add the pasta.\"}]}");

// user prompt
chat.AppendUserInput("How can I make good Spaghetti Puttanesca?");

Before looking at the completion of the example, let’s have a look at the exception handling for this “creative” API. When ChatGPT is asked a question, which is different from the context defined above (e. g. “How can I built a robot?”), it will generate an answer which deviates from the predefined machine-readable format. This is technically correct, as the context defined above, did not specify how to handle different kind of questions.

// user prompt for different topic
chat.AppendUserInput("How can I built a robot?");

// answer from ChatGPT (but not as JSON)
// I'm sorry, but as an API for cooking recipes from Italy, I can only provide recipes for food and cooking-related information. I don't have the capability to provide instructions for building a robot. Is there anything else I can assist you with?

However this can be added to the context of the prompt, also with a JSON example how to format this kind of exception. But I prefer to keep the this example simple, and we will only do a light exception handling in the end of the code.

So back to our initial user question to ChatGPT in the user context (“How can I make good Spaghetti Puttanesca?”). After retrieving the prompt completion, it’s content is deserialized as JSON stream using Newtonsoft.Json library. If it’s not JSON, and exception will be raised and written to the error stream of the operating system.

// get completion
var completion = new StringBuilder();
await foreach (var completionResponse in chat.StreamResponseEnumerableFromChatbotAsync())
{
    completion.Append(completionResponse);
}

// deserialize it
try
{
    var json = JsonConvert.DeserializeObject<JObject>(completion.ToString());
    Console.WriteLine($"Completion (JSON): {json}");
}
catch (Exception ex)
{
    Console.Error.WriteLine(ex);
    throw;
}

Does the example work? Yes, below the output of the prompt closure is shown! But even with temperature set to zero, variations in the recipe can be observed, while the JSON format remains stable. Even with slight instabilities due to the non-deterministic character of ChatGPT, generating machine-readable outputs from ChatGPT with correct error-/retry-handling is valuable. The example shows you have an excellent technological base to create an Italian cucina website with dynamic content based on machine-readable and therefore transformable content for your website.

Completion (JSON): {
  "ingredients": [
    "spaghetti",
    "olive oil",
    "garlic",
    "anchovy fillets",
    "canned tomatoes",
    "black olives",
    "capers",
    "red pepper flakes",
    "parsley",
    "salt",
    "black pepper"
  ],
  "actions": [
    "Cook the spaghetti according to package instructions.",
    "In a large skillet, heat olive oil over medium heat.",
    "Add minced garlic and cook until fragrant.",
    "Add anchovy fillets and cook until they dissolve.",
    "Add canned tomatoes, black olives, capers, and red pepper flakes.",
    "Simmer the sauce for about 10 minutes, until it thickens slightly.",
    "Season with salt and black pepper to taste.",
    "Drain the cooked spaghetti and add it to the skillet.",
    "Toss the spaghetti in the sauce until well coated.",
    "Serve hot, garnished with chopped parsley."
  ]
}

Now a small hop away from C#/.NET to the question why is ChatGPT non-deterministic. When testing different LLMs, you will see other models are more stable. At the moment there is no official statement from Open AI regarding this topic, but two issues seem to hold:
A. The floating-point operations are CUDA optimised, and even little imprecisions can make the system fluctuate. You might know that floating-point numbers in numerical calculations can become unstable when above certain precision requirements (see IEEE 754 for more information on this topic).
B. From little literature available, it seems that ChatGPT uses are sparse MoE (Mixture of Experts) model behind, meaning that a natively dense model is split to various sub-models (experts), which are addressed by a routing layer. Experts run as batches and the results from the expert models is then recombined in to the final closure. Depending the used expert model and the splitting of tokens to the expert batches, runtime variances influence the final closure. More information about sparse MoE architectures can be found at https://arxiv.org/abs/2308.00951.

I hope this article helps you to integrate ChatGPT as machine-readable data source into your C#/.NET applications! Happy coding 🚀!