[HAI5014] Week 5: Smarter than Eliza

Disclaimer: This blog provides instructions and resources for the workshop part of my lectures. It is not a replacement for attending class; it may not include some critical steps and the foundational background of the techniques and methodologies used. The information may become outdated over time as I do not update the instructions after class.

Our first LLM based chatbot

Create a new repository in GitHub called HAI5014-MyFirstLLMChatbot

• You can make it public or private to your liking
• Select Add a README file
• Select Add .gitignore and select Python
• Create the repository

Add dependencies to requirements.txt

• Create a requirements.txt file in your repository
• Add the following lines to the file:

  azure-ai-inference
  ipykernel
  openai
  

• Commit the changes

Create a devcontainer configuration

• Create a .devcontainer/devcontainer.json file in your repository

• Add the following lines to the file:

  /*
  * Dev containter configuration file for class HAI- 5014
  * Sungkyunkwan University 2025 Spring Semester - by camphouse.me
  */
  
  {
      "name": "Python 3",
      // Use the official Python 3.11 image as a base
      "image": "mcr.microsoft.com/devcontainers/python:0-3.11-bullseye",
      "features": {
      },
  
      // Configure the Codespaces Machine to use 2 CPUs
      "hostRequirements": {
          "cpus": 2
      },
  
      // Use 'postCreateCommand' to run commands after the container is created.
      "postStartCommand": "export PATH=/usr/local/python3.11/bin:$PATH",
      "postCreateCommand": "pip3 install --user -r requirements.txt",
      
      // Install preferred extensions into VS Code
      "customizations": {
          "vscode": {
          "extensions": [
          "github.copilot",
          "ms-python.python",
          "ms-toolsai.datawrangler",
          "ms-toolsai.jupyter",
          "DavidAnson.vscode-markdownlint",
          "esbenp.prettier-vscode"
          ]
      }
      }
  }
  

• Commit the changes

Open a new Codespace

Because it will take some time to create the Codespace, we will do this in the background while we are setting up the Azure OpenAI API.

• Go to the main page of your repository in GitHub
• Click on the Code button and select the Codespaces tab
• Click on Create Codespace on main
• Wait for the Codespace to be created and opened

Get your Azure OpenAI API Endpoint and key

In a new tab, go to the Azure OpenAI Studio

• In the left menu, click on Deployments and find your GPT-4o deployment
• Click on the deployment name to open the deployment details
• Find the Endpoint and Key for your deployment

Save the Endpoint and Key as secrets in your GitHub repository

• Go to your repository in GitHub
• Click on Settings
• Click on Secrets and variables in the left menu
• Click on Codespaces
• Click on New repository secret
• Name the secret AZURE_AI_ENDPOINT and paste the endpoint into the value field. You can remove the last part of the endpoint after gpt-4o. (it should look like https://hai5014-fofhihturehiw.openai.azure.com/openai/deployments/gpt-4o)
• Click on Add secret

• Do the same for the key, but name it AZURE_AI_SECRET

  Environment variables in GitHub cannot start with GITHBUB_ or GITHUB_. and therefore we cannot use the same name as used in the code example. From here, make sure to use AZURE_AI_SECRET instead of GITHUB_TOKEN in your code.

Check if the enviroment variables are set correctly

• In your codespace, create a new file called test.py

• Add the following code to the file:

    import os
  
    endpoint = os.environ["AZURE_AI_ENDPOINT"]
    token = os.environ["AZURE_AI_SECRET"]
  
    print("Endpoint:", endpoint)
    print("Token:", token)
  

Building the chatbot

In a new tap, open the GitHub Models page. Let’s play around (did someone say raspberry?) with some of the models. After discovery, we will build our own chatbot using the OpenAI GPT-4o model:

1. Create a new file called chatbot.py in your repository
2. Add the example code from GitHub Models:
   1. Go to the OpenAI GPT-4o model in GitHub
   2. Click on the <> Use this model button
   3. Select Python as the language and Azure AI inference as SDK
      • Walk through the steps until you reach the code example at 3
   4. Copy the code from chapter 3. Run a basic code sample
   5. Paste the code into your chatbot.py file and save the file
3. Commit & push the changes

Change the code to use your own model

To use the model you created in Azure, we need to change the code a bit. We will use the gpt-4o model we created in the previous steps.

• Find the following lines in your chatbot.py file:

  endpoint = "https://models.inference.ai.azure.com"
  model_name = "gpt-4o"
  token = os.environ["GITHUB_TOKEN"]
  

• Change them to:

  endpoint = os.environ["AZURE_AI_ENDPOINT"]
  model_name = "gpt-4o"
  token = os.environ["AZURE_AI_SECRET"]
  

• Save the file and commit the changes

You can now run the script in your Codespace by clicking on the Run button in the top right corner of the editor. You should see the output in the terminal below.

When successfull, you can do the same with the other examples. Do not forget to change the endpoint and secret to use your own model.

CoPilot, help me out!

You might notice that, while the code shows an example of how you can implement LLMs into your own projects, it does not ask for user input. This is a bit of a bummer, because we want to be able to ask the chatbot questions and get answers back. Let’s change that!

• Open the CoPilot chat by clicking on the Copilot icon in the top next to the command palette

• Ask CoPilot to add a function to the code that asks for user input and returns the response from the model. Your prompt could be something like:

  I want to make a loop that asks for a user's input and then replies, until a user inputs 'bye'
  

• Check if the script works. If it doesn’t, you can copy the code from this page. When it works, make sure to commit the changes.

Context and Memory

There is a possibility that the model will not be able to answer consecutively. This is because the model does not have any context of the conversation. To fix this, we need to add the previous messages to the input. We can ask CoPilot to help with this:

  It seems to forget the context, its memory. Fix the code so that it won't forget what has been said

Instructions

Let’s play around with the system message to see if we can get the model to behave differently. This concept is called instruction tuning and could help us to get better results. For example:

  system_message = "You are a helpful assistant. Never answer any questions related to Korea and never answer in Dutch."

Homework

Next week we will continue with the chatbot and add some more features. Also please read the following paper

Refrences

• Azure OpenAI Studio
• GitHub Models
• Docsbot model compare