Building a Multi-Modal AI Agent

A Step-by-Step Guide to Creating Your Own Multi-modal, Multi-Lingual AI Agent

Introduction

In this tutorial, we're going to create a Multi Modal and Multi Lingual AI Agent that can be extremely helpful for religious studies or as a study guide for any textbook. We'll be focusing on building a agent, specifically a BibleBot, that can interact with users on platforms like WhatsApp or Facebook. This agent will be capable of analyzing pictures of Bible pages and engaging in meaningful conversations about the content.

We support incoming and outgoing languages from Whisper v3 + v2, Google Chirp, Meta Seamless and Azure

TRY THE BIBLE BOT HERE:

Bible Bot • Gooey.AI • Gooey.AIgooey.ai

Step 1: Setting Up the Agent's Identity

Accessing the Workflow

- Navigate to the Gooey homepage.

- Click on "Explore" to view the available workflows.

Customizing the Agent

- Select the agent workflow.

- Rename it to "BibleBot."

- Edit the initial prompt to introduce BibleBot to users. For example:

Hello! Welcome to BibleBot. I'm an AI designed to help you better understand the Bible. You can send me a text or a voice note with any Bible-related question, and I'll give you a great answer. I'm constantly trying to improve, so please give your feedback by tapping the up-down buttons.

Step 2: Uploading the Bible PDF

- Download a PDF version of the Bible, such as the Holy King James version.

- Go to the "Downloads" folder and upload the PDF to the agent. The file size might be around 10 megabytes.

Step 3: Enabling Image Recognition

- To allow users to take pictures of the Bible, set up the agent with photo intelligence capabilities.

- Avoid using GPT Vision for OCR as it may not be very effective.

- Opt for GPT-4 Turbo for faster processing.

- In the “Photo Intelligence” dropdown configure the agent to extract text from images and provide responses

Here we have uploaded a passage from the Bible, with a specific question. The bot uses OCR and GPT4-Vision and provides an answer to the question.

Step 4: Configuring Voice Responses

- Enable the agent to read back answers to the user.

- Use a service like Google Voice for text-to-speech functionality.

Step 5: Submitting and Processing

- After setting up the agent, hit "Submit" to start the embedding process.

- The agent will process the uploaded Bible PDF, which may take some time depending on the size of the document.

- The processed text is stored in a cache for quick access in future interactions.

Step 6: Crafting the Bot's Guidelines

Initial Prompt

- Define what the agent is and how users should interact with it.

- Provide clear instructions for when a user greets the agent.

Response Guidelines

- Establish rules for how the agent should search for answers within the Bible text.

- Decide how the agent should handle questions when sufficient information is not available in the Bible:

to refuse either - Choose to either refuse to answer or attempt to provide a relevant response.

- Set limitations on the topics the agent can discuss, focusing on the Bible and Christianity.

- Optionally, include the use of appropriate emojis for a more engaging experience.

Conclusion

Once you've completed these steps, your BibleBot will be ready for a demo. This AI agent serves as a powerful tool for engaging with religious texts and can be adapted for other educational purposes. Remember to test the agent thoroughly and make adjustments based on user feedback to ensure the best possible experience.