{"id":21686,"date":"2024-05-29T11:22:46","date_gmt":"2024-05-29T18:22:46","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/?p=21686"},"modified":"2024-05-29T11:22:46","modified_gmt":"2024-05-29T18:22:46","slug":"do-more-in-microsoft-mesh-with-data-and-ai","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/do-more-in-microsoft-mesh-with-data-and-ai\/","title":{"rendered":"Do more in Microsoft Mesh with data and AI"},"content":{"rendered":"<p>At the beginning of the year, Microsoft Mesh became <a href=\"https:\/\/devblogs.microsoft.com\/microsoft365dev\/microsoft-mesh-now-available-for-creating-innovative-multi-user-3d-experiences-for-the-workplace\/\">generally available<\/a> for powerful 3D collaboration in Teams and in the <a href=\"https:\/\/www.microsoft.com\/store\/productId\/9NLXZJ1FDBD7?ocid=pdpshare\">Mesh app<\/a> on PC &amp; Quest. Since then, Mesh developers have been working on enhancing their custom Mesh experiences by <a href=\"https:\/\/devblogs.microsoft.com\/microsoft365dev\/web-content-in-microsoft-mesh-powered-by-webview\/\">infusing them with live data<\/a> and scaling to more users. However, scaling Mesh events, catering to Mesh newcomers, and reasoning over large amounts of data to make informed business decisions are challenging tasks for any organization. That\u2019s where AI can help. AI can <a href=\"https:\/\/mitsloan.mit.edu\/ideas-made-to-matter\/what-microsofts-satya-nadella-thinks-about-work-future\">democratize expertise<\/a> by providing the right information, in the right place, at the right time, and by taking actions on your behalf. We know that organizations around the world are looking for ways to transform their businesses with AI, so we\u2019ve started exploring what it means to unlock AI for developers building with the Mesh Toolkit.<\/p>\n<p>In this blog we\u2019ll share a first look at how we\u2019re thinking about AI extensibility in Mesh, explain how you can get started today with AI in Mesh, and finally we\u2019ll highlight an example of a partner who is already leveraging AI to elevate their custom Mesh experience.<\/p>\n<h2>AI is better in Mesh<\/h2>\n<p>The first AI scenario we\u2019re unlocking with the Mesh Toolkit is support for custom AI guides (which can also be called chatbots or virtual assistants). Essentially, it\u2019s the ability to get user input and pass it to your AI backend of choice along with your business context. Custom AI guides can address some of our customer\u2019s biggest pain points, including:<\/p>\n<ul>\n<li>Scaling Mesh experiences beyond the capacity of human-staffed events<\/li>\n<li>Orienting Mesh newcomers to a specific space or an experience<\/li>\n<li>Answering company or scenario-specific questions<\/li>\n<li>Acting as a collaborator, reasoning over large amounts of data, providing additional perspectives, and democratizing expertise<\/li>\n<\/ul>\n<p>You may already be familiar with some of these AI benefits from using Copilot across your M365 apps, but what makes AI in Mesh particularly powerful is Mesh\u2019s inherent <strong>spatial context<\/strong>. When you place a custom AI guide in a specific location in your 3D environment, a user will automatically expect the AI interaction to be about the place or thing it is anchored to. Therefore, as a developer, you only need to pass minimal context to the AI in order to make it conversant in your scenario, and you can save time and money in the long run by leveraging smaller, faster, cheaper large language models (LLMs). For example, you don\u2019t need to provide your AI with complicated information like an avatar\u2019s gaze or position to know what someone might be looking at when they interact with your AI. They\u2019re looking at the spot where you placed it!<\/p>\n<p>The ability to call an AI backend with 1) user input, 2) scenario-specific business context, and 3) the spatial context inherent to Mesh is all a developer needs in order to create a useful chatbot experience, so we\u2019re excited to make this available to you now.<\/p>\n<h2>Start adding AI with Mesh 201<\/h2>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-content\/uploads\/sites\/73\/2024\/05\/Screenshot-2024-05-29-202209.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-21707\" src=\"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-content\/uploads\/sites\/73\/2024\/05\/Screenshot-2024-05-29-202209.png\" alt=\"Mesh\" width=\"1205\" height=\"738\" srcset=\"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-content\/uploads\/sites\/73\/2024\/05\/Screenshot-2024-05-29-202209.png 1205w, https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-content\/uploads\/sites\/73\/2024\/05\/Screenshot-2024-05-29-202209-300x184.png 300w, https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-content\/uploads\/sites\/73\/2024\/05\/Screenshot-2024-05-29-202209-1024x627.png 1024w, https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-content\/uploads\/sites\/73\/2024\/05\/Screenshot-2024-05-29-202209-768x470.png 768w\" sizes=\"(max-width: 1205px) 100vw, 1205px\" \/><\/a><\/p>\n<p>Ready to get started building your own custom AI guide in Mesh? Last week at Build we shared an <a href=\"https:\/\/aka.ms\/Mesh201\">updated Mesh 201 developer sample<\/a> which will walk you through the basics. Building on the familiar wind farm scenario from <a href=\"https:\/\/learn.microsoft.com\/en-us\/mesh\/develop\/getting-started\/mesh-101-tutorial\/mesh-101-01-overview-and-setup\">Mesh 101<\/a>, first we deep dive into how to add live data to your scene via WebSlates and scripting \u2013 in this case for collecting and displaying information about potential wind farm locations. Then, using the power of cloud scripting and UI that we provide, we show you how to quickly and easily build a data-driven, AI-powered chatbot to reason about this information and help you decide where to build your next windfarm. Check out the full Build session below, or <a href=\"https:\/\/youtu.be\/Q5LUtwjobgA?si=U-nZf_4A5_xaKVUq&amp;t=648\">jump here<\/a> for the part highlighting AI specifically.<\/p>\n<p style=\"text-align: center;\"><iframe src=\"\/\/www.youtube.com\/embed\/Q5LUtwjobgA?si=LH_zQXCfA6w4GIbp\" width=\"560\" height=\"314\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>While we\u2019re starting to support this scenario with text input as seen in this sample, support for voice input is coming soon. When available, we\u2019ll provide an updated sample that will handle speech to text for you so you don\u2019t have to worry about the additional infrastructure setup or cost of additional services, and your developer workflow will be the same regardless of the input type. We\u2019ll also provide updated UI so that the security and compliance states of the interaction (When is the AI active? Who is it listening to? Can I turn it off\/opt out?) are clearly communicated and handled for you.<\/p>\n<p>This sample uses the Azure OpenAI deployment of GPT 3.5-turbo as our backend, and you\u2019ll see how just a few lines of code are needed to pass our live data and shape the model\u2019s system prompt to create a business-specific AI assistant for data-driven decision-making. However, this is just an example, and your creativity is the limit! You can use any LLM or you can even leverage the power of Copilot via <a href=\"https:\/\/www.youtube.com\/watch?v=pI28N5wMoiE\">Copilot extensions<\/a>, for example.<\/p>\n<p>And of course, building a chatbot is just the beginning for AI extensibility in Mesh. Let\u2019s explore how one of our partners, Sulava is already creatively leveraging the current capabilities to enhance their Mesh experiences.<\/p>\n<h2>Spotlight on Sulava<\/h2>\n<p>Sulava is a Microsoft partner from Finland focused on building cutting-edge solutions for a variety of clients via generative AI and Copilot. As one of the first European countries to develop an official national AI strategy, Finland has been at the forefront of AI and Copilot, and now Sulava wants to bring the AI revolution into Mesh.<\/p>\n<p style=\"text-align: center;\"><iframe src=\"\/\/www.youtube.com\/embed\/Fz931wr5hxw\" width=\"560\" height=\"314\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>Using the capabilities available in the Mesh Toolkit today, Sulava has already incorporated AI in a variety of clever ways into their Mesh experiences. The first example is a teleporting assistant, utilizing AI to navigate large environments with ease. A user can express with natural language where they want to go, which breaks down language barriers and reduces the barrier to entry for new users navigating an unfamiliar space.<\/p>\n<p style=\"text-align: center;\"><iframe src=\"\/\/www.youtube.com\/embed\/_zLnpiUQF2k\" width=\"560\" height=\"314\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>Another creative example they have built is using AI to assist in brainstorming, giving users the ability to write down their ideas and refine them into a draft project plan in seconds. This plan can then be saved into Microsoft Teams utilizing an integration built with Microsoft Power Automate and .NET Core.<\/p>\n<h2>Ready, set, go!<\/h2>\n<p>This is just the beginning for AI in Microsoft Mesh! To get started with Mesh and deep dive into the resources mentioned in this blog, start at <a href=\"https:\/\/developer.microsoft.com\/en-us\/microsoft-365\/mesh\">aka.ms\/MeshCreator<\/a>. Stay tuned for more updates from us as we explore AI representation beyond 2D chat, unlock more sources of context, and leverage the spatial information inherent to Mesh for creating richer, AI-powered experiences.<\/p>\n<p><em>Follow us on <a href=\"http:\/\/twitter.com\/microsoft365dev\">X (Twitter) \/ @Microsoft365Dev<\/a> and subscribe to our <a href=\"https:\/\/www.youtube.com\/microsoft365developer\">YouTube channel<\/a> to stay up to date on the latest developer news and announcements.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Read how you can get started leveraging AI to elevate your custom Microsoft Mesh experience with the Mesh Toolkit.<\/p>\n","protected":false},"author":160852,"featured_media":21706,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[186,327,231,242,328,267,266],"class_list":["post-21686","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft-365-developer","tag-net","tag-ai","tag-azure-openai","tag-copilot","tag-copilot-extensions","tag-mesh-toolkit","tag-microsoft-mesh"],"acf":[],"blog_post_summary":"<p>Read how you can get started leveraging AI to elevate your custom Microsoft Mesh experience with the Mesh Toolkit.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/posts\/21686","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/users\/160852"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/comments?post=21686"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/posts\/21686\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/media\/21706"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/media?parent=21686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/categories?post=21686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/wp-json\/wp\/v2\/tags?post=21686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}