Keeping Up With the Pace of Change Within Microsoft AI Tools
Preview
In this blog post, we discuss the rapid pace of change in AI-building technologies, and share a few of the methods that Gravity Union uses to keep up to date and build future-proof AI solutions through:
Ongoing AI sharing practices
Keeping a log of architectural options
Creating an experiment log per project
Keeping and tracking output quality as a priority metric for success
The Gravity Union Team has been building AI solutions on the Microsoft platform for over five years. Microsoft’s Cognitive Services served as the foundation for our client chatbot agents. Power Virtual Agents supported our Power Platform solutions. When Microsoft introduced semantic capabilities in Azure Search Services in 2021, we built our first RAG index. Every one of our employees has been licensed for Copilot since its launch. If you were to look for any of these tools today, you’d find that they not only have a new name, but they have evolved significantly.
Recency is competing with experience
The AI arena is rapidly evolving. Within Microsoft's own tooling, product names keep changing, and the changes go well beyond rebranding. Layers of newly released AI capabilities are crowding admin dashboards and dominating licensing tables. It's a one-two (and three) punch: the tool you used yesterday not only has a new name and a new location, but has also been joined by another feature that sounds remarkably similar. There's a constant feeling that perhaps there's a better tool for the job.
With the pace of change, keeping on top of new features can feel like a full-time job.
So how do you keep up with the rapid pace of change and build solid, future-proof solutions?
Here's how we've adapted — along with a few of the tools we use along the journey:
We've elevated AI to a required competency
We want everyone in our organization to not only use AI, but to actively consider how to maximize productivity with it. To support this, we include AI in onboarding training and have updated our SDLC policies to keep developers current on new AI capabilities and safety concerns.
Our most impactful learning tool, though, has been our weekly "Thursday is AI Growth Day" sessions — a 30-minute drop-in where we share AI trends. Our monthly rhythm looks like this:
First Thursday – AI practical working tips
Second Thursday – Client enablement
Third Thursday – Innovation showcase
Fourth Thursday – Open mic
Fifth Thursday (~4 times/year) – AI safety and governance
Sessions are recorded and shared, keeping the team sharp and providing just-in-time resources when people need them.
We keep our own log of Microsoft AI capabilities
Microsoft promotes a number of AI building options and each one includes a number of configuration possibilities. For example, within Copilot Studio and Azure AI Foundry alone there are a number configuration choices— such as selecting between Microsoft's foundational models or choosing to use a bring-your-own-model approach. Common AI features such as temperature settings and RAG options may be surfaced uniquely or not available at all depending on these architecture choices. That inconsistency led us to start tracking what we could and couldn't do across our options.
Within our log, we track competing options for building AI solutions on the Microsoft platform, and create a detailed breakdown of what each solution is good at, bad at, and factors that might influence our use of that technology (such as licensing considerations).
Our log of agent options allows us to share insights with our team on which tool is best fit for the job.
We track our AI project experiments
In the past, choosing the best tool for the job wasn’t something that would changed frequently. But today, at the pace of change we’re experiencing with Microsoft, we now form a hypotheses about which tool to use and when. Architecture decisions have become weighted possibilities that require rapid testing.
Some of the questions we consider during AI experimentation:
Is this a single- or multi-agent problem?
Is there a model optimized for this type of work?
How do we optimize licensing — especially when the client is already paying for AI-building tools?
Where is data being processed, and how do we stay within required boundaries? (Note: this is shifting frequently as AI capabilities become more localized.)
We track all of this in an experimentation log. Each technical step in a workflow becomes a hypothesis with defined goals, assumptions, and performance factors. Like a scientist eliminating variables, the log moves us toward the best-fit answer.
A secondary benefit: we now have a historical record of what we've tried and why something did or didn't work.
Keeping track of experiments allows us to retain our learning on what we’ve tried and what did or didn’t work.
We continue to prioritize quality metrics
AI models have a dangerous ability to deliver confident wrong answers. Building solutions that produce reliable, accurate results should be a priority for every builder — and for us, this has become increasingly important as we test new features.
Factors that can influence AI quality include:
Model choice
Model settings (such as temperature and top-k)
Pipeline decisions, including orchestration and MCP choices
Document processing choices
Search and index factors (semantic and vector choices, chunking size)
RAG assembly and sizing
For each choice or setting adjustment, we evaluate:
Processing cost per transaction
Processing speed and response time
Accuracy of the response
Tone and character of the response
Benchmarking is one of the most important factors in ensuring AI is successfully adopted. Users — customers and employees alike — will quickly abandon tools like Q&A chatbots if the results aren't reliable.
Experimentation is a new AI competency
Building successful AI solutions requires more than understanding AI architectures. It requires ongoing engagement with the evolving tools and platforms those solutions are built on. Staying regularly informed, maintaining an architecture options log, and tracking experiments and quality are all ways to ensure your AI automation stays on track.
