Copilot Will Underwhelm You. I Know Why.
I get a version of this question at least once a week: "We turned on Copilot. Why isn't it doing what the demo showed us?"
And every week, before I even open their tenant, I already know what I'm going to find. Not a bad prompt. Not a permissions problem. A tenant full of duplicate documents and no agreed-upon source of truth — the one thing nobody thought to check before they bought the license.
Prompting and Permissions Aren't Wrong — They're Just Not the Whole Story
The first two answers people reach for are prompting and permissions, and I'll say plainly: both matter. If you ask Copilot a vague question, you get a vague answer. If your permissions are a mess — people able to see files they shouldn't, or blocked from files they need — Copilot inherits that mess exactly as it exists in your tenant. Fixing prompts and fixing permissions is real work, and skipping it will absolutely wreck your results.
But here's what I've watched happen dozens of times now: a business does that work. They write better prompts. They clean up their sharing permissions. And Copilot still comes back with answers that make them wonder if they wasted the money.
That's the moment the conversation gets interesting, because prompting and permissions were never the whole diagnosis. There's a third factor, and almost nobody thinks to check it before they buy.
The Real Reason Copilot Underwhelms: Your Data
Copilot can only work with what's actually sitting in your tenant. It doesn't know which version of a document is current unless something tells it. It doesn't know which of four similar contracts is the one that matters unless something distinguishes them. When the underlying data is duplicated, unlabeled, and scattered, Copilot doesn't fail loudly. It just picks one of several plausible answers and hands it to you with total confidence — and you have no way of knowing whether it picked the right one.
I've written before about how most SharePoint environments get built like a file server dragged over from a server closet — folders inside folders, the same document copied into three different project folders "just in case," filenames doing the organizational work that metadata should be doing instead. That structure is survivable for a human being who already knows where things live. It is not survivable for an AI trying to determine, in the space of a few seconds, which document represents the actual source of truth. I've walked through exactly why that folder model breaks down in the piece on SharePoint folders versus metadata — the mechanism is the same one quietly wrecking Copilot results in tenant after tenant.
Here's the part that catches people off guard: none of this is new. The duplicate contracts, the filename versioning, the folders inside folders — that mess has been sitting in the tenant for years, quietly costing people time every time they went looking for something themselves. Nobody flagged it as urgent, because a person searching manually can usually muddle through. They remember which folder they saved the real version in. They recognize the client's letterhead on the actual signed copy. Microsoft built the platform to be flexible enough that a business could organize it any way that worked for them — folders, metadata, some combination of both — and left the last-mile decision to the business itself. Most businesses never made that decision deliberately. They just let folders accumulate the way folders always do. Copilot is the first thing to walk into that environment with no memory, no context, and no patience for guessing, and it's the first thing to make the cost of that undecided last mile visible.
What a Messy Tenant Actually Does to Copilot
Picture a contracts folder with four files in it: Contract_v2.docx, Contract_v2_FINAL.docx, Contract_v2_FINAL_reallyfinal.docx, and Contract_signed.pdf. A person navigating that folder manually has a decent shot at guessing which one is current, mostly from memory and context they're carrying around in their head. Copilot doesn't have that memory. It has four documents with no field, no tag, no signal anywhere that says "this one is the one that counts." Ask it to summarize the current contract terms and it will answer — confidently — from whichever file it weighted as most relevant. That might be the signed version. It might be the draft from three revisions ago.
This is why AI accuracy falls apart exactly at the point where a business has more than one plausible answer sitting in the same library. It isn't a Copilot problem. It's a data problem that was there before Copilot ever arrived — Copilot just makes it visible, fast, and expensive to ignore.
Some of this is fixable without touching a folder at all. A properly configured metadata column — a Status field, an Effective Date field — can auto-populate a lot of what used to require someone remembering to rename a file correctly. I've covered how that autofill approach closes part of the gap in the piece on getting SharePoint a librarian. It helps. It is not, on its own, the fix. A tagging system on top of a duplicated, disorganized library still leaves Copilot choosing between five tagged-but-contradictory documents instead of five untagged ones. The underlying structure still has to be right.
The Question I Actually Ask
Before I'll tell a client whether Copilot is going to work for them, I ask one question, and I ask it the same way every time: Do you know your own source of truth? Can you tell me exactly where that lives? If you don't know, neither will Copilot.
Most people can't answer it right away. That's not a knock on them — it's the same diagnostic I lead every assessment with, going back to the very first thing I tell people about this platform: you can't fix what you don't know is broken, and most businesses don't know what they don't know about their own file environment. Getting a straight answer to "where does the real version of this live" usually takes a genuine walk through the tenant, not a guess from memory.
I don't start that walk-through by asking what people want Copilot to do. I ask them to show me where a specific document type actually lives today — not where it's supposed to live, where it actually does. More often than not, the honest answer involves at least two libraries, a shared drive nobody's fully retired, and a shrug. That shrug is the diagnosis. It's not a Copilot licensing problem, and it's not a prompting problem. It's an answer to a question the business never got asked before: where does the truth live, and does everyone agree on it?
When a business can answer that question clearly — one library, one tagged status field, one place everyone actually uses — Copilot's answers get sharper almost immediately. Nothing about the AI changed. The data underneath it did.
Frequently Asked Questions
If we fix our prompts and permissions first, will that get us most of the way there?
It'll get you part of the way, and it's still worth doing regardless of what else is wrong. But if your library has duplicate copies of the same document with no clear status field distinguishing them, better prompts won't tell Copilot which one to trust. Prompting and permissions are necessary. They aren't sufficient.
How do we know if our data is the problem before we spend money finding out?
Pick the document type your team asks about most — contracts, proposals, whatever you'd actually query Copilot about first — and count how many versions of the same thing exist across your library with no field marking which one is current. If the honest answer is "more than one, and we're not sure," that's your answer.
Does this mean we shouldn't buy Copilot until our data is perfect?
No — perfect isn't the bar, and waiting for it isn't realistic. The bar is knowing where your source of truth lives for the handful of document types your team actually relies on day to day. Get that much settled first. The rest can improve alongside the rollout.
Prompting and permissions get the blame because they're visible and easy to point at. The data underneath rarely gets mentioned, because checking it means admitting nobody's looked in years.
Know where your source of truth lives. Copilot will follow.
Accurate as of August 2026. Microsoft updates its products and pricing regularly.
J. Scott Clark is the President and CEO of The 365 Collective, Inc., a Microsoft 365 consulting and training firm serving small and mid-sized businesses across healthcare, finance, construction, engineering, publishing, and retail.
This is the kind of thing we do every day. If you want to dig into what it looks like for your specific situation, feel free to reach out.