Enterprise Document Intelligence RAG
System
Production AI case study - product management, architecture, implementation, and AWS deployment
Matt Alford | alford.oregon@gmail.com | v1.0
Current footprint 4 active users
2,000+ indexed SharePoint documents
Production AWS deployment
AI stack Microsoft Graph, S3, Bedrock Titan, Pinecone, Cohere Rerank, Claude
Impact Search went from manual folder hunting to seconds
Overview
I built a production retrieval-augmented generation system that syncs organizational knowledge from SharePoint, indexes it
semantically, and lets authenticated users search and summarize internal documents through an AI-powered web interface.
The product turned SharePoint from a passive document repository into an active knowledge layer for search, synthesis, and
decision support.
The system was designed to solve a common enterprise knowledge problem: important answers were scattered across
SharePoint documents, transcripts, PDFs, spreadsheets, and working files. Keyword search was not enough, and users
needed a faster way to find relevant information, understand it, and apply it to decisions.
This project demonstrates hands-on AI product work across enterprise search, document ingestion, RAG architecture, model
selection, source grounding, multi-tenant cloud deployment, and trust design.
Product Problem
As Pilot & Pixel grew rapidly, the volume of SharePoint documents grew with it. Finding the right document at the right time
became a real operating constraint: decisions were delayed, meetings stalled while people searched for context, and team
members wasted time navigating an expanding set of folders instead of moving work forward.
Teams were creating valuable knowledge faster than they could organize it. As the document library grew, people lost time
hunting through files, asking colleagues for context, or recreating prior work because the relevant source was hard to find.
The core user need was:
Help users ask natural-language questions across internal documents and receive useful, source-grounded
answers without manually opening and reading every file.
The product challenge was not simply to add a chatbot. The system needed to retrieve the right source material, preserve
attribution, support different levels of interpretation, and fit into a practical enterprise workflow.
Impact
The impact was immediate upon launch.
Enterprise Document Intelligence RAG System v1.0 | Matt Alford | alford.oregon@gmail.comTeam members no longer had to manually search through a seemingly endless set of SharePoint folders to find the
document they needed. Search went from manual folder hunting to seconds. The system sped up both retrieval and analysis
by letting users ask natural-language questions, inspect relevant source chunks, and generate source-aware summaries
from the same interface.
The system also improved the security posture of AI-assisted document analysis. Because Bedrock was selected as part of
the AWS/S3 workflow, confidential business content could be searched, summarized, and analyzed without sending it to an
external AI provider outside the controlled cloud environment.
The result was a production AI product that helped the team move faster in meetings, reduce document-hunting friction, and
make better use of the knowledge already stored across SharePoint.
Production Scale
The system was built for a geographically dispersed, remote team. In production, it currently supports four users and indexes
more than 2,000 SharePoint documents. Users needed reliable internet-accessible search across the full breadth of
SharePoint document types, with multiple team members able to use the system at the same time during live work.
Production requirements included:
- Support for multiple simultaneous users; currently four active users
- Multi-tenant operation
- Internet-accessible web interface
- Authenticated access
- Continuous SharePoint sync
- Coverage across more than 2,000 indexed documents, including Word documents, PDFs, spreadsheets, and transcript-style
source material
- Cloud deployment using AWS/S3, Bedrock, Pinecone, and the Microsoft Graph API
Users And Use Cases
The initial production users were the four members of the Pilot & Pixel team. Because the team is small, distributed, and
remote, the system needed to support the same people across several recurring use cases:
- Finding the right SharePoint document during live work or meetings
- Retrieving prior decisions, recommendations, and reference material
- Synthesizing answers from a growing document base
- Producing source-aware summaries instead of generic AI responses
My Role
I served as the product manager, architect, and hands-on builder for the production system.
I owned the product direction, architecture, technical implementation, local rollout, cloud migration, and production
deployment.
My work included:
- Gathering requirements and defining the product workflow and user experience
- Designing the system architecture
- Building the ingestion and retrieval pipeline
- Integrating Microsoft Graph, AWS Bedrock, Pinecone, Cohere reranking, and Claude
- Designing summary modes for different user jobs
Enterprise Document Intelligence RAG System v1.0 | Matt Alford | alford.oregon@gmail.com- Implementing the FastAPI web interface
- Designing trust, attribution, deployment, and operational controls
- Rolling the system out locally, migrating it to AWS, and operating it in production
Solution
The system automatically syncs files from a SharePoint document library, processes them into searchable chunks,
generates embeddings, stores them in a vector database, and lets users search through a responsive web interface.
When a user enters a query, the system:
1. Converts the query into an embedding using AWS Bedrock Titan.
2. Searches Pinecone for semantically relevant document chunks.
3. Retrieves more candidates than needed so the system has room to improve ranking.
4. Reranks candidates using Cohere Rerank through Bedrock.
5. Shows source chunks, file metadata, relevance scores, and sync information.
6. Optionally generates a streamed Claude summary grounded in the retrieved sources.
Architecture
The production architecture follows this flow:
Production Architecture Flow
SharePoint Graph Delta Sync S3 Storage Extract + Chunk
Claude Answer + Sources Cohere Rerank Pinecone Index Bedrock Embeddings
User queries follow the same retrieval path, then return grounded answers with visible source context.
Key Capabilities
- Automatic SharePoint sync: Polls the document library for new and changed files using Microsoft Graph delta queries.
- Multi-format ingestion: Extracts text from Word documents, PDFs, and Excel files.
- Semantic retrieval: Uses AWS Bedrock Titan embeddings and Pinecone vector search.
- Reranking: Applies Cohere Rerank to improve attribution quality and result relevance.
- AI summaries: Uses Claude on Bedrock to generate streamed summaries from retrieved source chunks.
- Mode selection: Lets users choose between strict RAG, explanatory context, and applied recommendations.
- Model choice: Supports faster Haiku responses or more capable Sonnet responses depending on the task.
- Authentication and tenant context: Supports session-based login, multiple users, and tenant-scoped retrieval.
Enterprise Document Intelligence RAG System v1.0 | Matt Alford | alford.oregon@gmail.com- Production AWS deployment: Runs as a cloud-deployed system with S3-backed storage, Bedrock model calls, persistent
sync state, and operational smoke tests.
Product Decisions
Strict RAG vs. Interpretation
Not every user question requires the same level of AI interpretation. I designed three summary modes to match different user
jobs:
- RAG: Answers only from retrieved document excerpts. This is the highest-trust mode for source-grounded answers.
- Explain: Uses the documents as the foundation, then adds definitions, background, or step-by-step explanation.
- Wisdom: Applies the source material to current situations, broader implications, or recommendations.
This design makes the AI's behavior more legible. Instead of hiding the tradeoff between groundedness and interpretation,
the product exposes it as a user choice. It also reduces the risk that a user will mistake a broad interpretation for a strictly
sourced answer.
Reranking for Trust
Initial vector retrieval can return semantically similar but poorly attributed chunks, especially when working with transcripts
where the speaker and the subject may differ.
To improve answer quality, the system retrieves a wider candidate set from Pinecone and reranks it with Cohere Rerank.
This improves the chance that the final context sent to Claude is both relevant and attributable.
Source Visibility
The interface does not only return a generated answer. It also shows retrieved source chunks, file names, file types,
relevance scores, chunk indexes, and sync metadata.
This supports a core trust behavior: users can inspect where an answer came from instead of treating the model output as a
black box.
Speed vs. Quality
The UI supports model selection:
- Haiku: Faster default for lightweight summarization.
- Sonnet: More accurate option for higher-stakes or more complex questions.
This reflects a practical AI product tradeoff: not every query deserves the same latency and cost profile.
Confidentiality vs. Model Capability
The SharePoint files contain confidential business material, so the AI stack had to support analysis without casually moving
sensitive data outside the controlled cloud environment.
I chose AWS Bedrock so embeddings, summarization, and reranking model calls could run within the AWS environment
connected to the S3-backed document workflow. This kept the system aligned with the security needs of confidential internal
knowledge while still giving users access to modern foundation-model capabilities.
Trust And Safety
Trust was a first-class product requirement.
The system includes several trust-oriented behaviors:
- Uses source-grounded prompting in RAG mode.
- Asks the model to cite source numbers in generated answers.
Enterprise Document Intelligence RAG System v1.0 | Matt Alford | alford.oregon@gmail.com- Returns source metadata alongside summaries.
- Includes explicit caveat handling when retrieved excerpts do not clearly support an answer.
- Handles empty-result cases with a clear "no relevant documents found" response.
- Uses tenant-scoped filtering so retrieval is constrained by tenant context.
- Keeps secrets and credentials in environment configuration rather than committed code.
- Includes production rollout checks for auth, secrets, tenant config, smoke tests, and rollback planning.
One subtle but important prompt design decision was handling spoken transcripts. The RAG prompt instructs the model to
distinguish between facts about a person and facts that person stated about someone else. This reduces a common
attribution error in transcript-based knowledge systems.
Production Deployment And Operations
The system is deployed in production on AWS infrastructure and runs in a multi-tenant, multi-user mode.
The production system has two long-running processes:
- Web API: FastAPI application serving the search UI, API endpoints, login flow, and summary streaming.
- Supervisor: Background process that continuously syncs SharePoint changes, processes files, writes state, and updates
storage/search indexes.
I first rolled the system out locally, then migrated it to the cloud and deployed it on AWS. The production deployment uses S3
for storage and AWS Bedrock for embeddings, summarization, and reranking model calls. The supervisor maintains
persistent sync state so restarts do not wipe delta progress or trigger unnecessary full resyncs.
Production operations include:
- Health and readiness endpoints
- Auth configuration
- Multi-tenant configuration
- Persistent sync state
- Secrets management
- Smoke tests
- Rollback planning
- Cloud operations documentation
Demo Flow
A typical demo shows:
1. A user searches for a strategic or operational question across the document library.
2. The system returns relevant files and document chunks with scores and metadata.
3. The user enables RAG mode to get a strict source-grounded answer.
4. The user compares Explain mode to see a more educational answer.
5. The user tries Wisdom mode to get recommendations or broader implications.
6. The user reviews the source list to verify where the answer came from.
7. The user switches between Haiku and Sonnet to see the speed-quality tradeoff.
Production Evaluation Approach
Enterprise Document Intelligence RAG System v1.0 | Matt Alford | alford.oregon@gmail.comFor the P&P production implementation, I evaluate and improve the system across both technical and product-quality
dimensions:
- Retrieval quality: Does the right document appear in the top results?
- Reranking quality: Does Cohere improve the final source order compared with raw vector similarity?
- Groundedness: Does RAG mode avoid claims unsupported by retrieved excerpts?
- Citation quality: Do cited sources actually support the answer?
- Mode clarity: Are RAG, Explain, and Wisdom distinct enough for users to understand when to use each?
- Latency: Are Haiku and Sonnet response times appropriate for their use cases?
- User confidence: Do users trust the answer more when source chunks and citations are visible?
- Task completion: Does the system reduce time spent searching through documents?
Current Production Footprint
Current production footprint:
- Number of indexed documents; currently more than 2,000
- Active users; currently four users
- Production AWS deployment
- Continuous SharePoint sync
- Multi-user authenticated access
P&P Production System And Next Enhancements
Production System
- SharePoint document sync
- S3 storage
- Text extraction for DOCX, PDF, and Excel files
- Chunking and embedding
- Pinecone vector search
- RAG summaries with source metadata
- Web search interface
- Multi-user authentication
- Tenant-scoped retrieval
- AWS deployment
Current Enhancement Areas For P&P
- Add GraphRAG so retrieval can use association context between documents, people, topics, and decisions instead of relying
only on chunk-level semantic similarity
- Add a SharePoint cleanup routine that omits files with the same name but an older version number, so only the latest document
versions remain in the index
- Feedback controls on answer quality
- Evaluation set for golden questions and expected sources
- Admin view for indexed files and sync status
Enterprise Document Intelligence RAG System v1.0 | Matt Alford | alford.oregon@gmail.com- Stronger unsupported-question behavior
- Improved observability for ingestion and summary failures
Enterprise Scale Opportunities
- Enterprise SSO
- Audit logs
- Role-based access controls
- Scheduled evaluation runs
- Source freshness monitoring
- Human review workflow for high-risk answers
- Analytics dashboard for adoption, latency, and answer quality
What I Learned
The biggest product lesson was that enterprise RAG is not just a retrieval pipeline. It is a trust product.
The model can only be useful if users understand where answers came from, when the system is being strict versus
interpretive, and when the answer should not be trusted. The product work is in making those boundaries visible.
The biggest technical lesson was that retrieval quality depends on the entire chain: document sync, extraction, chunking,
embeddings, metadata, vector search, reranking, prompt design, and UI presentation. Weakness in any one layer can show
up as a user trust problem.
One of the most painful implementation lessons was around SharePoint change detection. I initially tried to detect and push
document changes in near real time through Microsoft Graph. In practice, that was over-engineered for the team's needs:
Graph API configuration was difficult, error reporting was poor, and the extra responsiveness did not justify the operational
complexity. I simplified the system to poll SharePoint every five minutes, which was much more reliable and still fast enough
for the actual workflow.
Why This Matters For AI Product Management
This project shows that I can move beyond AI strategy into hands-on AI product execution.
It demonstrates that I can:
- Identify a high-value enterprise workflow for AI augmentation.
- Build a functional RAG system using modern AI infrastructure.
- Make product decisions around trust, grounding, user control, and deployment.
- Understand the tradeoffs between speed, accuracy, cost, and interpretability.
- Translate a working AI system into a production product with a clear path for continued enterprise hardening.
The result is a practical example of AI-native product management: finding a real user problem, building and operating the
system, understanding the architecture, and shaping the product around adoption and trust.
Enterprise Document Intelligence RAG System v1.0 | Matt Alford | alford.oregon@gmail.com