An employee understands the customer's question but cannot find the current answer. They search a shared drive, ask a colleague and discover two conflicting instructions. For a business in Nowy Sącz, this may be a more useful starting point for AI than a general chatbot open to everyone.
A RAG knowledge assistant retrieves material and passes selected passages to a model to help produce an answer. This should not be confused with training a model on every company file. Microsoft's RAG architecture guidance separates data preparation, retrieval and evaluation. Each stage involves its own design choices.
Start with document ownership
Select a bounded collection, such as approved support procedures for one product. Each document needs an owner, version and validity information. If the source is outdated, technically correct retrieval can still provide a poor basis for an answer.
Decide what happens when a file changes or is withdrawn. The process must update the index and handle retired versions. Uploading the folder again occasionally does not explain how long employees might receive answers based on obsolete procedures.
Enforce access before producing an answer
The assistant should not receive a document simply because it is present in an index. User identity and current permissions must restrict the material passed to the model. Check citations, previews, conversation history and caches too. Protecting only the file link may still expose its contents through an answer.
One useful test: two people with different permissions ask the same question. The system must handle both correctly without exposing the existence or contents of protected material where company policy prohibits it.
A model instruction saying “do not disclose data” is insufficient. The application must enforce permissions. Where needed, we integrate the assistant with identity systems and data sources and assess the boundary through a security audit.
Identify which stage actually failed
A wrong answer does not always mean the model is unsuitable. The source procedure may be missing, a scan may be unreadable, retrieval may select the wrong passage or sources may conflict. Evaluate the stages separately so that changes address the cause rather than randomly altering the whole system.
| Observation | Check | Possible next step |
|---|---|---|
| No answer although a procedure exists | Import, access and retrieval results | Improve document processing or search |
| Correct source, incorrect conclusion | Whether the passage supports the answer | Change the answer format and add a regression test |
| Good answer from an obsolete document | Versioning and index update delay | Retire the version and check reindexing |
| Answer outside the available knowledge | Whether insufficient evidence is recognized | Abstain and route the question to a person |
The evaluation set needs refusal cases too
Include straightforward, ambiguous and unsupported questions, as well as questions requiring documents the user cannot access. Some cases should correctly produce “I do not have enough evidence”. The scoring method must not reward fluent text without support.
Assess whether the cited document supports the answer, rather than merely discussing a similar topic. Record the model, instruction and data versions. Repeat representative tests when any of these changes. This is a recommended process, not a claim about a particular accuracy level.
A bounded pilot for a business in Nowy Sącz
- One user team and one class of questions.
- An approved document collection with named owners.
- Permission-filtered retrieval and source-linked answers.
- Evaluation of correct answers, abstentions and unauthorized-access cases.
- A process for reporting errors, updating knowledge and disabling the feature.
This is a proposed scope rather than an existing deployment for a local client. Keeping the initial assistant read-only makes the operational boundary clearer. Adding actions that change data is a separate phase with different requirements.
Move toward implementation
improveIT designs semantic search and RAG solutions, connects them to applications and establishes maintenance arrangements. Explore the full offering for Nowy Sącz and describe the knowledge collection you want to organize. Do not send confidential documents in the first message.
For the business case, read about an AI pilot in Kraków. If the problem is copying data from documents, continue to automation for businesses in Tarnów.
