Fix duplicate citations and malformed brackets in Azure AI responses

[Copilot]

• Analysis: Understand the two issues reported
  • Issue 1: Too many citations shown - OpenWebUI receives both filtered citation events AND unfiltered context.citations from SSE JSON
  • Issue 2: Empty brackets [] and malformed [[[docX]](url)] appearing in responses from followup generation
• Fix streaming path: Strip context.citations and context.all_retrieved_documents from SSE JSON chunks to prevent OpenWebUI from displaying unfiltered citations
• Fix non-streaming path: Strip context.citations and context.all_retrieved_documents from response JSON
• Fix malformed brackets: Add logic to clean up malformed bracket patterns like [[[doc1]](url)]] or []
• Fix duplicate URLs: Add negative lookahead to skip [docX] already followed by (url), and pattern to clean [[doc1]](url)(url) -> [[doc1]](url)
• Run linting: Validate changes with ruff check - passed
• Code review: Addressed feedback - removed duplicate comment, simplified variable assignment
• Security scan: CodeQL checker - no alerts found

Original prompt

---

This section details on the original issue you should resolve

<issue_title>azure_ai_foundry.py is working a treat, but I am getting too many sources</issue_title>
<issue_description>### Bug title

Only 4 or 5 sources are shown in the response but many more are added

Describe the bug

In this example I had 3 sources added (the ones mentioned in the response) but after 1 second, more got added.
The first 3 also have a high score (80%) and the rest is 50% or lower.

I tried turing off followup generation but this did not have an effect.

I tried analyzing this with Copilot but could not apply a fix:

What you’re seeing

During streaming, your function emits only the citations that are actually referenced in the content (e.g., [doc1], [doc2]). That’s by design in __emit_openwebui_citation_events which filters by the indices it finds in the text.
Shortly after the stream finishes, additional citations appear that are not referenced in the generated content.

Why this happens
In the streaming path, you do not remove or filter the raw context.citations included in the final message JSON that comes from Azure. Your code only emits filtered citation events, but it leaves the provider’s full context.citations list intact in the SSE JSON that the UI also receives. The UI then renders those extra citations when it processes the final message, which looks like “more citations arriving a bit later.”
Concretely:

_emit_openwebui_citation_events(...) filters to only the referenced [docX] indices. If none are found, it falls back to all citations (for backward compatibility).
stream_processor_with_citations(...) updates content (turning [docX] into clickable links) but does not prune choices[].message.context.citations (or choices[].delta.context...) from the SSE JSON lines it forwards.
In the non‑stream path, enhance_azure_search_response(...) converts [docX] to links but likewise does not filter the citations array in the returned response.

As a result, you get two sources of citations in the UI:

Your filtered OpenWebUI citation events (correct, only referenced), and
The provider’s unfiltered context.citations included in the final message (causing extra, unreferenced cards to appear afterwards).

Steps to reproduce

1. Create new funtion
2. Connect with azure ai and azure ai data source
3. Chat

Environment

OWUI latest version
Latest azure_ai_foundry.py

Additional context

No response</issue_description>

<agent_instructions>During followup generation, [] is sometimes used multiple times, or there are empty brackets, and sometimes it's incomplete. See this answer, which has been run multiple times with followup generation:

The available environment variables (ENV) options in the n8n Pipeline include configurations for integrating with external tools and services, such as n8n itself, Google Gemini, and Azure AI. Some key options are:

- **N8N Integration:**
  - `N8N_URL`: The webhook endpoint URL for n8n.
  - `N8N_BEARER_TOKEN`: Token for secure access.


These environment variables allow detailed customization of the pipeline's behavior, including safety settings, model handling, image processing, response streaming, and data source configurations [[doc3]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-integration.md)[[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)[[doc5]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md).The options available for environment variables (ENV) in the n8n Pipeline include configurations for integrating with various AI services and customizing pipeline behavior. Based on the retrieved documents, the main options are:

### Azure AI Integration
- `AZURE_AI_PIPELINE_PREFIX`: Prefix for pipeline display names [[[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md)].
- `AZURE_AI_API_KEY`: API key for Azure AI [[[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md)].
- `AZURE_AI_MODEL`: Model names, separated by semicolons or commas [[[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md)].
- `AZURE_AI_MODEL_IN_BODY`: Include model name in request body (`true`/`false`) [[[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md)].
- `AZURE_AI_USE_AUTHORIZATION_HEADER`: Use `Authorization: Bearer` header (`true`/`false`) [[[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md)].
- `AZURE_AI_DATA_SOURCES`: JSON configuration for Azure Search data sources [[[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md)].
- `AZURE_AI_ENHANCE_CITATIONS`: Enable enhanced citation display (`true`/`false`) [[[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md)]].

These environment variables allow detailed cust...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes owndev/Open-WebUI-Functions#123

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for you](https://github.com/owndev/Open-WebUI-Functions/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.

[owndev]

@copilot In follow-up generation, the Markdown URL is now set twice. See:

The available environment variables (ENV) options in the n8n Pipeline include configurations for integrating with external tools and services, such as n8n itself, Google Gemini, and Azure AI. Some key options are:

- **N8N Integration:**
  - `N8N_URL`: The webhook endpoint URL for n8n.
  - `N8N_BEARER_TOKEN`: Token for secure access.

These variables allow detailed customization of the pipeline's behavior, response handling, and integration with external AI services [[doc3]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-integration.md)[[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)[[doc5]](https://xxx.blob.core.windows.net/xxx/xxx/azure-ai-integration.md).The options available for environment variables (ENV) in the n8n Pipeline include configurations for integrating with n8n itself, Google Gemini, and Google Cloud Vertex AI. These variables allow you to customize the behavior, security, and performance of your workflows. Some key options are:

- **n8n Integration:**
  - `N8N_URL`: Specifies the webhook endpoint URL for n8n, e.g., `"https://n8n.yourdomain.com/webhook/your-webhook"` [[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/).

- **Security and Authentication:**
  - `N8N_BEARER_TOKEN`: Token for secure access to the n8n webhook.
  - `CF_ACCESS_CLIENT_ID` and `CF_ACCESS_CLIENT_SECRET`: Cloudflare Zero Trust tokens if behind Cloudflare Access (https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/).

- **Tool Input/Output Configuration:**
  - `INPUT_FIELD`: The request payload field used to send messages.
  - `RESPONSE_FIELD`: The response payload field used to read the output [[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/).

- **Display and Verbosity Settings:**
  - `TOOL_DISPLAY_VERBOSITY`: Controls how detailed the tool call display is, with options like `minimal`, `compact`, or `detailed` [[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/).
  - `TOOL_INPUT_MAX_LENGTH` and `TOOL_OUTPUT_MAX_LENGTH`: Limit the length of input and output texts for tool displays, defaulting to 500 characters, with `0` meaning unlimited [[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/).

- **Advanced Features:**
  - `N8N_PIPELINE_PREFIX`: Prefix for pipeline display names.
  - `N8N_WORKFLOW_ID`: Identifier for specific workflows.
  - `N8N_DEBUG`: Enable debug logging for troubleshooting (https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/).

These environment variables enable flexible configuration of the n8n pipeline, including security, display preferences, and integration details [[doc2]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/).The options available for environment variables (ENV) in the n8n Pipeline include configurations for integrating with n8n itself, Google Gemini, and Google Cloud Vertex AI. These variables allow you to customize the behavior, security, and performance of your workflows. Some key options are:

- **n8n Integration:**
  - `N8N_URL`: Specifies the webhook endpoint URL for n8n, e.g., `"https://n8n.yourdomain.com/webhook/your-webhook"` [[doc1]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md).
  - `N8N_BEARER_TOKEN`: Token for secure access to the n8n webhook [[doc1]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md).
  - `INPUT_FIELD`: The request payload field used by Open WebUI to send messages [[doc1]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md).
  - `RESPONSE_FIELD`: The response payload field used by Open WebUI to read responses [[doc1]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md).
  - `CF_ACCESS_CLIENT_ID` and `CF_ACCESS_CLIENT_SECRET`: Cloudflare Zero Trust tokens if behind Cloudflare Access [[doc1]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md).

- **Google Gemini Integration:**
  - `USE_PERMISSIVE_SAFETY`: Enable permissive safety settings (`true`/`false`) [[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)(https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md).
  - `GOOGLE_MODEL_CACHE_TTL`: Cache duration for model lists in seconds [[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)(https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md).
  - `GOOGLE_RETRY_COUNT`: Number of retry attempts for API calls [[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)(https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md).
  - `GOOGLE_IMAGE_MAX_SIZE_MB`, `GOOGLE_IMAGE_MAX_DIMENSION`, etc.: Settings for image processing [[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)(https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md).

- **Google Cloud Vertex AI:**
  - Similar environment variables are used to configure token limits, safety, retries, image handling, and other settings, enabling detailed customization of the AI interactions [[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)(https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md).

These environment variables provide flexible options to tailor the pipeline's operation, security, and integration capabilities [[doc1]](https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)(https://xxx.blob.core.windows.net/xxx/xxx/n8n-tool-usage-display.md)[[doc4]](https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md)(https://xxx.blob.core.windows.net/xxx/xxx/google-gemini-integration.md).

[Copilot]

@copilot In follow-up generation, the Markdown URL is now set twice. See:

Fixed in commit f5d8082. Added three changes:

1. DOC_REF_PATTERN now uses negative lookahead (?!\() to avoid converting [docX] when already followed by (url)
2. DUPLICATE_URL_PATTERN removes duplicate URLs: [[doc1]](url)(url) → [[doc1]](url)
3. Updated cleanup conditions to also trigger on ]( pattern

This prevents the double URL issue where Azure AI's existing URL was being duplicated with our converted URL.