fix: preserve multimodal content during handoffs with nest_handoff_hi…

[saksham-1304]

Fix: Preserve multimodal content during same-turn handoffs

Fix #2220

Summary

This PR fixes an issue where multimodal content (images, files, audio) uploaded by users was being lost when the agent performed a handoff to another agent in the same turn with nest_handoff_history=True (the default behavior).

Problem: When a user uploaded an image and the first agent handed off to a specialist agent, the target agent lost access to the uploaded image because the nest_handoff_history function converted all content to a plain text summary, discarding the multimodal data.

Solution: Modified nest_handoff_history to:

1. Extract and preserve multimodal content from user messages during the handoff
2. Include the preserved multimodal content as a separate user message alongside the text summary
3. Provide a cleaner summary that indicates the presence of multimodal content without dumping raw JSON
4. Mark preserved messages to prevent duplication across chained handoffs

Changes

Core Fix

• src/agents/handoffs/history.py:
  • Added _MULTIMODAL_CONTENT_TYPES constant to identify multimodal types
  • Added _PRESERVED_MULTIMODAL_MARKER constant to mark preserved messages and prevent duplication
  • Modified default_handoff_history_mapper() to extract, preserve, and carry forward multimodal content
  • Added _extract_multimodal_content() function to scan user messages for images, files, and audio
  • Added _collect_preserved_multimodal_content() function to collect already-preserved content from previous handoffs
  • Added _stringify_content_list() function to provide human-readable summaries of multimodal content

Tests

• tests/test_extension_filters.py:
  • Added 7 comprehensive tests covering:
    • Image content preservation
    • File content preservation
    • Multiple images across messages
    • Audio content preservation
    • Text-only messages (backward compatibility)
    • Multimodal content in non-user messages (negative case)
    • Chained handoffs (no duplication across A → B → C handoffs)

Testing

✅ All 20 existing extension filter tests pass
✅ All 14 handoff tool tests pass
✅ All 17 run step processing tests pass
✅ All 7 new multimodal preservation tests pass (including chained handoffs test)
✅ Code passes ruff format and ruff check
✅ Code passes mypy --ignore-missing-imports --no-warn-unused-ignores

Test Plan:

1. Run: pytest tests/test_extension_filters.py -v
2. Run: pytest tests/test_handoff_tool.py tests/test_run_step_processing.py -v
3. Run: ruff format && ruff check
4. Run: mypy src/agents/handoffs/history.py tests/test_extension_filters.py --ignore-missing-imports

Impact

Backward Compatible: Yes. Text-only conversations work exactly as before. Multimodal content is now additionally preserved when present.

Example Usage:

# Before: Image lost during handoff
# After: Image preserved alongside conversation summary
welcome_agent.handoffs = [
    handoff(specialist_agent, input_filter=HandoffFilters.welcome_to_specialist),
    # nest_handoff_history=True by default - now preserves multimodal content
]

Related Issue

Resolves the issue reported about multimodal content being lost during same-turn handoffs with nest_handoff_history=True.

Checklist

• New tests added for multimodal content preservation
• All existing tests pass
• Code formatted with ruff format
• Code passes linting with ruff check
• Code passes type checking with mypy
• Changes are backward compatible
• Commit messages are concise and in imperative mood
• Changes follow SDK guidelines for style and documentation

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[saksham-1304]

@seratch Sir Please review the PR