Decouple Log Archiving from Main Process Flow

[jswxstw]

Summary

Currently, log archiving functionality has a strong dependency on external storage systems. While log archiving is essentially a side-path feature, it can currently block or disrupt the main process flow when external storage is unavailable or encounters issues. This needs to be refactored to ensure that log archiving failures do not impact the core business logic and main process execution.

The proposed change should:

• Make log archiving asynchronous and non-blocking
• Implement proper error handling and fallback mechanisms
• Ensure external storage failures only affect the archiving feature itself
• Allow the main process to continue operating normally even when archiving fails

Use Cases

• High Availability Scenarios: When external storage systems (e.g., S3, NAS) experience downtime or network issues, the main application should continue functioning without interruption
• Degraded Mode Operation: When archiving storage is temporarily unavailable, the system should gracefully degrade the archiving feature while maintaining core functionality
• Production Stability: Prevent cascading failures where a non-critical feature (log archiving) causes critical business processes to fail

---

Message from the maintainers:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.

[Joibel]

If you depend upon logs I think you should disable them in workflows and use a dedicated product to collect them.
https://argo-workflows.readthedocs.io/en/latest/configure-archive-logs/