Feat: Embedded HTTP API Server #7 #9

DmarshalTU · 2025-09-19T09:09:36Z

Feat: Embedded HTTP API Server #7 requirements. Here's what was accomplished:

Changes:

SRE Server Implementation (internal/sre/server.go):
- Complete HTTP API server with REST endpoints
- Real-time event streaming via Server-Sent Events (SSE)
- Alert management and status tracking
- Integration with Kubernetes Hook resources
Workflow Integration:
- Updated workflow/coordinator.go to accept SRE server parameter
- Updated workflow/workflow_manager.go to pass SRE server to processor
- Updated pipeline/processor.go to integrate with SRE server
- Updated cmd/main.go to start the SRE server on port 8082

Enhanced API Implementation (FR1-FR6):

FR1: Embedded HTTP API Server
- ✅ Configurable port (default: 8082): Implemented
- ✅ Graceful startup/shutdown: Implemented with context cancellation
- ✅ Request routing, middleware, error handling: Implemented with HTTP mux
- ⚠️ Authentication/authorization: Basic structure (needs enhancement)
- ✅ Request logging: Implemented
FR2: Event Query Endpoints
- ✅ GET /api/v1/events: Implemented with query parameter filtering
- ✅ GET /api/v1/events/{namespace}/{name}/events: Implemented
- ✅ Query Parameters: namespace, eventType, resourceName, status filtering
- ✅ Response format: JSON with data and total count
  FR3: Event Statistics and Aggregation
- ✅ GET /api/v1/stats/events/summary: Implemented
- ✅ GET /api/v1/stats/events/by-type: Implemented with percentages
- ✅ Response format: Matches the specified JSON structure
  FR4: Real-time Event Streaming
- ✅ Server-Sent Events: /api/v1/events/stream implemented
- ✅ Connection management: Client channel management with cleanup
- ✅ Heartbeat mechanism: 30-second heartbeat for connection health
- ✅ CORS headers: Proper CORS configuration
  FR5: Hook Configuration Management
- ✅ GET /api/v1/hooks: Lists all Hook configurations
- ✅ Hook status integration: Syncs with active events
- ⚠️ Individual hook endpoints: Basic structure (needs enhancement)
  FR6: Health and Diagnostics
- ✅ GET /api/v1/health: Health check endpoint
- ✅ GET /api/v1/diagnostics: Comprehensive diagnostics
- ✅ GET /api/v1/metrics: Prometheus-style metrics
- ✅ Health information: API status, memory usage, connection counts

🔧 Key Features Implemented:

Dual API Structure: Both /api/v1/* (new) and /api/* (legacy) endpoints
Real-time Integration: Events automatically create alerts in the SRE server
Comprehensive Filtering: Query parameters for namespace, event type, resource name, status
Statistics & Metrics: Event counts, percentages, severity breakdowns
Streaming Support: Server-Sent Events for real-time updates
Health Monitoring: Detailed diagnostics and metrics endpoints

How to Use:

Start the controller: The SRE server will automatically start on port 8082
Access the API:
- Events: GET http://localhost:8082/api/v1/events
- Statistics: GET http://localhost:8082/api/v1/stats/events/summary
- Real-time stream: GET http://localhost:8082/api/v1/events/stream
- Health: GET http://localhost:8082/api/v1/health
Query Examples:
- Filter by namespace: GET /api/v1/events?namespace=production
- Filter by event type: GET /api/v1/events?eventType=pod-restart
- Filter by status: GET /api/v1/events?status=firing

DmarshalTU · 2025-09-19T11:23:41Z

This is general structure, don't approve till all functionality added

Signed-off-by: Denis Tu <dmarshaltu@Deniss-MacBook-Air.local> Signed-off-by: Denis Tu <dmarshaltu@gmail.com>

Signed-off-by: Denis Tu <dmarshaltu@gmail.com>

EItanya

This is super awesome, thanks so much for getting this started! This is gonna unlock so many awesome use-cases :)

I think there are a few conversations we need to have before merging, but otherwise super happy about this.

EItanya · 2025-09-19T15:06:33Z

api/v1alpha2/hook_types.go

+	// AgentId specifies the Kagent agent to call when this event occurs (legacy format)
+	// +kubebuilder:validation:Optional
+	AgentId string `json:"agentId,omitempty"`


Why was this added back in, this format has been deprecated and has many issues.

I guess a rebase gone wrong. @DmarshalTU - we switched to AgentRef instead of AgentId to be aligned with the rest of Kagent

EItanya · 2025-09-19T15:07:10Z

internal/pipeline/processor.go

 	deduplicationManager interfaces.DeduplicationManager,
 	kagentClient interfaces.KagentClient,
 	statusManager interfaces.StatusManager,
+	sreServer interface{},


Why is this an interface{} and not a concrete type?

EItanya · 2025-09-19T15:07:35Z

internal/pipeline/processor.go

-	agentRef := types.NamespacedName{
-		Name:      match.Configuration.AgentRef.Name,
-		Namespace: agentRefNs,
+	// Handle both agentId (legacy) and agentRef (new) formats


Not a fan of this as I mentioned above

EItanya · 2025-09-19T15:08:18Z

internal/pipeline/processor.go

 		// Continue even if status recording fails
 	}

+	// Add alert to SRE server if available


At a high level, why do we have a callout to the server from here, rather than the server sharing the same datasource as this component?

EItanya · 2025-09-19T15:08:46Z

internal/sre/server.go

@@ -0,0 +1,1590 @@
+package sre


Why is this specifically called sre. I think we should think about this as a general httpserver for khook rather than anything specifically for your UI.

apiServer would also work

EItanya · 2025-09-19T15:09:34Z

internal/sre/server.go

+// AlertSummary represents alert statistics
+type AlertSummary struct {
+	Total        int            `json:"total"`
+	Firing       int            `json:"firing"`
+	Resolved     int            `json:"resolved"`
+	Acknowledged int            `json:"acknowledged"`
+	BySeverity   map[string]int `json:"bySeverity"`
+	ByEventType  map[string]int `json:"byEventType"`
+}


Should we rely on a db for this similar to kagent. We can use sqlite by default, but rely on the same Postgres instance of kagent if it's available.

EItanya · 2025-09-19T15:10:49Z

internal/sre/server.go

+
+// convertAgentIDFormat converts agent ID from kagent/k8s-agent to kagent__NS__k8s_agent format
+// This matches the Python identifier format used by kagent
+func convertAgentIDFormat(agentRef string) string {


We have libraries for this in kagent

EItanya · 2025-09-19T15:18:34Z

internal/sre/server.go

A couple of general comments about this file:

Alerting is really an entirely new feature set, I'd personally prefer that we merge a v1 of the httpserver before adding alerting.

This file is very very large, can we split the handlers up by feature set.

I agree, would also like the api server and alerting as 2 separate features. And yes, splitting the handlers would make this easier to comprehend and change.

DmarshalTU · 2025-09-19T15:28:40Z

This is super awesome, thanks so much for getting this started! This is gonna unlock so many awesome use-cases :)

I think there are a few conversations we need to have before merging, but otherwise super happy about this.

So basically I showed case to @antweiss my use case for khook in the ide, and unfortunately I lost the source code, so this is fast attempt to recreate it for me and share my thoughts and needs of that, so I think no need to merge at least at that state and we should discuss main guidance and code new one but keep this PR for reference.

antweiss

@DmarshalTU in general - this is great. 2 important features many folks I'm sure can benefit from!
To summarize:

check out @EItanya comments regarding keeping in line with the rest of kagent - it's important we keep things uniform
lets change all sre references to somehting more generic like apiserver or httpserver
let's limit this PR to only adding an API server for existing types and add alerting as a separate feature (which I'm very thankful for)
Use agentRef rather then agentId

anything I missed?
I do realize these are massive changes. But then - this is a massive feature!

antweiss · 2025-09-23T08:14:37Z

api/v1alpha2/hook_types.go

+	// AgentId specifies the Kagent agent to call when this event occurs (legacy format)
+	// +kubebuilder:validation:Optional
+	AgentId string `json:"agentId,omitempty"`


I guess a rebase gone wrong. @DmarshalTU - we switched to AgentRef instead of AgentId to be aligned with the rest of Kagent

antweiss · 2025-09-23T08:17:28Z

cmd/main.go

 		os.Exit(1)
 	}

+	// Start SRE-IDE server


I have an issue with nomenclature - I know you built this specifically for your use case but would like to make stuff generic so other people can potentially build on them. I think just calling it "API server" will work better.

antweiss · 2025-09-23T08:20:07Z

internal/sre/server.go

@@ -0,0 +1,1590 @@
+package sre


apiServer would also work

antweiss · 2025-09-23T08:21:32Z

internal/sre/server.go

I agree, would also like the api server and alerting as 2 separate features. And yes, splitting the handlers would make this easier to comprehend and change.

antweiss · 2025-09-23T08:39:33Z

internal/deduplication/manager.go

 	EventTimeoutDuration = 10 * time.Minute
 	// NotificationSuppressionDuration is the window to suppress re-sending after success
-	NotificationSuppressionDuration = 10 * time.Minute
+	NotificationSuppressionDuration = 30 * time.Second


this needs to be discussed. While the original 10 minutes may be too long, 30 seconds is IMHO too short. Maybe we should at least make this configurable by values/configmap

antweiss · 2025-09-23T08:51:39Z

internal/sre/server.go

+}
+
+// determineSeverity determines alert severity based on event type
+func (s *Server) determineSeverity(eventType string) string {


I don't think we should hard code severity here. My plan for khook is to support pluggable event sources and - so potentially many more eventTypes in the future. Also - severity definitions may change for different teams/environments. I'd sugges to add an optional "severity" field on eventConfigurations in hook - then we can set the default values to what's defined here and allow to change this according to user's needs.

10000%, in fact I’d love to potentially start planning how we’d extend the API/system for multiple sources.

DmarshalTU force-pushed the main branch from 4493043 to c4b264f Compare September 19, 2025 09:13

DmarshalTU requested review from EItanya, antweiss and yuval-k as code owners September 19, 2025 11:46

DmarshalTU force-pushed the main branch 3 times, most recently from fe0510e to 01c3d64 Compare September 19, 2025 12:35

DmarshalTU closed this Sep 19, 2025

DmarshalTU reopened this Sep 19, 2025

Denis Tu and others added 5 commits September 19, 2025 15:57

Feat: Embedded HTTP API Server kagent-dev#7

7902946

Signed-off-by: Denis Tu <dmarshaltu@Deniss-MacBook-Air.local> Signed-off-by: Denis Tu <dmarshaltu@gmail.com>

added promised api and missing chart deps

28dc87b

Signed-off-by: Denis Tu <dmarshaltu@Deniss-MacBook-Air.local> Signed-off-by: Denis Tu <dmarshaltu@gmail.com>

ignore values test file

1adb29c

Signed-off-by: Denis Tu <dmarshaltu@Deniss-MacBook-Air.local> Signed-off-by: Denis Tu <dmarshaltu@gmail.com>

restore after rebase

d758c9c

Signed-off-by: Denis Tu <dmarshaltu@gmail.com>

remove checkKagentConnectivity

d452547

Signed-off-by: Denis Tu <dmarshaltu@gmail.com>

DmarshalTU force-pushed the main branch from b453e8c to d452547 Compare September 19, 2025 12:57

EItanya requested changes Sep 19, 2025

View reviewed changes

antweiss requested changes Sep 23, 2025

View reviewed changes

DmarshalTU mentioned this pull request Nov 8, 2025

Minimal API implementation #15

Open

11 tasks

Feat: Embedded HTTP API Server #7 #9

Are you sure you want to change the base?

Feat: Embedded HTTP API Server #7 #9

Uh oh!

Conversation

DmarshalTU commented Sep 19, 2025

Uh oh!

DmarshalTU commented Sep 19, 2025

Uh oh!

EItanya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DmarshalTU commented Sep 19, 2025

Uh oh!

antweiss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants