feat(kb): Adding support for more tags to the KB

[Pbonmars-20031006]

Summary

Adding more KB tag types + filters.

Type of Change

• New feature

Testing

Tested Manually.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Review	Updated (UTC)
docs	Skipped		Dec 19, 2025 4:02am

[icecrasher321]

@Pbonmars-20031006 also don't forget to generate the migration using bunx drizzle-kit generate to the sql, snapshot, etc shows up [you can do that before me merge]

[icecrasher321]

@Pbonmars-20031006 build failing here

Also, we realized that the write speed would become really slow if we have this many indices. Can we have 3 for number and 2 for date.

And for text have more filters like "contains" and "starts with" if they have equivalent sql ops -- e.g. https://stackoverflow.com/questions/1069302/using-the-star-sign-in-grep

[greptile-apps]

Greptile Summary

This PR extends the knowledge base tagging system to support multiple field types (number, date, boolean) in addition to the existing text tags. The implementation adds 10 new database columns (5 number, 2 date, 3 boolean) to both document and embedding tables with proper indexes.

Key Changes:

• Database schema extended with typed tag columns and indexes
• New structured filter system with operators (eq, neq, gt, contains, between, etc.)
• Type conversion logic for processing tag values (string→number/date/boolean)
• Backward-compatible API that supports both legacy and structured filter formats
• UI updates to support field type selection and operator configuration

Technical Highlights:

• Proper type safety throughout with TypeScript discriminated unions
• Slot allocation matches schema (5 number, 2 date, 3 boolean slots)
• Tag values propagate from documents to embeddings with correct types
• Query builder validates slots against field types
• Search API handles both simple equality and complex operators

Confidence Score: 4/5

• This PR is safe to merge with minor considerations
• The implementation is well-architected with proper type safety, backward compatibility, and comprehensive coverage across the stack. The score reflects that this is a significant feature addition that extends the data model and affects multiple layers (database, API, UI). While the code quality is high and previous schema mismatch issues have been resolved, the complexity and scope warrant careful testing in production to ensure all type conversions and filter operations work correctly across different data scenarios.
• Pay close attention to apps/sim/lib/knowledge/documents/service.ts for type conversion logic and apps/sim/app/api/knowledge/search/utils.ts for filter operator implementations

Important Files Changed

Filename	Overview
packages/db/schema.ts	Extends document and embedding tables with typed tag columns (number, date, boolean) and corresponding indexes
apps/sim/lib/knowledge/constants.ts	Defines slot configurations for all tag types with proper slot counts matching schema
apps/sim/lib/knowledge/tags/service.ts	Updates tag validation and usage tracking to support all field types, handles type-specific null checks
apps/sim/lib/knowledge/filters/query-builder.ts	New file providing SQL query builder for structured filters with operator support across all field types
apps/sim/lib/knowledge/documents/service.ts	Extends document processing to handle typed tags with proper type conversion and propagation to embeddings
apps/sim/app/api/knowledge/search/utils.ts	Implements filter logic for all tag types with operator support (eq, gt, contains, between, etc.)

Sequence Diagram

sequenceDiagram
    participant User
    participant UI as Tag Filters UI
    participant API as Search API
    participant TagSvc as Tag Service
    participant DocSvc as Document Service
    participant DB as Database (Postgres)

    Note over User,DB: Tag Definition Creation
    User->>UI: Create tag definition (name, fieldType)
    UI->>TagSvc: createTagDefinition()
    TagSvc->>TagSvc: getNextAvailableSlot(fieldType)
    TagSvc->>DB: INSERT knowledgeBaseTagDefinitions
    DB-->>TagSvc: Tag definition created
    TagSvc-->>UI: Success

    Note over User,DB: Document Upload with Tags
    User->>DocSvc: Upload document with tags
    DocSvc->>DocSvc: processDocumentTags(tagData)
    DocSvc->>TagSvc: getDocumentTagDefinitions()
    TagSvc-->>DocSvc: Tag definitions with fieldType
    DocSvc->>DocSvc: Type conversion (string→number/date/boolean)
    DocSvc->>DB: INSERT document with typed tag values
    DB-->>DocSvc: Document created
    DocSvc->>DocSvc: Generate embeddings
    DocSvc->>DB: INSERT embeddings (inherit typed tags)
    DB-->>DocSvc: Embeddings created

    Note over User,DB: Search with Structured Filters
    User->>UI: Configure filter (tag, operator, value)
    UI->>API: POST /api/knowledge/search<br/>{tagFilters: [{tagName, fieldType, operator, value}]}
    API->>TagSvc: getDocumentTagDefinitions()
    TagSvc-->>API: Map tagName → tagSlot + fieldType
    API->>API: Build structured filters
    API->>DB: SELECT with operator-based WHERE<br/>(eq, gt, contains, between, etc.)
    DB-->>API: Filtered results
    API->>API: Map tagSlots → displayNames
    API-->>UI: Results with typed tag values
    UI-->>User: Display search results

Loading

[greptile-apps]

_{24 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}