Hybrid list optimization

.cursor

@@ -0,0 +1 @@
+../../riper/src/modes

.github/workflows/test.yml

@@ -10,7 +10,7 @@ jobs:
     - name: Install Go
       uses: actions/setup-go@v2
       with:
-        go-version: 1.18.x
+        go-version: 1.25.x
     - name: Checkout code
       uses: actions/checkout@v2
     - name: Short test
@@ -22,7 +22,7 @@ jobs:
     - name: Install Go
       uses: actions/setup-go@v2
       with:
-        go-version: 1.18.x
+        go-version: 1.25.x
     - name: Checkout code
       uses: actions/checkout@v2
     - name: Test

.gitignore

@@ -0,0 +1,12 @@
+*.prof
+pgo_profiles/
+pgo_results/
+.idea/
+.cursor/
+memory-bank/private/
+.cursor/README.md
+.cursor/templates/
+.cursor/rules/
+
+# Memba per-developer user id
+.memba/.user_id

README.md

@@ -1,4 +1,4 @@
-Immutable [![release](https://img.shields.io/github/release/benbjohnson/immutable.svg)](https://pkg.go.dev/github.com/benbjohnson/immutable) ![test](https://github.com/benbjohnson/immutable/workflows/test/badge.svg) ![coverage](https://img.shields.io/codecov/c/github/benbjohnson/immutable/master.svg) ![license](https://img.shields.io/github/license/benbjohnson/immutable.svg)
+Immutable [![release](https://img.shields.io/github/release/arnonrgo/immutable.svg)](https://pkg.go.dev/github.com/arnonrgo/immutable) ![test](https://github.com/arnonrgo/immutable/workflows/test/badge.svg) ![coverage](https://img.shields.io/codecov/c/github/arnonrgo/immutable/master.svg) ![license](https://img.shields.io/github/license/arnonrgo/immutable.svg)
 =========
 
 This repository contains *generic* immutable collection types for Go. It includes
@@ -11,13 +11,64 @@ such as`slice` and `map`. The primary usage difference between Go collections
 and `immutable` collections is that `immutable` collections always return a new
 collection on mutation so you will need to save the new reference.
 
+This project is a fork of [github.com/benbjohnson/immutable](https://github.com/benbjohnson/immutable) with additional performance enhancements and builder APIs.
+
+**Performance**: This library includes  batch builders that provide high accelaration for bulk operations (vs. discreet insert), with optimized memory usage 
+and automatic batching. Regular operations maintain ~2x overhead compared to Go's 
+built-in collections while providing thread-safe immutability.
+
 Immutable collections are not for every situation, however, as they can incur
 additional CPU and memory overhead. Please evaluate the cost/benefit for your
 particular project.
 
 Special thanks to the [Immutable.js](https://immutable-js.github.io/immutable-js/)
 team as the `List` & `Map` implementations are loose ports from that project.
 
+Forked from https://github.com/benbjohnson/immutable with the following enhancements:
+
+### **Performance Optimizations**
+
+**Memory Architecture Improvements:**
+- **Hybrid Data Structures**:
+    - **List**: Uses a simple slice for small lists (< 32 elements) for up to 2x faster operations and ~85% less memory usage in common cases, transparently converting to a HAMT for larger sizes.
+    - **Map**: Small-structure fast paths via builder initial flush and tiny array-node updates (≤ 8 keys); core Map remains HAMT.
+- **Pointer-Based Array Sharing (planned)**: Reduce allocations in `mapHashArrayNode.clone()` via pointer-backed children with copy-on-write
+- **Lazy Copy-on-Write**: Arrays shared via pointers until actual modification, reducing memory overhead by 6-8%
+- **Cache-Friendly Design**: Improved memory layout for better CPU cache utilization
+
+
+### **Batch Builders**
+
+**Complete High-Performance Builder Suite:**
+- **`BatchListBuilder`**: up 19x faster in tests (vs. discreet ops) bulk list construction with configurable batch sizes
+- **`BatchMapBuilder`**: Measured gains on bulk construction; biggest wins for initial tiny batches and small structures
+- **`BatchSetBuilder`** & **`BatchSortedSetBuilder`**: Efficient bulk set construction
+- **`StreamingListBuilder`** & **`StreamingMapBuilder`**: Auto-flush with functional operations (filter, transform)
+- **`SortedBatchBuilder`**: Optimized sorted map construction with optional sort maintenance
+
+**Functional Programming Features:**
+- Stream processing with automatic memory management
+- Filter and transform operations for bulk data processing
+- Configurable auto-flush thresholds for memory efficiency
+
+### **Enhanced Testing & Validation**
+
+**Comprehensive Test Coverage:**
+- Extensive benchmark suite measuring performance improvements
+- Memory profiling and allocation analysis
+- Race condition testing (`go test -race`) for thread safety validation
+- Edge case and error condition testing for all new builders
+- Large-scale performance validation (100-100K elements)
+
+
+### **Architectural Enhancements**
+
+**Thread Safety & Immutability:**
+- Lock-free operations with atomic copying
+- Structural sharing maintains thread safety
+- Zero-overhead abstractions (Sets inherit Map optimizations)
+
+
 
 ## List
 
@@ -114,6 +165,10 @@ If you are building large lists, it is significantly more efficient to use the
 a list in-place until you are ready to use it. This can improve bulk list
 building by 10x or more.
 
+For even better performance with bulk operations (100+ elements), see the 
+[Advanced Batch Builders](#advanced-batch-builders) section which provides up 
+to 19x performance improvements.
+
 ```go
 b := immutable.NewListBuilder[string]()
 b.Append("foo")
@@ -217,6 +272,10 @@ If you are executing multiple mutations on a map, it can be much more efficient
 to use the `MapBuilder`. It uses nearly the same API as `Map` except that it
 updates a map in-place until you are ready to use it.
 
+For enhanced performance with bulk operations, see the 
+[Advanced Batch Builders](#advanced-batch-builders) section which provides 
+additional optimizations and functional programming capabilities.
+
 ```go
 b := immutable.NewMapBuilder[string,int](nil)
 b.Set("foo", 100)
@@ -240,6 +299,11 @@ creation.
 Hashers are fairly simple. They only need to generate hashes for a given key
 and check equality given two keys.
 
+**Security Note:** A poorly implemented `Hasher` can result in frequent hash
+collisions, which will degrade the `Map`'s performance from O(log n) to O(n),
+making it vulnerable to algorithmic complexity attacks (a form of Denial of Service).
+Ensure your `Hash` function provides a good distribution.
+
 ```go
 type Hasher[K any] interface {
 	Hash(key K) uint32
@@ -272,6 +336,10 @@ If you need to use a key type besides `int`, `uint`, or `string` or derived type
 need to create a custom `Comparer` implementation and pass it to
 `NewSortedMap()` on creation.
 
+**Security Note:** A slow `Comparer` implementation can severely degrade the
+performance of all `SortedMap` operations, making it vulnerable to Denial of Service
+attacks. Ensure your `Compare` function is efficient.
+
 Comparers on have one method—`Compare()`. It works the same as the
 `strings.Compare()` function. It returns `-1` if `a` is less than `b`, returns
 `1` if a is greater than `b`, and returns `0` if `a` is equal to `b`.
@@ -309,6 +377,151 @@ types.
 The API is identical to the `Set` implementation.
 
 
+## Advanced Batch Builders
+
+For high-performance bulk operations, this library provides advanced batch builders
+that can dramatically improve performance for large-scale data construction. These
+builders use internal batching and mutable operations to minimize allocations and
+provide up to **19x performance improvements** for bulk operations.
+
+### Batch List Builder
+
+The `BatchListBuilder` provides batched list construction with configurable batch
+sizes for optimal performance:
+
+```go
+// Create a batch builder with batch size of 64
+builder := immutable.NewBatchListBuilder[int](64)
+
+// Add many elements efficiently
+for i := 0; i < 10000; i++ {
+    builder.Append(i)
+}
+
+// Or add slices efficiently
+values := []int{1, 2, 3, 4, 5}
+builder.AppendSlice(values)
+
+list := builder.List() // 19x faster than individual Append() calls
+```
+
+**Performance**: Up to 19x faster than direct construction for large lists.
+
+### Batch Map Builder
+
+The `BatchMapBuilder` provides batched map construction with automatic flushing:
+
+```go
+// Create a batch map builder with batch size of 32
+builder := immutable.NewBatchMapBuilder[string, int](nil, 32)
+
+// Add many entries efficiently
+for i := 0; i < 10000; i++ {
+    builder.Set(fmt.Sprintf("key-%d", i), i)
+}
+
+// Or add from existing maps
+entries := map[string]int{"a": 1, "b": 2, "c": 3}
+builder.SetMap(entries)
+
+m := builder.Map() // 8% faster + 5.8% less memory than regular builder
+```
+
+**Performance**: 8% faster with 5.8% memory reduction compared to regular builders.
+
+### Streaming Builders
+
+Streaming builders provide auto-flush capabilities and functional operations:
+
+```go
+// Streaming list builder with auto-flush at 1000 elements
+builder := immutable.NewStreamingListBuilder[int](32, 1000)
+
+// Functional operations
+data := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
+
+// Filter even numbers
+builder.Filter(data, func(x int) bool { return x%2 == 0 })
+
+// Transform by doubling
+builder.Transform(data, func(x int) int { return x * 2 })
+
+list := builder.List() // Contains processed elements
+```
+
+```go
+// Streaming map builder with auto-flush and bulk operations
+builder := immutable.NewStreamingMapBuilder[int, string](nil, 32, 500)
+
+// Add individual entries (auto-flushes at 500 elements)
+for i := 0; i < 1000; i++ {
+    builder.Set(i, fmt.Sprintf("value-%d", i))
+}
+
+// Bulk operations with auto-flush
+builder.SetMany(map[int]string{10: "ten", 20: "twenty", 30: "thirty"})
+
+m := builder.Map()
+```
+
+### Batch Set Builders
+
+Set builders provide efficient bulk set construction:
+
+```go
+// Batch set builder
+builder := immutable.NewBatchSetBuilder[string](nil, 64)
+
+values := []string{"apple", "banana", "cherry", "apple"} // "apple" duplicate
+builder.AddSlice(values)
+
+set := builder.Set() // Contains 3 unique values
+
+// Sorted set builder with sort maintenance
+sortedBuilder := immutable.NewBatchSortedSetBuilder[int](nil, 32, true)
+numbers := []int{5, 2, 8, 1, 9, 3}
+sortedBuilder.AddSlice(numbers)
+
+sortedSet := sortedBuilder.SortedSet() // Automatically sorted: [1, 2, 3, 5, 8, 9]
+```
+
+### Sorted Batch Builder
+
+For sorted maps, use `SortedBatchBuilder` with optional sort maintenance:
+
+```go
+// Maintain sort order in buffer for optimal insertion
+builder := immutable.NewSortedBatchBuilder[int, string](nil, 32, true)
+
+// Add in random order - automatically maintained in sorted buffer
+builder.Set(3, "three")
+builder.Set(1, "one")
+builder.Set(2, "two")
+
+sm := builder.SortedMap() // Efficiently constructed sorted map
+```
+
+### Performance Guidelines
+
+**When to use batch builders:**
+- Building collections with 100+ elements
+- Bulk data import/export operations
+- Processing large datasets
+- When memory efficiency is critical
+
+**Batch size recommendations:**
+- **Small operations (< 1K elements)**: 16-32
+- **Medium operations (1K-10K elements)**: 32-64  
+- **Large operations (> 10K elements)**: 64-128
+- **Memory-constrained environments**: 16-32
+
+**Performance improvements:**
+- **List construction**: Up to 19x faster for bulk operations
+- **Map construction**: 8% faster with 5.8% memory reduction
+- **Set construction**: Inherits map performance benefits
+- **Streaming operations**: Automatic memory management with functional programming
+
+
 ## Contributing
 
 The goal of `immutable` is to provide stable, reasonably performant, immutable
@@ -320,3 +533,82 @@ issue will be closed immediately.
 
 Please submit issues relating to bugs & documentation improvements.
 
+
+### What's New (2025-09)
+
+- zero dependencies
+- Small-structure fast paths:
+  - List: Batch flush extends slice-backed lists in a single allocation
+  - Map: Empty-map batch flush builds an array node in one shot (last-write-wins); tiny array-node updates applied in-place when under threshold
+- New builder APIs:
+  - `(*BatchListBuilder).Reset()` and `(*BatchMapBuilder).Reset()` for builder reuse without reallocations
+- Concurrency:
+  - Added concurrent read benchmarks and mixed read/write benchmarks (immutable Map vs `sync.Map`)
+  - Added concurrency correctness tests (copy-on-write isolation under concurrent readers)
+
+### Current Performance Snapshot
+
+- Map Get (10K): immutable ~14.5 ns/op (0 allocs); builtin map ~6.8 ns/op; `sync.Map` Load ~20.3 ns/op
+- Map RandomSet (10K): ~595–687 ns/op, 1421 B/op, 7 allocs/op (after tuning)
+- Concurrent reads (ns/op, lower is better):
+  - 1G: immutable 3.53 vs `sync.Map` 6.03
+  - 4G: immutable 2.31 vs `sync.Map` 3.21
+  - 16G: immutable 2.39 vs `sync.Map` 3.24
+- Mixed read/write (ns/op):
+  - 90/10 (9R/1W): immutable 26.0 vs `sync.Map` 38.4
+  - 70/30 (7R/3W): immutable 24.6 vs `sync.Map` 65.0
+  - 50/50 (5R/5W): immutable 27.3 vs `sync.Map` 47.4
+
+### New APIs (Builders)
+
+```go
+// Reuse list builder across batches without reallocations
+lb := immutable.NewBatchListBuilder[int](64)
+// ... append/flush ...
+lb.Reset() // clears state, keeps capacity
+
+// Reuse map builder and retain hasher
+mb := immutable.NewBatchMapBuilder[string,int](nil, 64)
+// ... set/flush ...
+mb.Reset() // clears state, preserves hasher & buffer capacity
+```
+
+### Benchmarking & Profiling
+
+- Run all benchmarks with allocations:
+```bash
+go test -bench=. -benchmem -count=3 ./...
+```
+
+- Profile a representative write-heavy benchmark:
+```bash
+# CPU and memory profiles (example: Map RandomSet, size=10K)
+go test -bench=BenchmarkMap_RandomSet/size-10000 -benchmem -run="^$" \
+  -cpuprofile=cpu.prof -memprofile=mem.prof -count=1
+
+# Inspect hotspots
+go tool pprof -top cpu.prof
+go tool pprof -top -sample_index=alloc_space mem.prof
+```
+
+Optional: Enable PGO locally
+```bash
+# Generate a profile and write default.pgo
+go test -bench=. -run="^$" -cpuprofile=cpu.prof -count=1
+go tool pprof -proto cpu.prof > cpu.pb.gz
+go tool pgo -compile=local -o default.pgo cpu.pb.gz
+
+# Use the profile for builds/tests (Go 1.21+)
+go test -bench=. -benchmem -count=3 -pgo=auto ./...
+```
+
+- Compare immutable vs `sync.Map` concurrent reads:
+```bash
+go test -bench=BenchmarkConcurrentReads -benchmem -run="^$" -count=1
+```
+
+- Mixed workload (reads/writes):
+```bash
+go test -bench=BenchmarkConcurrentMixed -benchmem -run="^$" -count=1
+```
+