Skip to content

Conversation

@wlawt
Copy link
Contributor

@wlawt wlawt commented Jan 7, 2026

depends on #123

problem:

  • lots of same txns getting received and we're making a new and unique uuid:new_v4 for it and passing it to the audit topic

proposed solution:

  • use uuid:new_v5 which we can pass in bundle_hash to generate the bundle ID. this would make the UUIDs deterministic
  • have a TTL cache on the ingress layer that keeps track of known bundle IDs and expires after like 30s or something
  • that way, we reduce the number of bundles of the same tx being sent to our audit workers

notes:
This doesn't affect a transaction thats contained across multiple bundles, for example:

 Bundle A = [tx1, tx2, tx3]
 Bundle B = [tx1, tx4, tx5]
 Bundle C = [tx1]

All have tx1 but the UUID will all be different (see how bundle_hash is computed)

Comment on lines +116 to +118
let bundle_cache = Cache::builder()
.time_to_live(Duration::from_secs(20))
.build();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this can just be a simple LruCache with 1k size

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


// A TTL cache to deduplicate bundles with the same Bundle ID
let bundle_cache = Cache::builder()
.time_to_live(Duration::from_secs(20))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could be more strict by having this be 10s which is 5 blocks (or 50 flashblocks) which is quite a long time

@wlawt
Copy link
Contributor Author

wlawt commented Jan 9, 2026

going to merge this into #123 as the experiment turned out fine

@wlawt wlawt merged commit 39a87b4 into archive-event-perf Jan 9, 2026
7 checks passed
@wlawt wlawt deleted the audit-dedup branch January 9, 2026 15:04
wlawt added a commit that referenced this pull request Jan 9, 2026
* chore: use uuid v5 for determinism

* feat: add ttl

* diffs

* fix: in-flight archive task

* Revert "fix: in-flight archive task"

This reverts commit a759f2b.

* tmp: clear backlog by nooping

* chore: log out meterbundleres
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants