chore(async search): search per N-minute intervals - #471
Conversation
…support compaction
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #471 +/- ##
==========================================
- Coverage 70.44% 70.35% -0.10%
==========================================
Files 232 232
Lines 18371 18448 +77
==========================================
+ Hits 12942 12979 +37
- Misses 4453 4481 +28
- Partials 976 988 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| } | ||
|
|
||
| // acquireFractionsInRange returns a read-only subset of fractions within the range | ||
| func (r *fractionRegistry) acquireFractionsInRange(from, to seq.MID) ([]frac.Fraction, func()) { |
There was a problem hiding this comment.
By the way, we can reuse this API for regular fetches and searches. IIRC, now we acquire all fractions -- even those that do not intersect with user provided time range.
| return fmt.Sprintf("%d-%d", interval[0], interval[1]) | ||
| } | ||
|
|
||
| func (as *AsyncSearcher) doSearch(id string, fracProvider fractionAcquirer) { |
There was a problem hiding this comment.
One important thing to notice.
compaction and asyncsearcher can acquire the same fraction set. So, when both processes release fractions, compaction deletes merged fractions -- how result of asynchronous search will be returned in this case?
I understand that QPR is self-sufficient, but QPR contains IDSources which in their turn contain Hint (fraction name) which is used in Fetch stage.
| if f.Info().From > to { | ||
| break | ||
| } |
There was a problem hiding this comment.
I think I've broken the invariant from comment in fractionSnapshot definition:
fractions []frac.Fraction // The actual fractions in chronological orderNow, we build snapshot in this way and it is totally possible that chronological order is broken:
for s := range r.sealed.All() {
all.AddSealed(s)
}
for _, c := range r.compacting {
all.AddSealed(c)
}So I suggest to remove this comment and filter fractions using the same approach as we have here:
func (l List) FilterInRange(from, to seq.MID) List {
res := make(List, 0, len(l))
for _, f := range l {
if f.IsIntersecting(from, to) {
res = append(res, f)
}
}
return res
}
Description
search per N-minute intervals instead of per-fraction to support compaction
Fixes #463
If you have used LLM/AI assistance please provide model name and full prompt: