pg-smart-search

Changelog

Release notes and version history for pg-smart-search.

Changelog

All notable changes to this project will be documented in this file.

Current version: 2.0.0 — released 2026-07-27. All entries are verified against the published source. v1.3.1 and earlier are unaudited legacy entries.

[Unreleased]

Nothing yet — [2.0.0] is the latest release.


[2.0.0] - 2026-07-27

Major release. Promotes all fixes and changes that accumulated under [Unreleased] since v1.4.1, adds a real performance measurement suite, and corrects a critical bug in the CLI that generated invalid SQL for every ADVANCED-tier user.

Breaking Changes

  • Minimum Node.js: 18+ (uses AbortSignal, performance.now(), top-level await in CLI scripts).
  • filters API: Scalar equality shorthand ({ column: value }) still works, but the canonical form is now { column: { eq: value } } — see Filter Operators.

Fixed

  • init CLI — critical: invalid RUM SQL for ADVANCED tier. The interactive setup wizard generated CREATE EXTENSION rum and USING RUM (col rum_trgm_ops). The rum_trgm_ops operator class does not exist in stock Postgres or the RUM extension; running the generated SQL would crash every ADVANCED-tier migration. The engine itself never used RUM — AdvancedStrategy has always used a GIN trigram pre-filter with in-memory word_similarity sorting. The generated SQL is now corrected to CREATE EXTENSION pg_trgm + USING GIN (col gin_trgm_ops).
  • In-flight request cancellation: Cancelling one caller's AbortSignal no longer aborts a shared query out from under every other caller deduped onto the same in-flight request — cancellation is now reference-counted per subscriber.
  • skipTotalCount cache-write bug: Result pages were being skipped from the cache whenever skipTotalCount caused total to read as falsy, even when data was non-empty.
  • Zombie Query Prevention race: A caller's own abortSignal wasn't wired into hybridSearch's internal AbortController.
  • cacheHitRate formula: Was computing the wrong ratio; now cacheHits / (cacheHits + cacheMisses).
  • Cursor + relevance-sorted search: Passing cursor to ADVANCED/VECTOR now throws UnsupportedCursorError instead of silently applying OFFSET against a relevance ordering.
  • Gemini API key exposure: Moved from URL query parameter to the x-goog-api-key header.
  • Retry classification: 400/401-class API errors from OpenAI/Gemini are no longer retried.
  • Integration test handles keyset pagination cursor properly: Was passing an empty query string which is rejected by the validator. Fixed to use a dedicated real search term.

Added

  • Real benchmark suite (npm run bench:real): Concurrent worker pools (1 / 8 / 32 / 64 workers), 550+ unique queries extracted from the real corpus, plan cache invalidation before cold rounds, linear-interpolation percentiles (p50 / p90 / p95 / p99), and automatic baseline export to bench/baseline-1.4.1.json.
  • EXPLAIN plan capture (npm run bench:explain): Runs EXPLAIN (ANALYZE, BUFFERS) against LITE / STANDARD / ADVANCED tiers and writes the physical query plans to bench/explain_plans.txt for index tuning.
  • GIN-first seeding (npm run db:seed): Trigram index now defaults to gin_trgm_ops (GIN) instead of GiST. Override with TRIGRAM_INDEX_TYPE=GIST.

Changed

  • Removed SET LOCAL transaction wrapping around fuzzy/ADVANCED-tier queries (was 3 extra round trips: BEGIN / SET LOCAL / COMMIT). The permissive floor for pg_trgm.word_similarity_threshold is now set once per connection via a libpq startup option.
  • Removed SELECT pg_backend_pid() round trip — the PID needed for pg_cancel_backend is read from client.processID directly.
  • LIMIT / OFFSET are bind parameters, not interpolated SQL, so repeated pages reuse the same query plan.
  • Hedged hybrid execution: the trigram branch now starts after a short delay when ftsColumn is configured, instead of always racing both from the start.
  • Typed filter operators: filters accepts { eq, ne, in, gt, gte, lt, lte, between } per key.

Performance (measured, 1 M rows, GIN indexes, PostgreSQL 16)

ModeConcurrencyavgp95Throughput
Cold1 worker5.9 ms9.3 ms170 req/s
Cold8 workers6.6 ms9.5 ms1 208 req/s
Hot1 worker0.057 ms0.126 ms17 377 req/s
Hot8 workers0.344 ms0.552 ms22 649 req/s

At 32+ workers, cold latency degrades severely (~565 ms avg at 32 / ~815 ms at 64) due to the inFlightRequests deduplication lock serialising concurrent identical queries. This bottleneck is tracked for the next performance phase. At realistic per-process concurrency (≤ 8 workers) the engine scales linearly.


[1.4.1] - 2026-06-24

[1.4.1] - 2026-06-24

Fixed

  • Memory Security (OOM Guard): Extended MemoryCacheProvider with LRU eviction and maximum constraints (maxEntries: 10000).
  • Cache Eviction Cleanup: Added background sweep interval daemon in MemoryCacheProvider to free stale RAM assets.
  • Cache Key Clashing: Implemented strict alphanumeric order sorting of query parameters before hashing keys.
  • Chronological Metrics: MetricsCollector's latency buffer now correctly reads back oldest-to-newest after wrapping around.

Changed

  • DRY Refactoring: Extracted redundant filter syntax constructors and rows formatting loops into unified utility maps (result-mapper.ts).
  • Static Keyboards Mapping: Replaced dynamic hash generations inside keyboard translator (convertLayout) with pre-calculated key maps.
  • Latency Buffers: Upgraded collector buffer mechanics to use $O(1)$ ring buffer logic over flat arrays.

[1.4.0] - 2026-06-10

Added

  • Parallel Search Execution: FTS and standard search race via Promise.race, with the loser cancelled.
  • Active Zombie Suppression: the slower branch is aborted on the backend via pg_cancel_backend, not just abandoned client-side.
  • Dedicated cancelPool: Spawned isolated backend connection pool (size 3) reserved strictly for cancel interrupts.
  • Automatic Generated Vectors: Changed manual data indexing workflows to use GENERATED ALWAYS AS ... STORED vectors.

This version previously claimed "GiST/KNN Trigram Indices... database-backed fast sorting" and "cut FTS retrieval cold latencies by 20-30%" via ts_rank over ts_rank_cd. Neither is accurate for the current code: ADVANCED-tier ranking (word_similarity()) is sorted in memory, not index-served (see Search Tiers), and no controlled ts_rank vs ts_rank_cd benchmark exists in this repo to support a specific percentage.

[1.3.1] - 2026-05-22

Added

  • skipTotalCount Parameter: Added config flag to opt out of the COUNT(*) OVER() total-count calculation.

[1.3.0] - 2026-04-30

Added

  • Reliability System: Built-in health() checks, OOM protections (MAX_ROWS).
  • Intelligent Rate-Limiting Queues: Uses p-queue for AI APIs (OpenAI/Gemini) to handle 429 errors gracefully.
  • True Zero-Freeze Networking: Native AbortController propagation ensures timeouts actually kill underlying fetch requests.
  • ISO 9 Compliance: Standardized Cyrillic-to-Latin transliteration for cross-language search reliability.
  • Monitoring & Metrics: Real-time tracking via engine.metrics.getSummary() (cacheHitRate, avgDbLatencyMs, providerErrors).

This version previously listed "Redlock-style Cache Deduplication." No distributed locking exists anywhere in the codebase — RedisCacheProvider is a plain get/set/ delete/clear wrapper. What the engine actually has is process-local in-flight request deduplication (reference-counted), unrelated to Redis or Redlock. See Reliability.

[1.2.1] - 2026-04-30

Fixed

  • Internal bug fixes and stability improvements.

[1.2.0] - 2026-04-30

Added

  • Keyset Pagination: High-performance cursor-based pagination support for massive result sets.
  • CLI Migration Suite: Automated interactive database setup and index migration tool (npm run init).

[1.1.1] - 2026-04-30

Fixed

  • Performance and formatting fixes for SQL queries.

[1.1.0] - 2026-04-30

Added

  • Enterprise Security: Parameterized queries, strict SqlSanitizer whitelist for identifiers, and injection-safe filter keys.

[1.0.0] - 2026-02-23

Added

  • Initial release.
  • Parallel Search Strategy (FTS + Trigram).
  • Smart Hybrid Fallback (Linguistic -> ILIKE -> Trigram).
  • Professional Caching (Memory and Redis providers).
  • Semantic Vector Search integration (OpenAI/Gemini via pgvector).