VecDBLens: A Modular Framework for Diagnosing Vector Databases Retrieval Pipelines

Abstract

Vector databases are increasingly deployed as retrieval pipelines in which representation, indexing, and query-time execution jointly determine accuracy, throughput, and cost. Existing vector database evaluations primarily focus on indexing while treating embeddings as fixed inputs and ignoring the role of query-time execution, making it difficult to decide whether a bottleneck should be addressed by changing the embedding model, the index, or the query-processing stages. To address the above limitation, we introduce \textbf{VecDBLens}, a modular attribution framework that treats the component under diagnosis as an \emph{evaluated object} and all remaining dataset, workload, parameter, and runtime factors as matched \emph{evaluation conditions}. VecDBLens reports stratified marginal effects, paired matched-condition differences, Flat-relative retention, and monotonicity audits, thereby turning vector database benchmarking from leaderboard ranking into a reproducible diagnostic procedure. VecDBLens reveals two key insights. First, retrieval performance is fundamentally upper-bounded by embedding quality, revealing that many apparent indexing failures are therefore representation-side limitations. Second, stronger index parameterization is not always beneficial: overly aggressive configurations can reshape neighborhood topology and cause recall regression. These results challenge the common practice of evaluating vector databases primarily through indexing and show that real-world limitations emerge from stage-wise constraints rather than any single module alone. VecDBLens provides a principled basis for identifying the true performance drivers of vector databases and for guiding their design in practical applications. The code is available at \url{https://anonymous.4open.science/r/VecDBLens-83A4}.