DataStax partners with OpenSearch to enhance generative AI search

DataStax has joined the OpenSearch Software Foundation and announced integration efforts aimed at supporting generative AI developers.

DataStax, a real-time AI company, has aligned itself with the OpenSearch Software Foundation, a community-driven initiative launched by the Linux Foundation supporting open-source, enterprise-grade search and observability software.

The foundation's mission is to provide resources that enable the long-term sustainability of the OpenSearch project and ecosystem. This objective is consistent with DataStax's ongoing commitment to open-source database software.

The partnership will see DataStax and the OpenSearch project collaborate on a series of integration efforts to serve generative AI developers. Retrieval-augmented generation (RAG) is highlighted as a central design pattern in generative AI, where applications assemble contextual information from various sources.

This information is then processed by a large language model (LLM) to produce relevant responses. Serving such applications requires a blend of data retrieval and storage capabilities. OpenSearch and DataStax are working together to address the diverse needs of developers building these solutions.

DataStax and OpenSearch have previously worked together, announcing an OpenSearch integration with JVector, an open-source embedded vector search engine developed by DataStax co-founder Jonathan Ellis, last year.

This integration provides developers with flexible information retrieval capabilities using applications that are already familiar within many enterprises. The partnership is designed to bridge the gap between single-document question answering and open-domain question answering. It achieves this by combining OpenSearch's keyword search with the dense vector search capabilities of JVector, which is also utilised in Astra DB, Hyper-Converged Database (HCD), and Apache Cassandra.

According to DataStax, "DataStax and OpenSearch: A powerful combination. DataStax has been working with OpenSearch for some time. Last year, they announced an OpenSearch integration with JVector, an open-source, embedded vector search engine developed by DataStax co-founder Jonathan Ellis. The combination provides developers with extremely flexible information retrieval, using applications that many enterprises are already familiar with."

"It bridges the gap between single-document Q&A and open-domain Q&A, providing the ability to reason across diverse documents and texts by combining OpenSearch's keyword search with the dense vector search of JVector, which is the same indexing library used in Astra DB, DataStax Hyper-Converged Database (HCD), and Apache Cassandra."

The integration of JVector and OpenSearch supports hybrid search models.

Vector search in HCD enables developers to leverage proprietary data stored in Cassandra or DataStax Enterprise (DSE) databases for large language models, AI assistants, and real-time generative AI projects without compromising data security. The company claims that JVector technology delivers a tenfold improvement in vector search performance compared with traditional Lucene-based search.

OpenSearch offers an enterprise suite of features that cater to a broad range of requirements, including full-text search, advanced analytics, monitoring tools, and security functionalities.

Discussing the future direction, DataStax stated, "DataStax consider OpenSearch as the future of enterprise search, particularly as they continue to expand their self-managed search offerings."

"They are excited to see where the integrations with OpenSearch lead, and support OpenSearch users in getting the most out of their enterprise data estates."

DataStax has confirmed continued support for a JVector integration for OpenSearch, intending to provide OpenSearch as part of its self-managed platform, Hyper Converged Data Platform (HCDP), and as an integration for its cloud service, Astra.

Enterprises have made long-term investments in search infrastructure.

With OpenSearch's addition, DataStax can offer developers flexible information retrieval using established enterprise applications. As stated: "With the inclusion of OpenSearch, DataStax can provide developers the most flexible information retrieval possible using applications already familiar to many enterprises."

"OpenSearch bridges the gap between single-document Q&A and open-domain Q&A, essentially providing the ability to reason across multiple diverse documents and texts by combining keyword search in OpenSearch with the dense vector search of JVector in Astra and HCDP."

On the topic of generative AI, DataStax added, "For generative AI, relevance is critical, and through this partnership will ensure that your enterprise data estate can act as context for RAG and Gen AI workflows to provide as much data to the context as possible."