Making Each Search Rewarding: How Ibotta Remodeled Provide Discovery With Databricks

At Ibotta, our mission is to Make Each Buy Rewarding. Serving to our customers (whom we name Savers) discover and activate related provides via our direct-to-consumer (D2C) app, browser extension, and web site is a crucial a part of this mission. Our D2C platform helps thousands and thousands of customers earn cashback from their on a regular basis purchases—whether or not they’re unlocking grocery offers, incomes bonus rewards, or planning their subsequent journey. By the Ibotta Efficiency Community (IPN), we additionally energy white-label cashback packages for among the greatest names in retail, together with Walmart and Greenback Basic, serving to over 2,600 manufacturers attain greater than 200 million customers with digital provides throughout associate ecosystems.

Behind the scenes, our Knowledge and Machine Studying groups energy crucial experiences like fraud detection, supply advice engines, and search relevance to make the Saver journey customized and safe. As we proceed to scale, we want data-driven, clever programs that help each interplay at each touchpoint.

Throughout D2C and the IPN, search performs a pivotal function in engagement and must hold tempo with our enterprise scale, evolving supply content material, and altering Saver expectations.

On this publish we’ll stroll via how we considerably refined our D2C search expertise: from an formidable hackathon undertaking to a sturdy manufacturing characteristic now benefiting thousands and thousands of Savers.

We believed our search might higher sustain with our Savers

Consumer search conduct has advanced from easy key phrases to incorporating pure language, misspellings, and conversational phrases. Trendy search programs should bridge the hole between what customers sort and what they really imply, deciphering context and relationships to ship related outcomes even when question phrases don’t precisely match the content material.

At Ibotta, our authentic homegrown search system, at occasions, struggled to maintain tempo with the evolving expectations of our Savers and we acknowledged a possibility to refine it.

The important thing areas for alternative we noticed included:

• Enhancing semantic relevance: Specializing in understanding Saver intent over precise key phrase matches to attach them with the proper provides.
• Enhancing understanding: Decoding the complete nuance and context of person queries to supply extra complete and really related outcomes.
• Rising flexibility: Extra quickly integrating new supply sorts and adapting to altering Saver search patterns to maintain our discovery expertise rewarding.
• Boosting discoverability: We wished extra strong instruments to make sure particular varieties of provides or key promotions had been persistently seen throughout a wide selection of related search queries.
• Accelerating iteration and optimization: Enabling sooner, impactful enhancements to the search expertise via real-time changes and efficiency tuning.

We believed the system might higher hold tempo with altering supply content material, search behaviors, and evolving Saver expectations. We noticed alternatives to extend the worth for each our Savers and our model companions.

From hackathon to manufacturing: reimagining search with Databricks

Addressing the restrictions of our legacy search system required a targeted effort. This initiative gained important momentum throughout an inside hackathon the place a cross-functional group, together with members from Knowledge, Engineering, Advertising Analytics, and Machine Studying, got here along with the thought to construct a contemporary, different search system utilizing Databricks Vector Search, which some members had realized about on the Databricks Knowledge + AI Summit.

In simply three days, our group developed a working proof-of-concept that delivered semantically related search outcomes. Right here’s how we did it:

1. Collected supply content material from a number of sources in our Databricks catalog
2. Created a Vector Search endpoint and index with the Python SDK
3. Used pay-per-token embedding endpoints with 4 completely different fashions (BGE massive, GTE massive, GTE small, a multilingual open-source mannequin, and a Spanish-language-specific mannequin)
4. Linked all the pieces to our web site for a reside demo

The hackathon undertaking received first place, generated sturdy inside buy-in and momentum to transition the prototype right into a manufacturing system. Over the course of some months, and with shut collaboration from the Databricks group, we remodeled our prototype into a sturdy full-fledged manufacturing search system.

From proof of idea to manufacturing

Shifting the hackathon proof-of-concept to a production-ready system required cautious iteration and testing. This part was crucial not just for technical integration and efficiency tuning, but in addition for evaluating whether or not our anticipated system enhancements would translate into constructive adjustments in Saver conduct and engagement. Given search’s important function and deep integration throughout inside programs, we opted for the next strategy: we modified a key inside service that known as our authentic search system, changing these calls with requests directed to the Databricks Vector Search endpoint, whereas constructing in strong, swish fallbacks to the legacy system.

Most of our early work targeted on understanding:

Within the first month, we ran a take a look at with a small share of our Savers which didn’t obtain the engagement outcomes we had hoped for. Engagement decreased, significantly amongst our most energetic Savers, indicated by a drop in clicks, unlocks (when Savers categorical curiosity in a proposal), and activations.

Nonetheless, the Vector Search resolution supplied important advantages together with:

• Sooner response occasions
• A less complicated psychological mannequin
• Better flexibility in how we listed information
• New skills to regulate thresholds and alter embedding textual content

Happy with the system’s underlying technical efficiency, we noticed its higher flexibility as the important thing benefit wanted to iteratively enhance search outcome high quality and overcome the disappointing engagement outcomes.

Constructing a semantic analysis framework

Following our preliminary take a look at outcomes, relying solely on A/B testing for search iterations was clearly inefficient and impractical. The variety of variables influencing search high quality was immense—together with embedding fashions, textual content mixtures, hybrid search settings, Approximate Nearest Neighbors (ANN) thresholds, reranking choices, and lots of extra.

To navigate this complexity and speed up our progress, we determined to ascertain a sturdy analysis framework. This framework wanted to be uniquely tailor-made to our particular enterprise wants and able to predicting real-world person engagement from offline efficiency metrics.

Our framework was designed round an artificial analysis surroundings that tracked over 50 on-line and offline metrics. Offline, we monitored customary data retrieval metrics like Imply Reciprocal Rank (MRR) and precision@okay to measure relevance. Crucially, this was paired with on-line real-world engagement alerts reminiscent of supply unlocks and click-through charges. A key resolution was implementing an LLM-as-a-judge. This allowed us to label information and assign high quality scores to each on-line query-result pairs and offline outputs. This strategy proved to be crucial for fast iteration primarily based on dependable metrics and gathering the labeled information obligatory for future mannequin fine-tuning.

Alongside the way in which, we leaned into a number of components of the Databricks Knowledge Intelligence Platform, together with:

• Mosaic AI Vector Search: Used to energy high-precision, semantically wealthy search outcomes for analysis assessments.
• MLflow patterns and LLM-as-a-judge: Offered the patterns to judge mannequin outputs and implement our information labeling course of.
• Mannequin Serving Endpoints: Environment friendly deployment of fashions straight from our catalog.
• AI Gateway: To safe and govern our entry to 3rd social gathering fashions by way of API.
• Unity Catalog: Ensured the group, administration, and governance of all datasets used throughout the analysis framework.

This strong framework dramatically elevated our iteration pace and confidence. We carried out over 30 distinct iterations, systematically testing main variable adjustments in our Vector Search resolution, together with:

• Totally different embedding fashions (foundational, open-weights, and third social gathering by way of API)
• Numerous textual content mixtures to feed into the fashions
• Totally different question modes (ANN vs Hybrid)
• Testing completely different columns for hybrid textual content search
• Adjusting thresholds for vector similarity
• Experimenting with separate indexes for various supply sorts

The analysis framework remodeled our growth course of, permitting us to make data-driven selections quickly and validate potential enhancements with excessive confidence earlier than exposing them to customers.

The seek for the very best off-the-shelf mannequin

Following the preliminary broad take a look at that confirmed disappointing engagement outcomes, we shifted our focus to exploring the efficiency of particular fashions recognized as promising throughout our offline analysis. We chosen two third-party embedding fashions for manufacturing testing, accessed securely via AI Gateway. We carried out short-term, iterative assessments in manufacturing (lasting a couple of days) with these fashions.

Happy with the preliminary outcomes, we proceeded to run an extended, extra complete manufacturing take a look at evaluating our main third-party mannequin and its optimized configuration towards the legacy system. This take a look at yielded combined outcomes. Whereas we noticed total enhancements in engagement metrics and efficiently eradicated the detrimental impacts seen beforehand, these good points had been modest—principally single-digit share will increase. These incremental advantages weren’t compelling sufficient to totally justify an entire substitute of our present search expertise.

Extra troubling, nonetheless, was the perception gained from our granular evaluation: whereas efficiency considerably improved for sure search queries, others noticed worse outcomes in comparison with our legacy resolution. This inconsistency offered a major architectural dilemma. We confronted the unappealing alternative of implementing a posh traffic-splitting system to route queries primarily based on predicted efficiency—an strategy that might require sustaining two distinct search experiences and introduce a brand new, complicated layer of rule-based routing administration—or accepting the restrictions.

This was a crucial juncture. Whereas we had seen sufficient promise to maintain going, we would have liked extra important enhancements to justify totally changing our homegrown search system. This led us to start fine-tuning.

High-quality-tuning: customizing mannequin conduct

Whereas the third-party embedding fashions explored beforehand confirmed technical promise and modest enhancements in engagement, additionally they offered crucial limitations that had been unacceptable for a long-term resolution at Ibotta. These included:

1. Incapability to coach embedding fashions on our proprietary supply catalog
2. Problem evolving fashions alongside enterprise and content material adjustments
3. Uncertainty concerning long-term API availability from exterior suppliers
4. The necessity to set up and handle new exterior enterprise relationships
5. Community calls to those suppliers weren’t as performant as self-hosted fashions

The clear path ahead was to fine-tune a mannequin particularly tailor-made to Ibotta’s information and the wants of our Savers. This was made attainable due to the thousands and thousands of labeled search interactions we had amassed from actual customers by way of our LLM-as-a-judge course of inside our customized analysis framework. This high-quality manufacturing information turned our coaching gold.

We then launched into a methodical fine-tuning course of, leveraging our offline analysis framework extensively.

Key components had been:

• Infrastructure: We used AI Runtime with A10s in a serverless surroundings, and Databricks ML Runtime for stylish hyperparameter sweeping.
• Mannequin choice: We chosen a BGE household mannequin over GTE, which demonstrated stronger efficiency in our offline evaluations and proved extra environment friendly to coach.
• Dataset engineering: We constructed quite a few coaching datasets, together with producing artificial coaching information, in the end selecting:
  • One constructive outcome (a verified good match from actual searches)
  • ~10 detrimental examples per constructive, combining:
    • 3-4 “onerous negatives” (LLM labeled, human-verified inappropriate matches)
    • “In-batch negatives” (sampling of outcomes from unrelated search phrases)
• Hyperparameter optimization: We systematically swept issues like studying fee, batch measurement, period, and detrimental sampling methods to search out optimum configurations.

After quite a few iterations and evaluations throughout the framework, our top-performing fine-tuned mannequin beat our greatest third-party baseline by 20% in artificial analysis. These compelling offline outcomes offered the arrogance wanted to speed up our subsequent manufacturing take a look at.

Search that drives outcomes—and income

The technical rigor and iterative course of paid off. We engineered a search resolution particularly optimized for Ibotta’s distinctive supply catalog and person conduct patterns, delivering outcomes that exceeded our expectations and supplied the pliability wanted to evolve alongside our enterprise. Based mostly on these sturdy outcomes, we accelerated migration onto Databricks Vector Search as the inspiration for our manufacturing search system.

In our last manufacturing take a look at, utilizing our personal fine-tuned embedding mannequin, we noticed the next enhancements:

• 14.8% extra supply unlocks in search.
  This measures customers choosing provides from search outcomes, indicating improved outcome high quality and relevance. Extra unlocks are a number one indicator of downstream redemptions and income.
• 6% improve in engaged customers.
  This exhibits a higher share of customers discovering worth and taking significant motion throughout the search expertise, contributing to improved conversion, retention and lifelong worth.
• 15% improve in engagement on bonuses.
  This displays improved surfacing of high-value, brand-sponsored content material, translating straight to raised efficiency and ROI for our model and retail companions.
• 72.6% lower in searches with zero outcomes.
  The numerous discount means fewer irritating experiences and a significant enchancment in semantic search protection.
• 60.9% fewer customers encountering searches returning no outcomes.
  This highlights the breadth of affect, displaying that a big portion of our person base is now persistently discovering outcomes, bettering the expertise throughout the board.

Past user-facing good points, the brand new system delivered on efficiency. We noticed 60% decrease latency to our search system, attributable to Vector Search question efficiency and the fine-tuned mannequin’s decrease overhead.

Leveraging the pliability of this new basis, we additionally constructed highly effective enhancements like Question Transformation (enriching imprecise queries) and Multi-Search (fanning out generic phrases). The mix of a extremely related core mannequin, improved system efficiency, and clever question enhancements has resulted in a search expertise that’s smarter, sooner, and in the end extra rewarding

Question Transformation

One problem with embedding fashions is their restricted understanding of area of interest key phrases, reminiscent of rising manufacturers. To deal with this we constructed a question transformation layer that dynamically enriches search phrases in-flight primarily based on predefined guidelines.

For instance, if a person searches for an rising yogurt model the embedding mannequin won’t acknowledge, we will rework the question so as to add “Greek yogurt” alongside the model identify earlier than sending it to Vector Search. This supplies the embedding mannequin with obligatory product context whereas preserving the unique textual content for hybrid search.

This functionality additionally works hand-in-hand with our fine-tuning course of. Profitable transformations can be utilized to generate coaching information; as an example, together with the unique model identify as a question and the related yogurt merchandise as constructive ends in a future coaching run helps the mannequin study these particular associations.

Multi-Search

For broad, generic searches like “child,” Vector Search may initially return a restricted variety of candidates, probably filtered down additional by focusing on and price range administration. To deal with this and improve outcome range, we constructed a multi-search functionality that followers out a single search time period into a number of associated searches.

As a substitute of simply looking for “child,” our system mechanically runs parallel searches for phrases like “child meals,” “child clothes,” “child medication,” “child diapers,” and so forth. Due to the low latency of Vector Search, we will execute a number of searches in parallel with out growing the general response time to the person. This supplies a wider and extra various set of related outcomes for wide-ranging class searches.

Classes Discovered

Following the profitable last manufacturing take a look at and the complete rollout of Databricks Vector Search to our person base – delivering constructive engagement outcomes, elevated flexibility, and highly effective search instruments like Question Transformation and Multi-Search – this undertaking journey yielded a number of helpful classes:

1. Begin with a proof of idea: The preliminary hackathon strategy allowed us to shortly validate the core idea with minimal upfront funding.
2. Measure what issues to you: Our tailor-made 50-metric analysis framework was essential; it gave us confidence that enhancements noticed offline would translate into enterprise affect, enabling us to keep away from repeated reside testing till options had been really promising.
3. Do not bounce straight to fine-tuning: We realized the worth of completely evaluating off-the-shelf fashions and exhausting these choices earlier than investing within the higher effort required for fine-tuning.
4. Acquire information early: Beginning to label information from our second experiment ensured a wealthy, proprietary dataset was prepared when fine-tuning turned obligatory.
5. Collaboration accelerates progress: Shut partnership with Databricks engineers and researchers, sharing insights on Vector Search, embedding fashions, LLM-as-a-judge patterns, and fine-tuning approaches, considerably accelerated our progress.
6. Acknowledge cumulative affect: Every particular person optimization, even seemingly minor, contributed considerably to the general transformation of our search expertise.

What’s subsequent

With our fine-tuned embedding mannequin now reside throughout all direct-to-consumer (D2C) channels, we subsequent plan to discover scaling this resolution to the Ibotta Efficiency Community (IPN). This might deliver improved supply discovery to thousands and thousands extra customers throughout our writer community. As we proceed to gather labeled information and refine our fashions via Databricks, we imagine we’re effectively positioned to evolve the search expertise alongside the wants of our companions and the expectations of their prospects.

This journey from a hackathon undertaking to a manufacturing system proved that reimagining a core product expertise quickly is achievable with the proper instruments and help. Databricks was instrumental in serving to us transfer quick, fine-tune successfully, and in the end, make each search extra rewarding for our Savers.