Federated Search: The Demo

Federated Search: The Demo
brandt
Mon, 01/07/2019 - 14:57
Ken Rickard and Avi Schwab
Jan 7, 2019

See Palantir’s federated search application in action.

We recently published a blog post introducing our solution to Google Search Appliance being discontinued—an open source application we built and named Federated Search. If you haven’t already, we recommend checking out that first blog post to get the basics on how we built the application and why. Read on here to learn how you can see for yourself what the application does.

Search API Federated Solr is a complex application, and the best way to understand what's going on is to see it in action! Since the application requires a Solr instance in addition to a number of Drupal modules, we're not able to use Simplytest.me for demos. Instead, we've bundled all of the pieces together with Palantir's open source dev tools — the-vagrant and the-build — for a seamless demo experience that runs in a local virtual machine (VM) running on Vagrant. Head to GitHub to review the requirements, and then clone the repo and get started.

Setting up the environment

The-vagrant is a customizable vagrant environment that can be built into a project from scratch or easily retrofit an existing project (such as a new support client). On first setup, a handy install wizard takes users through a configuration process to choose hostnames, enable optional services like Solr, and enable further customization through Ansible tasks. The-vagrant is capable of handling single site, multi-site, or multiple-site (many docroot) setups in a single box, so it was a perfect match for our Federated Search environment.

The-build is a set of reusable phing targets for building Drupal projects. Once our VM is up and running, we use a standard set of these tasks to automate a number of complex tasks, such as:

  • Copying settings and services files into Drupal sites directories
  • Installing Drupal using an install profile and any existing config
  • Running post-install tasks like migrations
  • Running test suites
  • Importing databases from hosting environments
  • Deploying code to hosting environments

We have a shared set of phing targets that provide the foundation for many of these tasks, and each project extends them to meet their specific needs.

Building the demo

The Federated Search Demo repo builds a simulated multiple site environment, with a Solr server to boot, in the comfort of your own VM. Our demo site is expressly designed for both testing and development.

Because the application supports multisite, Domain Access, and standalone sites, we wanted to be able to demo (and develop for) all possible scenarios. To this end, the demo contains four docroots: Drupal 7 standalone, Drupal 7 Domain Access (coming soon), Drupal 8 standalone, Drupal 8 Domain Access. The D8 sites use the amazing core Umami profile to demo with real content, while the D7 site uses Devel Generate for some lorem ipsum-based content.

As of this writing, Domain Access is supported in the Drupal 7 module code, but not installed in the demo profile. The reverse is true for Drupal 8, and making the Drupal 8 version of Federated Search support Domain Access is under active development. We literally had to build the VM in order to finish those features!

There are a lot of dependencies involved, so let’s go to an application diagram:

There’s a lot going on there, but we suggest grabbing the repo and seeing for yourself.

What to expect

Once you clone the demo repo, there are full instructions on getting the VM and Drupal up and running. After installing all of the sites, you can start by visiting http://d8.fs-demo.local and use the search box to test a search (maybe try mushrooms, yum). You should see the React-powered search page with your results and a number of filters on the left side which you can experiment with.

Once you see the search results, you can dig in to how it works. In the Search App Settings (found at admin/config/search-api-federated-solr/search-app/settings) you can control a number of pieces of how the search page is displayed including it’s route and title. We set the page to default to ‘/search-app’ so as not to conflict with the default core configuration. Any changes made on this page should clear the cache for the search application and immediately be reflected on refresh.

Next, you may want to see how data is indexed. The search index field config page (found at admin/config/search/search-api/index/federated_search_index/fields) will show a list of all of the mapped fields the site is sending to the index. Clicking on Edit will show you the details of each, showing each bundle in the site and how it’s being sent to the index. The Edit modal includes a token picker, showing the true power of this tool—the ability to use tokens or text at the bundle level to send data to our index.

From this screen, try editing the config for a field, adding a token or changing a format. Once you do that, Search API will prompt you to re-index your data.

You can do so, then refresh the search results to see the changes. You might also want to inspect the raw data being sent to Solr. To do that, visit the Solr dashboard (at http://federated-search-demo.local:8983/solr/#/drupal8/query) and execute the default query. There you can see all of the fields being sent to the index.

Coming back to the search page, inspecting the results with the React Dev Tools will help you understand how the application is handling data. Once you install the browser extension, you can inspect the app, view the React components, see props being passed through the stack, and more. For an even deeper dive into the React application, you can clone that project and build it locally.

Contributing

In addition to providing a full demo environment, this repo also serves as a development environment for Search API Federated Solr and Search API Field Map. While those modules are installed by composer, the repo also links them into the ‘/src/’ directory for easy access. From there, you can add a GitHub remote or create patches for Drupal.org.

Issues for the demo can be raised on GitHub, and issues for the modules can be on either GitHub or Drupal.org. Be sure to read the handbook on Drupal.org for even more detail on how the system works.

Learn more about Federated Search in this presentation from Decoupled Days (or just view the slides).

Development
Drupal
Open Source

Original Article: