Customize JSP using Rest Search in WCS 7

As part of the framework, new JSTL Tag wcf:rest was introduced so that store front can easily invoke these REST Services. (The tag also takes care of local binding / remote binding of the services)

Below code snippet shows an example of how to use this tag to invoke productview/bySearchTerm GET REST service:

  

<wcf:rest var="catalogNavigationView" url="${searchHostNamePath}${searchContextPath}/store/${WCParam.storeId}/productview/bySearchTerm/*">
    <wcf:param name="pageSize" value="${pageSize}" />
    <wcf:param name="pageNumber" value="${pageNumber + 1}" />
    <wcf:param name="profileName" value="IBM_findProductsBySearchTerm_Summary" />    <wcf:param name="searchType" value="${searchType}" />
    <wcf:param name="searchTerm" value="${searchTerm}" />
    <wcf:param name="langId" value="${langId}"/>
    <wcf:param name="responseFormat" value="json"/>  
    <wcf:param name="catalogId" value="${WCParam.catalogId}"/>
    <wcf:param name="categoryId" value="${currentCategoryId}" />
    <wcf:param name="manufacturer" value="${newManufacturer}" />
    <wcf:param name="minPrice" value="${WCParam.minPrice}" />
    <wcf:param name="maxPrice" value="${WCParam.maxPrice}" />
</wcf:rest>

Note:
<wcbase:usebean> - To invoke data beans
<wcf:getData> - To invoke GET data component services
<wcf:rest> - To invoke REST services.

Required parameters to build the final Sol Query like searchTerm, pageNumber, pageSize, categoryId, minPrice, maxPrice etc., are passed to the REST Service using <wcf:param> tag.

These external query parameter names are converted into internal names based on the mapping defined in:

workspace\Search\xml\config\com.ibm.commerce.catalog\wc-component.xml

Sample Mapping:

<_config:valuemapping externalName="SearchControlParameterMapping" internalName="SearchControlParameterMapping">
    <_config:valuemap externalValue="searchType" internalValue="_wcf.search.type" />
    <_config:valuemap externalValue="categoryId" internalValue="_wcf.search.category" />
    <_config:valuemap externalValue="catalogId" internalValue="_wcf.search.catalog" />
    <_config:valuemap externalValue="storeId"       internalValue="_wcf.search.store.online" />
    <_config:valuemap externalValue="facet" internalValue="_wcf.search.facet" />
    <_config:valuemap externalValue="facetLimit" internalValue="_wcf.search.facet.field.limit" />
    <_config:valuemap externalValue="minPrice" internalValue="_wcf.search.price.minimum" />
    <_config:valuemap externalValue="maxPrice" internalValue="_wcf.search.price.maximum" />

    <_config:valuemap externalValue="searchTerm" internalValue="_wcf.search.term" />
    <_config:valuemap externalValue="profileName" internalValue="_wcf.search.profile" /></_config:valuemapping>

A Selection Criteria object will be built using these internal names which will be accessible to all the downstream processors like provider, pre-processor and post-processor

Along with the request, we also pass a special parameter by name "profileName". wc-search.xml (inside workspace\Search\xml\config\com.ibm.commerce.catalog) defines various search profiles which controls how Solr queries are built, the meta-data for the query, the return fields etc., 

A sample profile will look like:

<_config:profile name="IBM_findProductsBySearchTerm_Summary" indexName="CatalogEntry">
    <_config:query>

        <_config:param name="maxRows" value="500" />       
        <_config:provider classname="com.ibm.solr.SolrRESTSearchBasedMerchandisingExpressionProvider"/>
        <_config:provider classname="com.ibm.xxx.solr.SolrRESTSearchTermAssociationExpressionProvider"/>

        <_config:preprocessor classname="com.ibm.xxx.solr.SolrSearchDebugQueryPreprocessor"/>
        <_config:preprocessor classname="com.ibm.xxx.solr.SolrSearchResultFieldQueryPreprocessor"/>
        <_config:field name="name"/>
        <_config:field name="defaultSearch"/>
        <_config:field name="categoryname"/>
        <_config:field name="shortDescription"/>
        <_config:postprocessor classname="com.ibm.xxx.solr.SolrRESTSearchPreviewQueryPostprocessor" />
        <_config:postprocessor classname="com.ibm.solr.SolrRESTSearchExperimentQueryPostprocessor" />
    </_config:query>     
    <_config:result>
        <_config:field name="catentry_id"/>
        <_config:field name="storeent_id"/>
        <_config:field name="buyable"/>
          <_config:field name="mfName_ntk"/>
        <_config:field name="catenttype_id_ntk_cs"/>
        <_config:field name="price_*"/>
        <_config:field name="listprice_*"/>
        <_config:field name="parentCatgroup_id_facet"/>
        <_config:field name="childCatentry_id"/>
    </_config:result>
</_config:profile>

1. <_config:query> defines a set of providers and pre-processor. Each of these providers and pre-processor will contribute a part of Solr query. SolrRESTSearchExpressionProcessor will build the final query aggregating all the part queries built by these providers and pre-processor and executes it against the Solr index.

2. <_config:param> is used to add name/value parameters to final Solr Query directly and it can be used to control how query is executed or fine tune the query further.

3. <confg:provider> and <config:preprocessor> are used to define the providers and pre-processor used while building the query.

4. <config:field> defines the index field names against which search should be performed. (qf fields)

5. <config:result> defines the fields to be returned in the result set (fl fields)

6. You can also use <_config:sort>, <_config:highlight>, <_config:spellCheck> and other configurations to fine tune the Solr Query.

Search framework will retrieve the profileName from query parameters and identifies the suitable profile defined in wc-search.xml. It then invokes all the providers and pre-processors + default providers and pre-processors defined for the profile, to build the Solr Query which will then be executed against the Solr Index. Once Solr returns back the result set, the search framework invokes the post-processors defined against the profile in a sequential order. Each of these post-processors will modify the result set returned by the Solr. ( Add / modify / remove data from the result set ).
Finally the result set will be converted into required responseFormat ( JSON / XML ) and returned back to JSP.

Customizing Solr Services

WCS - Solr framework provides various touch points to customize the existing services or create new services. Some of the customization touch points which we will explore in detail are:

1. JSP

2. wc-search.xml

3. Provider

4. Pre-Processor

5. Post-Processor

Customization using JSP
The first step of customization is JSP, where the actual Search REST request is made. You can pass various parameters to control how the search is performed. You can control the amount of data returned by Solr Server by using appropriate profileName. (Use xxx_Summary profiles when you want to retrieve lesser data).
Let us look at how some advanced customization scenarios can be achieved.

Scenario 1: By default MinimumMatch property is set to "1" in wc-search.xml. Which means, when a shopper searches for "red apple" all products which either contains a word "red" or "apple" will be returned. So shopper will see - apples, red dress, red potatoes, red handbags, red ginsing etc., in the result set.
Let us assume that you want to control this parameter value from JSP while doing the search and set it to different values based on some logic.
SolrSearchEDismaxQueryPreProcessor is the PreProcessor class which looks for
 "_wcf.search.edismax.mm" parameter in the selection criteria object and uses the value to set "mm" parameter in the final Solr Query executed against Search Server. So you can pass this parameter from JSP using <wcf:param> tag.

<wcf:rest var="catalogNavigationView" url="${searchHostNamePath}${searchContextPath}/store/${storeId}/productview/bySearchTerm">
    <wcf:param name="pageSize" value="${pageSize}" />
    <wcf:param name="searchTerm" value="${term}" />
    xxx
    xxx
    <wcf:param name="_wcf.search.edismax.mm" value="2"/>
</wcf:rest>

With this param set in request, when you search for "red apple", you will see that results contains only products which matches both the words - red and apple.

You can always update wc-component.xml to map this internal name "_wcf.search.edismax.mm" to a user friendly external name like "mm" and start using "mm" in the JSP file.

Scenario 2: Let us assume that you have defined an attribute "Clearance" and made it searchable and facetable using CMC tooling. Now when someone searches for a product, you would like to take advantage of this attribute and return only those products which has this attribute.

To achieve this first identify the SearchFieldName ( index field name) for the "Clearance" attribute using below query:

select SRCHFIELDNAME from ATTRDICTSRCHCONF where attr_id in (select attr_id from attr where identifier = 'Clearance' and storeent_id = <storeId>);

Lets assume the above query returned SearchFieldName as: ads_f10101.

Search framework provides SolrRESTSearchByCustomExpressionProvider which allows you to add optional query ( OR constraint ), mandatory query ( AND constraint ) and Filter query ( fq ) part to the final Solr query. This custom provider looks for query parameter with name "_wcf.search.expr", "_wcf.search.mandatory.expr" and "_wcf.search.filter.expr". You can directly use these param names in JSP or create a mapping in wc-component.xml as below:

            <_config:valuemap externalValue="minMatch" internalValue="_wcf.search.edismax.mm" />
            <_config:valuemap externalValue="optionalQuery" internalValue="_wcf.search.expr" />
            <_config:valuemap externalValue="mandatoryQuery" internalValue="_wcf.search.mandatory.expr" />
            <_config:valuemap externalValue="filterQuery" internalValue="_wcf.search.filter.expr" />

With this mapping in place you can pass these additional parameters to <wcf:rest> tag in JSP.
// Return products which matches SearchTerm AND which has this attribute set to Clearance.

<wcf:param name="mandatoryQuery" value="ads_f10101_ntk_cs:(\"Clerance\")"/>

OR

// Return products which matches search term OR the products which has this attribute set to Cleranace

<wcf:param name="optionalQuery" value="ads_f10101_ntk_cs:(\"Clerance\")"/>

OR

// Return products which matches search term, filter out the results to return products with attribute Clerarance.

<wcf:param name="filterQuery" value="ads_f10101_ntk_cs:(\"Clerance\")"/>

FilterQuery is similar to ManatoryQuery, but filter Query performs the filter on the returned result set and hence maintains the relevance score better.

So by making use of these additional parameters and appropriate search profiles you can control the modify the solr query to suit your needs.

Customization using wc-search.xml
If needed you can define your own custom profiles and use it in JSP pages while making REST Service calls. 

Create custom profiles in wc-search.xml ( in workspace\Search\xml\config\com.ibm.commerce.catalog\ext folder ). 

You can use existing providers, pre-processors, post-processors or create your own custom providers / pre-processors / post-processors and use it in the search profile. You can make use of various other aspects of wc-search.xml like <_config:field>, <_config:param>, <_config:sort> to customize the solr query.

Customization using custom Providers
Sometimes custom scenarios may be complex and you may not be able achieve required features just by using the existing profiles or providers. Passing in mandatoryQuery, optionalQuery or filterQuery may not be sufficient or relevant for some customization scenarios.
If the customization calls for modifying the Solr Query to add additional constraints or boost the result set o modify the query based on certain conditions, then you can always create a custom provider. 

Let us look at the following REST service which retrieves product data byId.
http://localhost/search/resources/store/10001/productview/byIds?id=10034&id=10014&id=10906&profileName=IBM_findProductByIds_Summary

When you inspect the result set from this service, you will notice that the products are not in the same order in which the product ids were passed (10034, 10014, 10906). The order is completely random (probably based on the order in which the data was indexed).

Now if you would like to maintain the order of products, then one option is to use the boost query to boost the products as below, so that the sequence order is maintained and id1 is returned first and id3 is returned at the end.

bq=catentry_id:"id1"^10.0 catentry_id:"id2"^9.0 catentry_id:"id3"^8.0

Lets create a provider which will add this query part to the Selection Criteria object, so that it can be consumed by the final ExpressionProcessor while building the Solr Query.

Above listing shows a sample Provider Class which extends from AbstractSolrSearchExpressionProvider and implements SearchExpressionProvider interface.

It overrides invoke method and execute some custom logic to build the boost query string. getControlParameterValue gives you access to the input query parameters. Notice that we are using SearchServiceConstants.CTRL_PARAM_SEARCH_TERM as the key name while retrieving the product ids.

Once the boost search query is built, it is added back to the Selection Criteria object using addControlParameterValue method. The key used in this case is SearchServiceConstants.CTRL_PARAM_SEARCH_INTERNAL_BOOST_QUERY. This is the predefined name which is used by the Expression Processors to build the "bq" Solr Query.

SOLR framework in WCS

Starting with FEP-7, WCS provided REST Services to interact with Solr Server directly without going through BOD / Component Services layer. This post explores the new search framework in detail and touches upon various customization points provided by the framework to implement customer requirements.

The best way to start learning WCS Solr framework is to first get yourself familiarized with basics of Solr. Solr mainly consists of 2 parts -

1. Indexing the data.

2. Querying the index to get the desired results.

In this article we will look at how to query the index without worrying too much about the steps involved in building the index. So it's better to get familiar with Solr edisMax (parser used by WCS) query syntax. At the minimum you should understand the usage of: q, fq, fl, qf, facet, bq parameters. (You can install Solr Admin UI which provides a intuitive user interface to query the index and learn more about the query syntax.

Once you are comfortable with Solr basic query syntax you are good to start with WCS - Solr framework. Let us consider a simple example of searching for keyword 'Apple'.

How do you write a DB query to select products whose name or short description contains keyword 'Apple' ?

Lets start with a basic query :
Select * from catentry where catentry_id in (select catentry_id fromcatentdesc where language_id = -1 and ( name like '%Apple%' orshortdescription like '%Apple%'))

Then add store constraint:
and member_id = (select member_id from storeent where storeent_id = 10001))


To return only product beans and ignore item beans, add below constraint:
and catenttype_id = ‘ProductBean’

To return only published catalog entries, add below constraint:
and catentdesc.published = 1

To return products belonging to current sales catalog, the constraint will be:
join catgpenrel where catentry_id in ( ) and catalog_id = ( )

What happens if we want to return products within certain price range, belonging to certain category and want the result set to be sorted by price or relevance ? The query starts getting more and more complicated and difficult to manage.

Let us see how the SOLR query looks in this case.

You will start with a simple query using 'q' parameter:

q = "Apple" 

And then tell Solr to search in name and short description fields. Also products where searchTerm appears in name are more relevant than products where search term appears in shortDescription field.

qf=name^10.0 shortDescription^5.0

Add filter queries to return only published products ( ignore item Beans ) belonging to current catalog and store.

fq=catalog_id:"10052"

fq=storeent_id:("10001")
fq=published:1
fq=-catenttype_id_ntk_cs:ItemBean

Return only few selected fields.
fl=catentry_id,storeent_id,buyable,partNumber_ntk

Also return facets based on category and brand name 
facet=true
facet.field=parentCatgroup_id_search
facet.field=mfName_ntk_cs 
f.mfName_ntk_cs.facet.limit=21 
f.mfName_ntk_cs.facet.mincount=1
f.mfName_ntk_cs.facet.sort=count

Add spell check 
spellcheck=true
spellcheck.count=5 
spellcheck.onlyMorePopular=false
spellcheck.accuracy=0.3
spellcheck.alternativeTermCount=5 
spellcheck.maxResultsForSuggest=3
spellcheck.q=apple

Add some meta data around pagination, debugging: 
start=0
rows=50
timeAllowed=15000 
defType=edismax
echoHandler=true
echoParams=all
degug = true

As you can see Solr provides you a simple yet powerful query syntax to query the index and organize the results the way you want. You can set the fields to query, set the fields to retrieve, boost the results based on field names, add sorting, spell check, highlighting features to result set, get the stats about the result set, control the facets, filter query based on various conditions etc., quite easily.

The final Solr query will look like something below:

fl=catentry_id,partNumber_ntk,name,shortDescription,thumbnail,storeent_id,childCatentry_id,catentry_id,partNumber_ntk,name,shortDescription,thumbnail,storeent_id,childCatentry_id&start=0&rows=4&timeAllowed=15000&defType=edismax&qf=name^10.0 defaultSearch^1.0 categoryname^100.0 shortDescription^5.0 name_suggest^1.0 shortDesc_suggest^1.0&pf=name^10.0 defaultSearch^1.0 categoryname^100.0 shortDescription^5.0 name_suggest^1.0 shortDesc_suggest^1.0&ps=100&mm=1&tie=0.1&tie=0.1&wt=json&json.nl=map&q="apple"&fq=catalog_id:"10052"&fq=storeent_id:("10001")&fq=published:1&fq=-(catenttype_id_ntk_cs:ItemBean AND parentCatentry_id:[* TO *])

Once the final Solr Query is built, it will be executed by SolrRESTSearchExpressionProcessor class. But who is responsible for building this Solr Query ? The entire query is NOT built by a single java class. Instead the query is split into separate logical parts like q, fq, fl, qf etc., and each part of the query is built by a separate java class known as Providers and PreProcessors. The parts of query are then assembled by SolrRESTSearchExpressionProcessor and executed against Solr Server.