Performing a Query

Datastreamer API results strongly depend on the query passed in the query parameter. A query is usually broken up into terms and operators. Here is how terms and operators can be used to retrieve the required result from the API.

Term & Phrase

• A Single Term is a single word such as "test" or "hello" (quotes not required).
• A Phrase is a group of words surrounded by double quotes, such as "hello john".
• Multiple terms can be combined together with Boolean operators to form a more complex query.

Operator

Operators allow terms to be combined through logic operators. Search Query supports the following operators:

AND / &&: The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets.

+ : The + or required operator requires that the term after the + symbol exist somewhere in the field of a single document.

OR / || : The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exists in a document.

NOT / ! : The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT.

- : The - or prohibit operator excludes documents that contain the term after the - symbol.

📘

Escape Special Characters

The current list of special characters is: + - && || ! ( ) { } [ ] ^ " ~ * ? : \

Term Modifiers

Apache Lucene supports modifying query terms to provide a wide range of searching options. Here are the search options available to consumers to design the request in order to retrieve the efficient results.

Wildcard Search

There is a support for single and multiple character wildcard searches within single terms. To perform a single character wildcard search developer can use the ? symbol. To perform a multiple character wildcard search the * symbol can be used.

The single character wildcard search looks for terms that match that with the single character replaced. For example, to search for "text" or "test" you can use the search:

te?t

Multiple character wildcard searches looks for 0 or more characters. For example, to search for test, tests or tester, you can use the search:

test*

You can also use the wildcard searches in the middle of a term.

te*t

🚧

Illegal Usage

You cannot use a * or ? symbol as the first character of a search.

Fuzzy Search

Our API supports fuzzy searches which are done by a tilde ~. For example to search for a term similar in spelling to "roam" use the fuzzy search:

roam~

This search will find terms like foam and roams.

Proximity Search

The API supports finding words within a specific distance of each other. To do a proximity search, use the tilde ~ symbol at the end of a Phrase. For example, to search for "apache" and "jakarta" within 10 words of each other in a document, use the search below. See Proximity search for more details on how proximity search works.

"jakarta apache"~10

Date Range Search

Date Range Queries allow one to filter by date range on any date/time field. In this example, the content will be filtered to content with a published date/time between and including November 1st, 2021, and November 7th.

"query": "content.published:[2021-11-01 TO 2021-11-07]",

📘

Date fields

If you set no dates, a search covers everything in the index. To limit the window, set query_start_date and query_end_date (ISO 8601) inside query, or add a document date (doc_date) range to the query string as described in "Historical Searching".

Square Brackets [] means range search will be performed inclusive of the values given within the brackets. Curly Bracket {} will search the values exclusive of the parameters given within the brackets.

You can also add a timeframe in addition to the date.

"query": "content.published:[2021-12-01T00:00:00 TO 2021-12-07T11:15:37]",

📘

Query in API

Here are the APIs that use Query

• Search API
• Count API
• Aggregate APIs (/api/aggregate/term and /api/aggregate/date_histogram)

Historical Searching

Search API supports the searching of content historically.

If you set no dates, a search covers everything in the index. To limit a search to a period, set query_start_date and query_end_date inside query:

"query_start_date": "2021-10-01T00:00:00Z",
"query_end_date": "2022-01-31T00:00:00Z"

Or include a document date (doc_date) range in your query string specifying the range you would like to view documents from. Both methods return the same documents.

"doc_date" is a required field for every data source ingested.

"query": "content.body: movies coming next summer AND doc_date:[2021-10-01 TO 2022-01-31]"

🚧

Historical Search Limitations

The farther back you perform a historical search and the more complex the query, the slower your query will perform. If your query takes over 10 minutes, the query will not complete. It is recommended to apply a sort other than default relevance.

Highlighting

Search API supports the highlighting of terms within responses matching the query.

To enable highlighting, simply indicate using the code snippet below:

"highlight": true

Within results, a new field content.body.highlight will contain the content with <highlight></highlight> tags before and after the content pieces matching the query. To use your own tags, set highlight_pre_tex and highlight_post_text (the parameter name is spelled highlight_pre_tex).