Indri Structured Query Retrieval
Contents
- Overview
- Applications
- Indri Structured Query Language
The Indri structured query language brings structured queries to
language modeling retreival. Among other things, this query language
enables the use of proximity operators (ordered and unordered windows)
and field operators in a language modeling context.
The Indri structured query language can only be used with an Indri index (as built by IndriBuildIndex or BuildIndex with indexType=indri).
Pseudo-feedback is implemented via relevance model (RM1) expansion.
RetEval
See "Lemur Retrieval Applications" section.
This application runs retrieval evaluation using the Indri SQL with smoothing options. The indri applications, IndriBuildIndex, IndriDaemon, and IndriRunQuery accept parameters from either the command line or from
a file. The parameter file uses an XML format. The command line uses dotted
path notation. The top level element in the parameters file is named parameters.
Retrieval Parameters:
- memory
- an integer value specifying the number of bytes to use for the
query retrieval process. The value can include a scaling factor by
adding a suffix. Valid values are (case insensitive) K = 1000, M =
1000000, G = 1000000000. So 100M would be equivalent to 100000000. The
value should contain only decimal digits and the optional
suffix. Specified as <memory>100M</memory> in the parameter
file and as -memory=100M on the command line.
- index
- path to an Indri Repository. Specified as
<index>/path/to/repository</index> in the parameter file and
as -index=/path/to/repository on the command line. This element
can be specified multiple times to combine Repositories.
- server
- hostname of a host running an Indri server (Indrid). Specified as
<server>hostname</server> in the parameter file and
as -server=hostname on the command line. The hostname can
include an optional port number to connect to, using the form
hostname:portnum. This element
can be specified multiple times to combine servers.
- count
- an integer value specifying the maximum number of results to
return for a given query. Specified as
<count>number</count> in the parameter file and
as -count=number on the command line.
- rule
- specifies the smoothing rule (TermScoreFunction) to apply. Format of
the rule is:
( key ":" value ) [ "," key ":" value ]*
Here's an example rule in command line format:
-rule=method:linear,collectionLambda:0.2,field:title
and in parameter file format:
<rule>method:linear,collectionLambda:0.2,field:title</rule>
This corresponds to Jelinek-Mercer smoothing with background lambda
equal to 0.2, only for items in a title field.
If nothing is listed for a key, all values are assumed.
So, a rule that does not specify a field matches all fields. This makes
-rule=method:linear,collectionLambda:0.2 a valid rule.
Valid keys:
- method
- smoothing method (text)
- field
- field to apply this rule to
- operator
- type of item in query to apply to { term,
window }
Valid methods:
- dirichlet
- (also 'd', 'dir') (default mu=2500)
- jelinek-mercer
- (also 'jm', 'linear') (default
collectionLambda=0.4, documentLambda=0.0), collectionLambda is also
known as just "lambda", either will work
- twostage
- (also 'two-stage', 'two') (default mu=2500,
lambda=0.4)
If the rule doesn't parse correctly, the default is Dirichlet, mu=2500.
- stopper
- a complex element containing one or more subelements named word,
specifying the stopword list to use. Specified as
<stopper><word>stopword</word></stopper> and
as -stopper.word=stopword on the command line. This is an
optional parameter with the default of no stopping.
Formatting Parameters:
- queryOffset
- an integer value specifying one less than the starting query number, eg 150 for
TREC formatted output. Specified as
<queryOffset>number</queryOffset> in the parameter file and
as -queryOffset=number on the command line.
- runID
- a string specifying the id for a query run, used in TREC scorable
output. Specified as
<runID>someID</runID> in the parameter file and
as -runID=someID on the command line.
- trecFormat
- the symbol true to produce TREC scorable output, otherwise
the symbol false. Specified as
<trecFormat>true</trecFormat> in the parameter file and
as -trecFormat=true on the command line. Note that 0
can be used for false, and 1 can be used for true.
Pseudo-Relevance Feedback Parameters:
- fbDocs
- an integer specifying the number of documents to use for
feedback. Specified as
<fbDocs>number</fbDocs> in the parameter file and
as -fbDocs=number on the command line.
- fbTerms
- an integer specifying the number of terms to use for
feedback. Specified as
<fbTerms>number</fbTerms> in the parameter file and
as -fbTerms=number on the command line.
- fbMu
- a floating point value specifying the value of mu to use for
feedback. [NB: document the feedback formulae]. Specified as
<fbMu>number</fbMu> in the parameter file and
as -fbMu=number on the command line.
- fbOrigWeight
- a floating point value in the range [0.0..1.0] specifying the weight
for the original query in the expanded query. Specified as
<fbOrigWeight>number</fbOrigWeight> in the parameter file and
as -fbOrigWeight=number on the command line.
The structured query operators are classified as either
belief, proximity, or field operators. Belief
operators allow beliefs about terms, proximity expressions, and other
complex expressions to be combined. The primary operators for most
queries are #combine and #weight.
TERMS / PROXIMITY
Terms:
- term -- stemmed / normalized term
- "term" -- unstemmed / unnormalized term
- #base64( ... ) -- converts from base64 -> ascii and then stems and normalizes. useful for including non-parsable terms in a query
- #base64quote( ... ) -- same as #base64 except the the ascii term is unstemmed and unnormalized
Examples:
- dogs
- "NASA"
- #base64(Wyh2Lm4ucC5hLnIucy5hLmIubC5lLild) -- equivalent to query term [(u.n.p.a.r.s.a.b.l.e.)]
Proximity terms:
- #odN( ... ) -- ordered window -- terms must appear ordered, with at most N-1 terms between each
- #N( ... ) -- same as #odN
- #uwN( ... ) unordered window -- all terms must appear within window of length N in any order
- #uw( ... ) -- unlimited unordered window -- all terms must appear within current context in any order
Examples:
- #1(white house) -- matches "white house" as an exact phrase
- #2(white house) -- matches "white * house" (where * is any word or null)
- #uw2(white house) -- matches "white house" and "house white"
Synonyms:
- #syn( ... )
- { ... }
- < ... >
Each of these forms does the same thing. They treat all of the expressions listed as the same term.
Examples:
- #syn( #1(united states) #1(united states of america) )
- {dog canine}
- <#1(light bulb) lightbulb>
NOTE: The arguments given to this operator can only be term/proximity expressions.
"Any" operator:
- #any -- used to match extent types
Examples:
- #any:PERSON -- matches any occurence of a PERSON extent
- #1(napolean died in #any:DATE) -- matches exact phrases of the form: "napolean died in <date>...</date>"
Field restriction / evaluation:
- expression.f1,,...,fN(c1,...,cN) -- matches when the expression appears
in field f1 AND f2 AND ... AND fN and evaluates the expression using the
language model defined by the concatenation of fields c1...cN within the
document.
Examples:
- dog.title -- matches the term dog appearing in a title extent (uses
document language model)
- #1(trevor strohman).person -- matches the phrase "trevor strohman" when it
appears in a person extent (uses document language model)
- dog.(title) -- evaluates the term based on the title language model for
the document
- #1(trevor strohman).person(header) -- builds a language model from all of
the "header" text in the document and evaluates #1(trevor strohman).person
in that context (matches only the exact phrase appearing within a person
extent within the header context)
COMBINING BELIEFS
Belief operators:
- #sum
- #wsum
- #wand (weighted and)
- #or
- #combine
- #weight
- #max
- #not
- #band (boolean and)
Examples:
- #combine( <dog canine> training )
- #combine( #1(white house) <#1(president bush) #1(george bush)> )
- #weight( 1.0 #1(white house) 2.0 #1(easter egg hunt) )
NOTE: If you are unsure which belief operator to use, it always "safest" to default to
using the #combine operator. This operator is the best choice for queries that combine
evidence from simple term/proximity expressions.
Extent retrieval:
- #beliefop[field]( query ) -- evaluates #beliefop( query ) for all extents
of type "field" in the document and returns a score for each. the language
model used to evaluate the query is formed from the text of the extent.
Example:
- #combine[sentence]( #1(napolean died in #any:DATE ) ) -- returns a scored
list of sentence extents that match the given query
See the additional extent retrieval documentation for more details. Also, the PowerPoint presentation found
here discusses more of the details and gives further
examples.
FILTER OPERATORS
Filter operators:
- #filreq -- filter require
- #filrej -- filter reject
Examples:
- #filreq( sheep #combine(dolly cloning) ) -- only consider those documents matching the query "sheep" and rank
them according to the query #combine(dolly cloning)
- #filrej( parton #combine(dolly cloning) ) -- only consider those documents NOT matching the query "parton" and rank them according to the query #combine(dolly cloning)
NOTE: first argument must always be a term/proximity expression
NUMERIC / DATE FIELD OPERATORS
General numeric operators:
- #less( F N ) -- matches numeric field extents of type F if value < N
- #greater( F N ) -- matches numeric field extents of type F if value > N
- #between( F N_low N_high ) -- matches numeric field extents of type F if N_low < value < N_high
- #equals( F N ) -- matches numeric field extents of type F if value == N
Date operators:
- #date:after( D ) -- matches numeric "date" extents if date is after D
- #date:before( D ) -- matches numeric "date" extents if date is before D
- #date:between( D_low, D_high ) -- matches numeric "date" extents if D_low < date < D_high
Accepted date formats:
- 11 january 2004
- 11-JAN-04
- 11-JAN-2004
- January 11 2004
- 01/11/04 (MM/DD/YY)
- 01/11/2004 (MM/DD/YYYY)
Examples:
- #filreq(#less(READINGLEVEL 10) george washington) -- if each document in a collection contained a
numeric tag that specified the reading level of the document, then this query will only retrieve
documents that have a reading level below grade 10 and documents will be ranked according to the
query "george washington".
- #combine( european history #date:between( 01/01/1800, 01/01/1900 ) ) -- such a query may be
constructed to find information about 19th century european history, as this query will find
pages that discuss "european history" and contain 19th century dates.
NOTE: The general numeric operators only work on indexed numeric fields, whereas the date
operators are only applicable to a specially indexed numeric field named "date". See the
indexing documentation for more on numeric fields.
DOCUMENT PRIORS
Prior
- #prior( NAME ) -- creates the document prior specified by the name given
Example:
- #combine(#prior(RECENT) global warming) -- we might create a prior named RECENT to be used to
give greater weight to documents that were published more recently.
NOTE: Please see the documentation on priors for more detailed information on how to specify and
use priors.