Task Description
Temporalia at NTCIR-11 will offer two subtasks to address temporal
information access technologies as follows.
Interested researchers and research groups can participate in either or both of the subtasks.
Please read our
important notification before running formal run data with your system.
Temporal Query Intent Classification (TQIC) subtask
TQIC subtask asks participants to classify a given query string to one of temporal classes.
Participants will receive a set of query strings and query submitting date,
and develop a system to classify each of the query strings to one of temporal classes.
Participants were allowed to use any external resources to complete the TQIC subtask
as long as the detail of external resource usage is described in their report.
This subtask does not require to index our
document collection.
Sample queries and corresponding temporal classes
| Class | Query |
|---|
| Past | Who Was Martin Luther |
| Past | when did the titanic sink |
| Past | Yuri Gagarin Cause of Death |
| Past | History of Coca-Cola |
| Past | price hike in bangladesh 2008 |
| Recency | apple stock price |
| Recency | Number of Millionaires in USA |
| Recency | time in london |
| Recency | Trendy Plus Size Clothing |
| Recency | Did the Pirates Win Today |
| Future | 2013 MLB Playoff Schedule |
| Future | College Baseball Regional Projections |
| Future | disney prices 2014 |
| Future | long term weather forecast |
| Future | release date for ios7 |
| Atemporal | blood pressure monitor |
| Atemporal | distance from earth to sun |
| Atemporal | how to start a conversation |
| Atemporal | New York Times |
| Atemporal | lose weight quickly |
Query submitting date is assumed to be on Feb 28, 2013 GMT+0:00 for all queries shown above.
Conceptual (working) definitions of temporal classes
- Past query: query about past events, whose search results are not expected to
change much along with the time passage.
- Recency query: query about recent things, whose search results are expected to be timely
and up to date (usually this type of query refers to events that happened in very near past or at present time,
unlike the “past” query that tends to refer to events in relatively distant past).
The information contained in search results usually changes quickly along with the time passage.
- Future query: query about predicted or scheduled events,
the search results of which should contain future-related information.
- Atemporal query: query without any clear temporal intent (its returned search results are
not expected to be related to time). Navigational queries are considered to be atemporal.
What participants need to submit, and how to measure the performance
Participants are asked to submit a temporal class (Past, Recency, Future, or Atemporal) for each of the queries.
The performance of submitted runs will be measured by the number of queries with correct temporal classes divided
by the total number of queries. We will provide the breakdown of the performance based on temporal classes.
Number of queries to be delivered
We plan to deliver 100 queries for dry runs, and deliver another 100 300 queries for formal runs.
Temporal Information Retrieval (TIR) subtask
TIR subtask asks participants to retrieve a set of documents in response to a search topic that incorporates time factor.
In addition to a typical search topic description (i.e., title, description, and sub topics),
TIR search topic description will contain a query submitting date information to indicate the relationship between
the query and searcher. Participants were allowed to use any external resources to complete the TIR subtask as long as
the detail of external resource usage is described in their report. This subtask require to index our
document collection.
Sample search topics
| Title | Girl with the Dragon Tattoo |
|---|
| Description | I've recently watched a film called Girl with the Dragon Tattoo,
and really liked it. Therefore, I would like to gather information about the movie. |
|---|
| Past subtopic | How did the casting of the film develop? |
|---|
| Recency subtopic | What did the recent reviews say about the film? |
|---|
| Future subtopic | Is there any plan about its sequel? |
|---|
| Atemporal subtopic | What are the names of main actors and actresses of the film? |
|---|
| Search date | 28 Feb 2013 GMT+0:00 |
|---|
| Title | Father's Day |
|---|
| Description | I am from a country where Father's Day is not a common custom,
and I would like to learn more about the Father's day and its relation to the role of father in society. |
|---|
| Past subtopic | What is the history and origin of the Father's day? |
|---|
| Recency subtopic | What famous persons has lately done during Father's day? |
|---|
| Future subtopic | What is the future outlook for the problem of fatherlessness and
how the father's role is supposed to change? |
|---|
| Atemporal subtopic | What are common gifts for Father's day? |
|---|
| Search date | 1 Mar 2013 GMT+0:00 |
|---|
What participants need to submit, and how to measure the performance
Participants are asked to submit the top 100 documents for each of temporal subtopic per topic
(i.e., top 100 documents for past subtopic, another 100 for recency subtopic, etc.).
The retrieval effectiveness will be evaluated by the precision at 20 for each of the temporal subtopic.
We will provide the breakdown of the performance across temporal classes.
Number of search topics to be delivered
We plan to deliver 10 search topics (with 4 temporal subtopic per topic) for dry runs.
We plan to deliver 50 search topics for formal runs.
Topic fields to use for your ranking
It is up to you about what fields of topic descriptions are used as system inputs.
However, please make sure to report the input fields when you submit a run and when you write a participant report.
Typical combinations might be
- Title and subtopic
- Title, Description, and subtopic
Please do not use the subtopic class information.
For the convenience of data management, subtopic IDs of TIR contain a token of temporal classes
such as 001p for past subtopic and 001f for future subtopic.
However, please do not use this token as an input to your system.
Feedback or suggestions?
NTCIR-11 Temporalia is a pilot task which means that we are still in development of evaluation methodology.
Therefore, if you have any feedback or suggestions for our task design,
please feel free to post a message to our
Google Groups!