NTCIR Temporal Information Access (Temporalia) Task

Task Description


Temporalia at NTCIR-11 will offer two subtasks to address temporal information access technologies as follows. Interested researchers and research groups can participate in either or both of the subtasks.

Please read our important notification before running formal run data with your system.


Temporal Query Intent Classification (TQIC) subtask

TQIC subtask asks participants to classify a given query string to one of temporal classes. Participants will receive a set of query strings and query submitting date, and develop a system to classify each of the query strings to one of temporal classes. Participants were allowed to use any external resources to complete the TQIC subtask as long as the detail of external resource usage is described in their report. This subtask does not require to index our document collection.

Sample queries and corresponding temporal classes
ClassQuery
PastWho Was Martin Luther
Pastwhen did the titanic sink
PastYuri Gagarin Cause of Death
PastHistory of Coca-Cola
Pastprice hike in bangladesh 2008
Recencyapple stock price
RecencyNumber of Millionaires in USA
Recencytime in london
RecencyTrendy Plus Size Clothing
RecencyDid the Pirates Win Today
Future2013 MLB Playoff Schedule
FutureCollege Baseball Regional Projections
Futuredisney prices 2014
Futurelong term weather forecast
Futurerelease date for ios7
Atemporalblood pressure monitor
Atemporaldistance from earth to sun
Atemporalhow to start a conversation
AtemporalNew York Times
Atemporallose weight quickly

Query submitting date is assumed to be on Feb 28, 2013 GMT+0:00 for all queries shown above.

Conceptual (working) definitions of temporal classes
  • Past query: query about past events, whose search results are not expected to change much along with the time passage.
  • Recency query: query about recent things, whose search results are expected to be timely and up to date (usually this type of query refers to events that happened in very near past or at present time, unlike the “past” query that tends to refer to events in relatively distant past). The information contained in search results usually changes quickly along with the time passage.
  • Future query: query about predicted or scheduled events, the search results of which should contain future-related information.
  • Atemporal query: query without any clear temporal intent (its returned search results are not expected to be related to time). Navigational queries are considered to be atemporal.

What participants need to submit, and how to measure the performance
Participants are asked to submit a temporal class (Past, Recency, Future, or Atemporal) for each of the queries. The performance of submitted runs will be measured by the number of queries with correct temporal classes divided by the total number of queries. We will provide the breakdown of the performance based on temporal classes.

Number of queries to be delivered
We plan to deliver 100 queries for dry runs, and deliver another 100 300 queries for formal runs.

Temporal Information Retrieval (TIR) subtask

TIR subtask asks participants to retrieve a set of documents in response to a search topic that incorporates time factor. In addition to a typical search topic description (i.e., title, description, and sub topics), TIR search topic description will contain a query submitting date information to indicate the relationship between the query and searcher. Participants were allowed to use any external resources to complete the TIR subtask as long as the detail of external resource usage is described in their report. This subtask require to index our document collection.

Sample search topics
TitleGirl with the Dragon Tattoo
DescriptionI've recently watched a film called Girl with the Dragon Tattoo, and really liked it. Therefore, I would like to gather information about the movie.
Past subtopicHow did the casting of the film develop?
Recency subtopicWhat did the recent reviews say about the film?
Future subtopicIs there any plan about its sequel?
Atemporal subtopicWhat are the names of main actors and actresses of the film?
Search date28 Feb 2013 GMT+0:00

TitleFather's Day
DescriptionI am from a country where Father's Day is not a common custom, and I would like to learn more about the Father's day and its relation to the role of father in society.
Past subtopicWhat is the history and origin of the Father's day?
Recency subtopicWhat famous persons has lately done during Father's day?
Future subtopicWhat is the future outlook for the problem of fatherlessness and how the father's role is supposed to change?
Atemporal subtopicWhat are common gifts for Father's day?
Search date1 Mar 2013 GMT+0:00


What participants need to submit, and how to measure the performance
Participants are asked to submit the top 100 documents for each of temporal subtopic per topic (i.e., top 100 documents for past subtopic, another 100 for recency subtopic, etc.). The retrieval effectiveness will be evaluated by the precision at 20 for each of the temporal subtopic. We will provide the breakdown of the performance across temporal classes.

Number of search topics to be delivered
We plan to deliver 10 search topics (with 4 temporal subtopic per topic) for dry runs. We plan to deliver 50 search topics for formal runs.

Topic fields to use for your ranking
It is up to you about what fields of topic descriptions are used as system inputs. However, please make sure to report the input fields when you submit a run and when you write a participant report. Typical combinations might be

  • Title and subtopic
  • Title, Description, and subtopic
Please do not use the subtopic class information. For the convenience of data management, subtopic IDs of TIR contain a token of temporal classes such as 001p for past subtopic and 001f for future subtopic. However, please do not use this token as an input to your system.

Feedback or suggestions?

NTCIR-11 Temporalia is a pilot task which means that we are still in development of evaluation methodology. Therefore, if you have any feedback or suggestions for our task design, please feel free to post a message to our Google Groups!