Run Formats
Please read our
important notification
before running formal run data with your system.
To submit the outputs of your system (called "runs"),
you need to follow the run format defined as below.
The runs formatted in a wrong way will not be assessed by the organizers.
Temporal Query Intent Classification (TQIC) subtask:
id class group_id run_id
001 future roi_y roi_y_system1
002 past roi_y roi_y_system1
...
Each line should contain tab separated result for a different query.
"id" is query identifier, while "class" represents the temporal class of query's
intent and should be one of: past, future, recent, atemporal.
"group_id" is an identifier of the group submitting the run and "run_id" is
the run number for the group.
For example, if you have two systems with different algorithms or parameters,
please use a different run ID for different system settings.
Each group can submit
up to three runs (i.e., Run IDs) for the TQIC subtask.
Temporal Information Retrieval (TIR) subtask:
id rank doc_id group_id run_id
001a 1 DOC1 roi_y roi_y_system2
001p 2 DOC2 roi_y roi_y_system2
...
Each line should contain
tab separated result containing relevant document
to a given subtopic. You can return up to 100 documents for each subtopic.
"id" is the query id, which is a concatenation of the query id and "p", "a",
"r" or "f" that indicate the past, atemporal, recent and future subtopics, respectively.
"rank" is the rank of the document for that query, "group_id" is an identifier of
the group submitting the run and "run_id" is the run number for the group.
For example, if you have two systems with different algorithms or parameters,
please use a different run ID for different system settings.
Documents are identified by doc_id.
Each group can submit
up to three runs (i.e., Run IDs) for the TIR subtask.
File name and format
Please use one file per subtask, and name your files with the subtask name and group id
(e.g., tqic_roi_y and tir_roi_y).
You can append multiple runs in a single file.
Please use one file per run ID for formal runs.
Regardless of the number of subtasks you're submitting, please zip all the files,
and name it as "group_id.zip" before uploading to the
submission system.