Competition Management

Competition Management

Competition templates are created and managed from the DRES web UI. Log in with an administrator account and navigate to the Evaluation Template Builder section. They can also be managed via the CLI (template create/delete/copy/rename/list/show/prepare/export/import).

A competition template is expanded by adding teams (with users), task types, task groups, and tasks.

Task Types

A task type is an organisational template for a task group. It defines:

• What types of media items the task accepts
• How the query hints (task description) are built
• The default duration
• Scoring method and submission filters

For the large international evaluation campaigns (VBS, LSC), DRES provides built-in task type templates.

Target

Defines what a correct submission must identify:

Option	Description
MEDIA_ITEM	A single media item is the target
MEDIA_ITEM_TEMPORAL_RANGE	A specific time segment within a video item is the target
JUDGEMENT	Human judges decide whether each submission is correct
JUDGEMENT_WITH_VOTE	Non-judge users vote on submissions (audience/crowd voting); see Judgement
TEXT	A free-text target, matched against the submitted text

Task Components

Defines what query hints are shown to participants:

Option	Description
EXTERNAL_IMAGE	Path to an external image file
EXTERNAL_VIDEO	Path to an external video file
IMAGE_ITEM	An image from a media collection
TEXT	A textual query hint
VIDEO_ITEM_SEGMENT	A video clip from a media collection

Scoring

Option	Description
KIS	Known Item Search scoring. Score decreases the longer a team takes to find the correct item, with a penalty per wrong submission.
AVS	Ad-hoc Video Search scoring. Teams score points for each distinct correct video found, with a penalty per wrong submission per video.
LEGACY_AVS	The previous AVS scoring formula, kept for backwards compatibility with older evaluation setups.
NOOP	No scoring. Useful for practice tasks or tasks where scoring is handled externally.

Submission Filters

Option	Description
NO_DUPLICATES	Duplicate submissions are rejected
LIMIT_CORRECT_PER_TEAM	Only the first correct submission per team counts; further correct submissions are ignored
LIMIT_WRONG_PER_TEAM	Limits how many wrong submissions a team may make
LIMIT_TOTAL_PER_TEAM	Limits the total number of submissions a team may make
LIMIT_CORRECT_PER_MEMBER	Limits correct submissions per individual team member
TEMPORAL_SUBMISSION	The temporal position of the submission relative to the task time is taken into account
TEXTUAL_SUBMISSION	Submission is validated as free text
ITEM_SUBMISSION	Submission is validated as a media item reference
MINIMUM_TIME_GAP	Enforces a minimum time gap between successive submissions from the same team

Options

Option	Description
HIDDEN_RESULTS	Submission results (correct or wrong) are not shown in the viewer to other participants
MAP_TO_SEGMENT	Submissions are internally mapped to the nearest known video segment. Usually desired for video tasks.
PROLONG_ON_SUBMISSION	Extends the task's duration if a submission arrives within a certain time of the task ending

Task Groups

A task group is a collection of tasks that all share the same task type. For example, you might create one task type and then two groups under it — one for a practice round and one for the actual competition.

Tasks

A task has a name, a target media collection, and a duration. It also has one or more targets (depending on the task type) and one or more query hints.

Query Hint Timing

By default, a hint with no explicit start time is shown from the very start of the task (offset 0) — hints are not automatically spread across the task duration. You can control timing by setting explicit start times (in seconds from the beginning of the task):

• A hint with no start time set is visible from the beginning.
• A hint with start = 30 becomes visible 30 seconds into the task.
• A hint with start = 60 becomes visible 60 seconds into the task.