Submissions, SubmissionStatuses, SubmissionBundles

What are Submissions?

In Synapse, a Submission is your entry to a challenge or evaluation queue. When you participate in a computational challenge or collaborative project, you submit your work (such as predictions, models, or analysis results) as a Submission to be evaluated and scored.

Key Concepts

Before working with Submissions, it’s helpful to understand how they fit into Synapse:

  • Entity: Your actual work stored in Synapse (like a File containing predictions or a Docker image with your model)
  • Evaluation: A queue that accepts and organizes submissions for a specific challenge or project
  • Submission: The object associated with submitting your Entity to an Evaluation queue, creating a record that can be tracked and scored

How Submissions Work

When you submit an Entity to an Evaluation:

  • The Submission creates an immutable record linking your Entity to that Evaluation
  • The Evaluation owns this Submission record (not you as the submitter)
  • Organizers can add scores and feedback through a SubmissionStatus object
  • You can track your Submissions and view their statuses

What You’ll Learn

This tutorial covers two perspectives:

  1. Participating in challenges: Making and tracking your submissions
  2. Organizing challenges: Scoring and managing submissions from participants

Tutorial Purpose

In this tutorial:

As a participant of a Synapse challenge, you will

  1. Make a submission to an existing evaluation queue on Synapse
  2. Fetch your existing submission
  3. Count your submissions
  4. Fetch all of your submissions from an existing evaluation queue on Synapse
  5. Check the status of your submission
  6. Cancel your submission

As an organizer of a Synapse challenge, you will

  1. Annotate a submission to score it
  2. Batch-update submission statuses
  3. Fetch the submission bundles for an evaluation queue
  4. Allow cancellation of submissions
  5. Delete submissions

Prerequisites

  • You have an existing Evaluation on Synapse to work from
  • You have an existing entity with which to make a submission (can be a File or Docker Repository)
  • You have the correct permissions on the Evaluation queue for your desired tutorial section (participant or organizer)

1. Participating in a Synapse challenge

Initial setup

library(synapser)
synLogin()

# REQUIRED: Set these to your actual Synapse IDs
# Do NOT leave these as NULL - the script will not work properly
EVALUATION_ID <- NULL # Replace with the evaluation queue ID you submitted to
ENTITY_ID <- NULL # Replace with the entity ID you want to submit

stopifnot(
  "EVALUATION_ID must be set to the evaluation queue ID" = !is.null(EVALUATION_ID),
  "ENTITY_ID must be set to the entity ID you want to submit" = !is.null(ENTITY_ID)
)

print(sprintf("Working with Evaluation: %s", EVALUATION_ID))
print(sprintf("Submitting Entity: %s", ENTITY_ID))

1. Make a submission to an existing evaluation queue on Synapse

# Create a new Submission object
submission <- Submission(
  entity_id = ENTITY_ID, evaluation_id = EVALUATION_ID, name = "My Tutorial Submission"
)

# Submit the entity to the evaluation queue
submission <- submission |> synStore()

print("Submission created successfully!")
print(sprintf("Submission ID: %s", submission$id))
print(sprintf("Submitted Entity: %s", submission$entity_id))
print(sprintf("Evaluation: %s", submission$evaluation_id))
print(sprintf("Submission Name: %s", submission$name))
print(sprintf("Created On: %s", submission$created_on))

# Store the submission ID for later use
submission_id <- submission$id

2. Fetch your existing submission

# Retrieve the submission we just created
retrieved_submission <- Submission(id = submission_id) |> synGet()

print("Retrieved submission:")
print(sprintf("  ID: %s", retrieved_submission$id))
print(sprintf("  Name: %s", retrieved_submission$name))
print(sprintf("  Entity ID: %s", retrieved_submission$entity_id))
print(sprintf("  Submitter: %s", retrieved_submission$submitter_alias))
print(sprintf("  Created On: %s", retrieved_submission$created_on))

3. Count your submissions

# Get the total count of submissions for this evaluation
submission_count <- synGetSubmissionCount(evaluation_id = EVALUATION_ID)

print(submission_count)

4. Fetch all of your submissions from an existing evaluation queue on Synapse

# Get all of your submissions for this evaluation
user_submissions <- as.list(Submission() |> synGetUserSubmissions(evaluation_id = EVALUATION_ID))

print(sprintf("Found %d submissions from the current user:", length(user_submissions)))
for (i in seq_along(user_submissions)) {
  sub <- user_submissions[[i]]
  print(sprintf("  %d. ID: %s, Name: %s, Created: %s", i, sub$id, sub$name, sub$created_on))
}

5. Check the status of your submission

# Fetch the status of our submission
status <- SubmissionStatus(id = submission_id) |> synGet()

print("Submission status details:")
print(sprintf("  Status: %s", status$status))
print(sprintf("  Modified On: %s", status$modified_on))
print(sprintf("  Can Cancel: %s", status$can_cancel))
print(sprintf("  Cancel Requested: %s", status$cancel_requested))

# Check if there are any submission annotations (scores, feedback, etc.)
annotations <- status$submission_annotations
if (length(annotations) > 0) {
  print("  Submission Annotations:")
  for (key in names(annotations)) {
    print(sprintf("    %s: %s", key, paste(unlist(annotations[[key]]), collapse = ", ")))
  }
} else {
  print("  No submission annotations available")
}

6. Cancel your submission

Only cancel a submission if it allows it. The cancellation code is commented out by default; uncomment it if you want to test this functionality.

# cancelled_submission = submission |> synCancel()
# print(sprintf("Submission %s has been requested for cancellation", cancelled_submission$id))
#
# # Check the updated status
# updated_status = SubmissionStatus(id = submission_id) |> synGet()
# print(sprintf("Cancel requested: %s", updated_status$cancel_requested))

print(sprintf("Your submission ID %s is ready for evaluation.", submission_id))
print("Check back later to see if the organizers have scored your submission.")

2. Organizing a Synapse challenge

Initial setup

library(synapser)
synLogin()

# REQUIRED: Set these to your actual Synapse IDs
# Do NOT leave these as NULL - the script will not work properly
EVALUATION_ID <- NULL # Replace with the evaluation queue ID you submitted to
SUBMISSION_ID <- NULL # Replace with the ID of a submission you made

stopifnot(
  "EVALUATION_ID must be set to the evaluation queue ID" = !is.null(EVALUATION_ID),
  "SUBMISSION_ID must be set to the ID of a submission you made" = !is.null(SUBMISSION_ID)
)

print(sprintf("Working with Evaluation: %s", EVALUATION_ID))
print(sprintf("Working with Submission: %s", SUBMISSION_ID))

1. Annotate a submission to score it

# First, get the submission status
status <- SubmissionStatus(id = SUBMISSION_ID) |> synGet()
print(sprintf("Retrieved submission status for submission %s", SUBMISSION_ID))
print(sprintf("Current status: %s", status$status))

# Update the submission status with scoring information
status$status <- "SCORED"
status$submission_annotations <- list(
  accuracy = list(0.85),
  precision = list(0.82),
  feedback = list("Good performance!"),
  validation_errors = "None detected",
  score_errors = "None detected"
)

# Store the updated status
updated_status <- status |> synStore()
print("Successfully scored submission!")
print(sprintf("Status: %s", updated_status$status))
print("Annotations added:")
for (key in names(updated_status$submission_annotations)) {
  print(sprintf("  %s: %s", key, paste(unlist(updated_status$submission_annotations[[key]]), collapse = ", ")))
}

2. Batch-update submission statuses

# First, get all submission statuses that need updating
statuses_to_update <- synGetAllSubmissionStatuses(
  evaluation_id = EVALUATION_ID,
  status = "RECEIVED" # Get submissions that haven't been scored yet
)

print(sprintf("Found %d submissions to batch update", length(statuses_to_update)))

if (length(statuses_to_update) > 0) {
  # Update each status with validation information
  for (i in seq_along(statuses_to_update)) {
    status <- statuses_to_update[[i]]
    status$status <- "VALIDATED"
    status$submission_annotations <- list(
      validation_status = list("PASSED"),
      validation_timestamp = list("2024-11-24T10:30:00Z"),
      batch_number = list(i),
      validator = list("automated_system")
    )
  }

  # Perform batch update
  batch_response <- synBatchUpdateSubmissionStatuses(
    evaluation_id = EVALUATION_ID,
    statuses = statuses_to_update,
    is_first_batch = TRUE,
    is_last_batch = TRUE
  )

  print("Batch update completed successfully!")
  print(batch_response)
} else {
  print("No submissions found with 'RECEIVED' status to update")
}

3. Fetch the submission bundles for an evaluation queue

# Get all scored submission bundles for the evaluation
print("Fetching all submission bundles for the evaluation...")

bundles <- as.list(SubmissionBundle() |> synGetEvaluationSubmissionBundles(
  evaluation_id = EVALUATION_ID,
  status = "SCORED" # Only get scored submissions
))

print(sprintf("Found %d scored submission bundles", length(bundles)))

first_bundles <- head(bundles, 5) # Show first 5
for (i in seq_along(first_bundles)) {
  submission <- first_bundles[[i]]$submission
  status <- first_bundles[[i]]$submission_status

  print(sprintf("Bundle %d:", i))
  if (!is.null(submission)) {
    print(sprintf("  Submission ID: %s", submission$id))
    print(sprintf("  Submitter: %s", submission$submitter_alias))
    print(sprintf("  Entity ID: %s", submission$entity_id))
    print(sprintf("  Created: %s", submission$created_on))
  }

  if (!is.null(status)) {
    print(sprintf("  Status: %s", status$status))
    print(sprintf("  Modified: %s", status$modified_on))
    annotations <- status$submission_annotations
    if (length(annotations) > 0) {
      print("  Scores:")
      for (key in names(annotations)) {
        if (key %in% c("accuracy", "f1_score", "precision", "recall")) {
          print(sprintf("    %s: %s", key, paste(unlist(annotations[[key]]), collapse = ", ")))
        }
      }
    }
  }
}

4. Allow cancellation of submissions

# First, check if any submissions have requested cancellation
all_statuses <- synGetAllSubmissionStatuses(evaluation_id = EVALUATION_ID)

cancellation_requests <- Filter(function(status) isTRUE(status$cancel_requested), all_statuses)

print(sprintf("Found %d submissions with cancellation requests", length(cancellation_requests)))

# Process cancellation requests
for (status in cancellation_requests) {
  print(sprintf("Processing cancellation request for submission %s", status$id))

  # Update to allow cancellation (organizer decision)
  status$can_cancel <- TRUE
  status$status <- "CLOSED"
  # submission_annotations comes back as an R copy, so update the copy
  # (adding or overwriting these keys) and assign it back
  annotations <- as.list(status$submission_annotations)
  annotations[c("cancellation_reason", "cancelled_by", "cancellation_date")] <- list(
    list("User requested cancellation"),
    list("organizer"),
    list("2024-11-24")
  )
  status$submission_annotations <- annotations

  # Store the update
  updated_status <- status |> synStore()
  print(sprintf("  Approved cancellation for submission %s", updated_status$id))
}

# Example: Proactively allow cancellation for a specific submission
print("Enabling cancellation for a specific submission...")
target_status <- SubmissionStatus(id = SUBMISSION_ID) |> synGet()
target_status$can_cancel <- TRUE
target_status <- target_status |> synStore()
print(sprintf("Cancellation enabled for submission %s", SUBMISSION_ID))

5. Delete submissions

Find the submissions whose submitters have requested cancellation, and delete them. The deletion code is commented out by default; uncomment it if you want to test this functionality.

# Get all submission statuses to check for cancellation requests
all_statuses <- synGetAllSubmissionStatuses(evaluation_id = EVALUATION_ID)

# Find submissions that have been requested for cancellation
submissions_to_delete <- list()
for (status in all_statuses) {
  if (isTRUE(status$cancel_requested)) {
    submissions_to_delete <- append(submissions_to_delete, status$id)
  }
}

print(sprintf("Found %d submissions with cancellation requests", length(submissions_to_delete)))

# Delete each submission that was requested for cancellation
for (submission_id in submissions_to_delete) {
  submission <- Submission(id = submission_id) |> synGet()
  submission |> synDelete()
  print(sprintf("Successfully deleted submission %s", submission_id))
}

if (length(submissions_to_delete) > 0) {
  print(sprintf("Completed deletion of %d requested submissions", length(submissions_to_delete)))
}