Context-Driven Detection of Invertebrate Species in Deep-Sea Video

R. Austin McEver, Bowen Zhang, Connor Levenson, A S M Iftekhar, B.S. Manjunath


Each year, underwater remotely operated vehicles (ROVs) collect thousands of hours of video of unexplored ocean habitats revealing a plethora of information regarding biodiversity on Earth. However, fully utilizing this information remains a challenge as proper annotations and analysis require trained scientists’ time, which is both limited and costly. To this end, we present a Dataset for Underwater Substrate and Invertebrate Analysis (DUSIA), a benchmark suite and growing large-scale dataset to train, validate, and test methods for temporally localizing four underwater substrates as well as temporally and spatially localizing 59 underwater invertebrate species. DUSIA currently includes over ten hours of footage across 25 videos captured in 1080p at 30 fps by an ROV following pre-planned transects across the ocean floor near the Channel Islands of California. Each video includes annotations indicating the start and end times of substrates across the video in addition to counts of species of interest. Some frames are annotated with precise bounding box locations for invertebrate species of interest, as seen in Figure 1. To our knowledge, DUSIA is the first dataset of its kind for deep sea exploration, with video from a moving camera, that includes substrate annotations and invertebrate species that are present at significant depths where sunlight does not penetrate. Additionally, we present the novel context-driven object detector (CDD) where we use explicit substrate classification to influence an object detection network to simultaneously predict a substrate and species class influenced by that substrate. We also present a method for improving training on partially annotated bounding box frames. Finally, we offer a baseline method for automating the counting of invertebrate species of interest.


Demo video

arXiv paper

Code repo

Docker container coming soon

CVPR 2022 Poster

DUSIA Video Data Download Instructions

In order to download the videos of our Dataset for Underwater Invertebrate Species Analysis (DUSIA), please visit Vision Research Lab's BisQue instance at, and follow the steps below.

  1. Make an account using this form. BisQue admin will reach out shortly via email using with a username and password.

  2. Log in and visit this link to find the DUSIA videos within BisQue.

  3. Near the top left of the page, select Download.

    • From the drop down menu select "Download Manager".

    • Near the bottom left, uncheck "Include annotations".

    • Near the bottom left, select the arrow to the left of "Download". Select "as TARball". Other archive formats may not be supported for DUSIA at this time.

    • Choose the download destination.

  4. Once downloaded, extract the videos.

  5. We provide md5 checksums below for each video file to ensure the integrity of all data transfers.

DUSIA md5checksums