Project 1: Data and Systems Support for Scene and Activity Understanding
Computer vision serves as the foundation of situational awareness in streetscape applications. Modern computer vision approaches rely on machine learning, which dictates the need for training across large datasets, as well as computing workflows that optimize training and inference. For the former, CS3 is developing training datasets required to achieve high-fidelity situational awareness through its own testbeds and third-party sources and developing mechanisms to automate the labeling and curation of these datasets. For the latter, CS3 is developing new workflows to optimize execution performance of training and inference workloads, while introducing protections that preserve situational awareness without revealing personally identifiable information.
Project 2: Scene and Activity Understanding
Detecting and understanding streetscape objects (e.g., pedestrians, vehicles) and activities (e.g., cross the street) is the core of situational awareness. Beyond the inherent complexity and dynamism of modern streetscape objects and activities, these tasks are complicated by a broad range of factors, from low-resolution camera feeds, to environmental occlusions (e.g., fog, rain, traffic), to scenes that cannot be captured by a single camera. To address these challenges, CS3 is advancing the fundamental science and engineering of scene and activity understanding for complex streetscape scenarios, under variable resolution and occlusion, over multi-camera networks.
Project 3: Trajectory Analysis and Prediction
Future streetscape applications depend not only on the current state of the streetscape, but on the anticipated future state. As examples, future smart intersections must anticipate a pedestrian’s intent to cross, and future traffic safety applications must anticipate a vehicle’s intent to change lanes, even if the driver fails to signal. CS3 is developing new mechanisms and systems to forecast object trajectories within modern streetscapes over near and far time horizons (e.g., 1s, 10s) to enable these applications.
Project 4: Multi-Modal Integration
Computer vision, which primarily relies upon video data, plays an important role in realizing situational awareness across many streetscape applications. In some cases, however, non-video data sources can complement or replace the computer vision pipeline. This includes LiDAR, mmWave radar, environmental sensors, RTK/GPS, and data (intentionally) provided by pedestrians (e.g., inertial data from smartphones). CS3 is exploring the ways in which these multi-modal sources can be integrated within its situational awareness framework to enhance scene and activity understanding.
Researchers
Andrew W. Smyth
Center Director & Principal Investigator; Professor of Civil Engineering and Engineering Mechanics, Columbia University
Mubarak Shah
Situational Awareness Research Co-Lead; Trustee Chair Professor of Computer Science, University of Central Florida
Carl Vondrick
Situational Awareness Research Co-Lead; YM Associate Professor of Computer Science, Columbia University
Jorge Ortiz
Applications Research Lead; Assistant Professor of Electrical and Computer Engineering, Rutgers University
Zoran Kostic
Professor of Professional Practice, Electrical Engineering, Columbia University
Mohamed Abdel-Aty
Pegasus Professor and Chair of Civil, Environmental & Construction Engineering, University of Central Florida
Sharon Di
Associate Professor of Civil Engineering and Engineering Mechanics, Columbia University
Mehmet Kerem Türkcan
Associate Research Scientist in Civil Engineering, Columbia University
Trainees
Boshra Khalili
Ph.D. Student in Civil Engineering & Engineering Mechanics at Columbia University
Chengbo Zang
SLC Industry Liaison; Ph.D. Student in Electrical Engineering, Columbia University
Dai Quoc Tran
Postdoc in Civil, Environmental and Construction Engineering at University of Central Florida
Fazil Kagdi
M.S. Student in Computer Vision at University of Central Florida
Rodrigo Vena Garcia
Ph.D. Student at University of Central Florida
