Project 1: Data and Systems Support for Scene and Activity Understanding
Computer vision serves as the foundation of situational awareness in streetscape applications. Modern computer vision approaches rely on machine learning, which dictates the need for training across large datasets, as well as computing workflows that optimize training and inference. For the former, CS3 is developing training datasets required to achieve high-fidelity situational awareness through its own testbeds and third-party sources and developing mechanisms to automate the labeling and curation of these datasets. For the latter, CS3 is developing new workflows to optimize execution performance of training and inference workloads, while introducing protections that preserve situational awareness without revealing personally identifiable information.
Project 2: Scene and Activity Understanding
Detecting and understanding streetscape objects (e.g., pedestrians, vehicles) and activities (e.g., cross the street) is the core of situational awareness. Beyond the inherent complexity and dynamism of modern streetscape objects and activities, these tasks are complicated by a broad range of factors, from low-resolution camera feeds, to environmental occlusions (e.g., fog, rain, traffic), to scenes that cannot be captured by a single camera. To address these challenges, CS3 is advancing the fundamental science and engineering of scene and activity understanding for complex streetscape scenarios, under variable resolution and occlusion, over multi-camera networks.
Project 3: Trajectory Analysis and Prediction
Future streetscape applications depend not only on the current state of the streetscape, but on the anticipated future state. As examples, future smart intersections must anticipate a pedestrian’s intent to cross, and future traffic safety applications must anticipate a vehicle’s intent to change lanes, even if the driver fails to signal. CS3 is developing new mechanisms and systems to forecast object trajectories within modern streetscapes over near and far time horizons (e.g., 1s, 10s) to enable these applications.
Project 4: Multi-Modal Integration
Computer vision, which primarily relies upon video data, plays an important role in realizing situational awareness across many streetscape applications. In some cases, however, non-video data sources can complement or replace the computer vision pipeline. This includes LiDAR, mmWave radar, environmental sensors, RTK/GPS, and data (intentionally) provided by pedestrians (e.g., inertial data from smartphones). CS3 is exploring the ways in which these multi-modal sources can be integrated within its situational awareness framework to enhance scene and activity understanding.
Recent Publications
Researchers
Andrew W. Smyth
Center Director & Principal InvestigatorProfessor of Civil Engineering and Engineering Mechanics
Columbia University
Carl Vondrick
Situational Awareness Research LeadYM Associate Professor of Computer Science
Columbia University
Jorge Ortiz
Applications Research LeadAssistant Professor of Electrical and Computer Engineering
Rutgers University
Sharon Di
Associate Professor of Civil Engineering and Engineering MechanicsColumbia University
Mehmet Kerem Türkcan
Associate Research Scientist in Civil EngineeringColumbia University
Markus Schläpfer
Assistant Professor of Civil Engineering and Engineering MechanicsColumbia University
Zhengbo Zou
Assistant Professor of Civil Engineering and Engineering MechanicsColumbia University
