The goal of this dissertation is to develop a vision-based system to evaluate risk scores of roadway intersections where frequent interactions of pedestrians and bicyclists with motorized traffic exist. The research outcome is valuable for preventing traffic incidents involving pedestrians or bicyclists, which are becoming a significant concern in urban areas. This research focuses on vehicle-pedestrian and vehicle-bicycle conflicts at signalized intersections. Multi-way video sequences were collected at several intersections in the City of San Diego, and computer vision algorithms, including object detection (Faster-R-CNN) and visual tracking (KCF), were developed to extract trajectory data of vehicles or pedestrians in videos. Learning-based 3D localization method was introduced to get rid of the perspective effect and further improve the usability and accuracy of trajectory data. These visual perception results are used in safety surrogate analysis to proactively evaluate safety at intersections. Results with comparative studies suggest the proposed algorithm can significantly improve localization accuracy, especially when using noisy video data, and result in improved safety measures.