Modern cities are demanding for a “city-brain” to keep track of the big data that is generated from the traffic & surveillance systems for better understanding how the city’s traffic is evolving. In our case, to build such as “city-brain”, city-wide real time video analytics must be applied to hundreds of thousands of video streams distributed in 100+ video processing stations with up to 2000 streams per station and an estimated 200 PB of video data per city per month. That demands for a highly efficient & scalable analytical platform that enables analytical algorithms to be plugged-in & pipelined to support rich and flexible analytics and a scalable, high efficient, and cost effective big data platform to store and index the extracted content with automated tags from the video streams. We present a noval solution that uses streaming technology, machine learning, and deep learning for video analytics, in a Hadoop-Spark-Kafka environment to identify, extract, label & recognize moving objects with proper tags that are recognized progressively using dozens of algorithms (such as colors, size, position, time, direction, nature of the object, plate number, facial info of the driver, make/model of the vehicle, and etc) together with fog & light filtering and background cancelling. With all the moving objects data (images, tags, timestamps) extracted from city-wide cameras and the background in each camera scene, we can then re-build the entire scenes for years for the view-field of each camera.