We've Moved!
Visit SDSU’s new digital collections website at https://digitalcollections.sdsu.edu
Description
Climate science is a notorious producer of big data. The National Oceanic and Atmospheric Administration (NOAA) has many Petabytes of data at its disposal. Users have not adopted lossy data compression techniques, because of a fear that compression will unsatisfactorily diminish data quality or that compression will be too slow to be worth the effort. This research objective aims to compress large amount of climate data into smaller usable datasets by using state of the art video compression techniques. The test data consists of 32 years of Sea Surface Temperature taken daily by a NOAA satellite equipped with an Advanced Very High Resolution Radiometer (AVHRR). Such data is stored in NetCDF (Network Common Data Form) files, which were taken from the NOAA website. A cautious assessment of different resolutions was calculated using the Advanced Video Coding H.264 MPEG-4/AVC, as well as their influence on data quality, storage and performance. The quality of decoded images was calculated using different techniques like, Peak Signal to Noise Ratio (PSNR), Mean Square Error (MSE), Image Quality Index (IQ), Structural Similarity Index (MSSI), and a computer vision technique called Scale Invariant Feature Transform (SIFT). The decoded image dataset with the best visual result comes from the Full High Definition decoded images (1080P). The compression ratio of the encoded video using the H.264 codec is of at least 46 and with a high of 595 for 1080P resolution, depending on the bitrate and framerate selected. The encoded video is about 300 to 4000 times smaller than the original data. In general, this paper is able to show that video compression techniques provide an exceptional means of storing NetCDF databases at a shareable size, while preserving image quality.
