Home Set — Jessica Jabr Website

Machine Learning Stunning Afterglows
FEB-MAY 2021

Image Machine Learning training using StyleGan model. Python coding & Tensorflow

Filed under:
ML, Design, Coding

Overview

I’m working with my own photos of “sunsets” which I’ve taken over the years. These photos have been taken in several countries and regions in the world. Sunset time is special to me and it gives me hope of a day gone and a new day coming. It’s also very dreamy and romantic to see how the sky’s colors change. Most of the sunsets in NYC are beautiful and the colors are stunning especially when it turns pink but this is only because of pollution. The more the air pollution, the more the sky turns pink when the sun is setting.

I hope to use the machine learning from the colors of the sky at sunset time to show variations of patterns in relation to pollution, and also use these patterns to predict more beautiful sunset skies.

Concept Exploration

Air index quality affects afterglow colors in sunsets especially in populated and dense areas, where air pollution is higher.
Comparing and contrasting between lesser density areas and afterglow colors.
Leveraging this tool for creating unique stunning sunset backdrops for design.

Dataset

1st dataset: 1470 photos, 629 frames extracted from 2 videos using opencv [by Aarati Akkapeddi https://colab.research.google.com/drive/1WWHNG4YqGSHfIYIUrC2tmoPQ3HOgU--e]

I have 2 videos which I transformed to images to add to the dataset. In total I planned to have around 2000 images for lengthy periods of training.

This dataset included photos taken in the U.S (several states), Lebanon, Spain, France, Turkey

Resized to 256x256

All in jpg format

2nd dataset: 787 frames, taken in New York City on June 25, 2019

extracted from 2 videos using opencv [by Aarati Akkapeddi]

Resized to 256x256

All in jpg format

Training Process

I trained dataset 1 for 20 hours
Downloaded pkl files
Interpolation
Generated a new dataset with generate.py code and selected names it dataset 2
I started re-training on the same worksheet, trained dataset 2 for 16 hours
Interpolation
Then used the last pickle file from dataset 1 training (000266.pkl) to train it with dataset 2 images. (cross pollination). Used the learning from the last dataset onto the latest training.
Interpolation
Downloaded pckl file 000048 from the last combined training (14th).

Outcome

Seeds from training dataset 1

Seeds from training dataset 2

Noticeable Outcomes

The Fine Styles — Color Scheme is preserved in outputs 1, 2, 3
There's adaptive normalization in 1 & 2
Comparison between the sunset output in Lebanon (mostly) and NY. Less dense areas were definitely lighter in sunset colors. While pink and orange were mostly in new york sunsets since the air is more polluted.
StyleGAN has generated realistic images
Slight loss in style and content in training 3

Further Development

I WANT TO INTERPOLATE USING DIFFERENT INPUTS AND MANIPULATE OUTPUTS USING EXISTING LEARNING TOOLS
EXPERIMENT INTERACTIVITY THROUGH PROCESSING - P5.JS
I AM DESIGNING SUNSETS, EACH DESIGN IS UNIQUE AND YOU CAN OWN IT

Applications to look into

Image Classification

Upload a picture, and the model tries to classify it depending on what it “sees” on the picture. This model uses transfer learning and is based on MobilenetV2.

Style transfer

Blending image style adding these sunset images as backdrops. Using image style to create a sunset.

Learning and experimenting with deep-learning challenges and approaches for sky images

First images sample

Process

I have a primary 870 images dataset. I began training with a portion of this set and hope to add all of the images to it.

I used the styleGAN training code and ran into a couple of errors in the first 3 trials after I finally was able to get an output.

The code is still running for a better outcome and I’ll upload more outputs in the upcoming week.

Early stage outcome

We can notice the clarity of the images becoming better with time & getting closer from looking real.

The below image looks interesting. Since most of the time the sky is positioned upwards and it seems like there’s a typical way we see sunsets, yet here it’s kind of rotated. I’m searching for more interesting findings; in general the colors and shapes seem close to the reality of true sunsets.

Bibliography

A. Implementation and Concept Research

I- What dust and pollution don't do

It is often written that natural and manmade dust and pollution cause colorful sunrises and sunsets. Indeed, the brilliant twilight "afterglows" that follow major volcanic eruptions owe their existence to the ejection of small particles high into the atmosphere (more will be said on this a bit later). If, however, it were strictly true that low-level dust and haze were responsible for brilliant sunsets, cities such as New York, Los Angeles, London, and Mexico City would be celebrated for their twilight hues. The truth is is that tropospheric aerosols --- when present in abundance in the lower atmosphere as they often are over urban and continental areas --- do not enhance sky colors --- they subdue them. Clean air is, in fact, the main ingredient common to brightly colored sunrises and sunsets.

Afterglow colors are affected by smoke, air quality, and other factors. The stunning hues when affected by smoke are mostly pinkish

Source: https://www.spc.noaa.gov/publications/corfidi/sunset/#:~:text=Typical%20pollution%20droplets%20such%20as,are%20on%20the%20order%20of%20.&text=Similarly%2C%20the%20vibrant%20oranges%20and,more%20than%20soften%20sky%20colors.

II- Machine Learning which sunsets are considered beautiful from social media data

Source:

https://twitter.com/Senor_Sunset
Luminar: Sky replacement tool - AI Powered Tool - picture editor
Adobe photoshop https://www.theverge.com/2020/9/21/21449124/photoshop-sky-replacement-tool-ai-machine-learning
SunsetWx - Sunset and sunrise forecasts. Sunset & Sunrise Predictions: Model using an in-depth algorithm comprised of meteorological factors. https://sunsetwx.com/
https://twitter.com/sunset_wx?ref_src=twsrc%5Etfw%7Ctwcamp%5Eembeddedtimeline%7Ctwterm%5Eprofile%3Asunset_wx&ref_url=https%3A%2F%2Fsunsetwx.com%2F
Sunsets, Fraternities, and Deep Learning http://obsessionwithregression.blogspot.com/2016/05/sunsets-fraternities-and-deep-learning.html

Image-to-Image Translation with Conditional Adversarial Networks https://arxiv.org/abs/1611.07004

B. Air Quality Index - Pollution Levels NYC

Air quality index in NYC 5 and ozone at current concentrations in New York City. Health Department estimates show that each year, PM2. 5 pollution in New York City causes more than 3,000 deaths, 2,000 hospital admissions for lung and heart conditions, and approximately 6,000 emergency department visits for asthma in children and adults.