Machine Learning Stunning Afterglows
Image Machine Learning training using StyleGan model. Python coding & Tensorflow
OverviewI’m working with my own photos of “sunsets” which I’ve taken over the years. These photos have been taken in several countries and regions in the world. Sunset time is special to me and it gives me hope of a day gone and a new day coming. It’s also very dreamy and romantic to see how the sky’s colors change. Most of the sunsets in NYC are beautiful and the colors are stunning especially when it turns pink but this is only because of pollution. The more the air pollution, the more the sky turns pink when the sun is setting.
I hope to use the machine learning from the colors of the sky at sunset time to show variations of patterns in relation to pollution, and also use these patterns to predict more beautiful sunset skies.
- Air index quality affects afterglow colors in sunsets especially in populated and dense areas, where air pollution is higher.
- Comparing and contrasting between lesser density areas and afterglow colors.
- Leveraging this tool for creating unique stunning sunset backdrops for design.
1st dataset: 1470 photos, 629 frames extracted from 2 videos using opencv [by Aarati Akkapeddi https://colab.research.google.com/drive/1WWHNG4YqGSHfIYIUrC2tmoPQ3HOgU--e]
I have 2 videos which I transformed to images to add to the dataset. In total I planned to have around 2000 images for lengthy periods of training.
This dataset included photos taken in the U.S (several states), Lebanon, Spain, France, Turkey
Resized to 256x256
All in jpg format
2nd dataset: 787 frames, taken in New York City on June 25, 2019
extracted from 2 videos using opencv [by Aarati Akkapeddi]
Resized to 256x256
All in jpg format
- I trained dataset 1 for 20 hours
- Downloaded pkl files
- Generated a new dataset with generate.py code and selected names it dataset 2
- I started re-training on the same worksheet, trained dataset 2 for 16 hours
- Then used the last pickle file from dataset 1 training (000266.pkl) to train it with dataset 2 images. (cross pollination). Used the learning from the last dataset onto the latest training.
- Downloaded pckl file 000048 from the last combined training (14th).
Seeds from training dataset 1
Seeds from training dataset 2
- The Fine Styles — Color Scheme is preserved in outputs 1, 2, 3
- There's adaptive normalization in 1 & 2
- Comparison between the sunset output in Lebanon (mostly) and NY. Less dense areas were definitely lighter in sunset colors. While pink and orange were mostly in new york sunsets since the air is more polluted.
- StyleGAN has generated realistic images
- Slight loss in style and content in training 3
- I WANT TO INTERPOLATE USING DIFFERENT INPUTS AND MANIPULATE OUTPUTS USING EXISTING LEARNING TOOLS
- EXPERIMENT INTERACTIVITY THROUGH PROCESSING - P5.JS
- I AM DESIGNING SUNSETS, EACH DESIGN IS UNIQUE AND YOU CAN OWN IT
Applications to look into
- Image Classification
Upload a picture, and the model tries to classify it depending on what it “sees” on the picture. This model uses transfer learning and is based on MobilenetV2.
- Style transfer
Blending image style adding these sunset images as backdrops. Using image style to create a sunset.
- Learning and experimenting with deep-learning challenges and approaches for sky images
First images sample
ProcessI have a primary 870 images dataset. I began training with a portion of this set and hope to add all of the images to it.
I used the styleGAN training code and ran into a couple of errors in the first 3 trials after I finally was able to get an output.
The code is still running for a better outcome and I’ll upload more outputs in the upcoming week.
Early stage outcome
We can notice the clarity of the images becoming better with time & getting closer from looking real.
The below image looks interesting. Since most of the time the sky is positioned upwards and it seems like there’s a typical way we see sunsets, yet here it’s kind of rotated. I’m searching for more interesting findings; in general the colors and shapes seem close to the reality of true sunsets.
A. Implementation and Concept Research
I- What dust and pollution don't do
It is often written that natural and manmade dust and pollution cause colorful sunrises and sunsets. Indeed, the brilliant twilight "afterglows" that follow major volcanic eruptions owe their existence to the ejection of small particles high into the atmosphere (more will be said on this a bit later). If, however, it were strictly true that low-level dust and haze were responsible for brilliant sunsets, cities such as New York, Los Angeles, London, and Mexico City would be celebrated for their twilight hues. The truth is is that tropospheric aerosols --- when present in abundance in the lower atmosphere as they often are over urban and continental areas --- do not enhance sky colors --- they subdue them. Clean air is, in fact, the main ingredient common to brightly colored sunrises and sunsets.
Afterglow colors are affected by smoke, air quality, and other factors. The stunning hues when affected by smoke are mostly pinkish
II- Machine Learning which sunsets are considered beautiful from social media data
- Luminar: Sky replacement tool - AI Powered Tool - picture editor
- Adobe photoshop https://www.theverge.com/2020/9/21/21449124/photoshop-sky-replacement-tool-ai-machine-learning
- SunsetWx - Sunset and sunrise forecasts. Sunset & Sunrise Predictions: Model using an in-depth algorithm comprised of meteorological factors. https://sunsetwx.com/
- Sunsets, Fraternities, and Deep Learning http://obsessionwithregression.blogspot.com/2016/05/sunsets-fraternities-and-deep-learning.html
Image-to-Image Translation with Conditional Adversarial Networks https://arxiv.org/abs/1611.07004
B. Air Quality Index - Pollution Levels NYC
Air quality index in NYC 5 and ozone at current concentrations in New York City. Health Department estimates show that each year, PM2. 5 pollution in New York City causes more than 3,000 deaths, 2,000 hospital admissions for lung and heart conditions, and approximately 6,000 emergency department visits for asthma in children and adults.
- Reference: https://aqicn.org/map/brooklyn/
Get in touch for more info & resources on this project