⚡Geospatial Lightning Atmospheric Dashboard

Today I am thrilled to announce the successful completion of my end-to-end data engineering project on :zap:Geospatial Lightning Atmospheric Data :chart_with_upwards_trend:.

:cloud_with_lightning: Overview:

I analyzed geo-located, time-tagged lightning event data using various open-source tools and technologies, including Prefect, Docker, SQLite/Spatialite, and Streamlit.

:cloud_with_lightning: Components:

:whale: Docker Container: Developed docker image for portability.
:100: Prefect 2.0: Seamlessly orchestrated and automated the data workflows.
:blue_book: Pandas: Leveraged for data exploration, transformation, and analytics.
:floppy_disk: SQLite with Spatialite extension: Stored and managed geospatial data for single user.
:balloon: Streamlit Dashboard app: GIS data viewer, filters, summary plots and charts.

:cloud_with_lightning: Achievements:

:ballot_box_with_check: Used some basic transformations using pandas.
:ballot_box_with_check: Conducted data analysis & visualization on weather datasets.
:ballot_box_with_check: Successfully handled large-scale data processing.
:ballot_box_with_check: Built a portable python data pipeline.
:ballot_box_with_check: Implemented robust data engineering workflows.

:cloud_with_lightning: Skills Enhanced:

:white_check_mark: Data Engineering
:white_check_mark: Data Analysis
:white_check_mark: Containerization
:white_check_mark: GIS Visualization

:cloud_with_lightning: Explore the Project:

:world_map: Dashboard app: https://lightning-containers.streamlit.app
:open_book: GitHub Repo: GitHub - BayoAdejare/lightning-containers: Docker powered starter for geospatial analysis of lightning atmospheric data.

:cloud_with_lightning: Acknowledgments:

Thanks to the open-source community for valuable tools and technologies and to US National Oceanic and Atmospheric Administration (NOAA) for the datasets.

:cloud_with_lightning: Looking Forward:

Excited about the journey ahead, exploring more opportunities, and continuously growing in the fascinating world of data engineering.

:cloud_with_lightning: Connection:

I am open to feedback, discussions, and connecting with fellow data & software engineer. Feel free to explore the project, share your thoughts, or connect with me for further discussions.

:cloud_with_lightning: Architecture:


1 Like