Skip to content

Smart Flow

About The Project

This section is part of the tasks developed within the PGTEC project. Its main objective is to describe and provide the infrastructure required to deploy a data space using FIWARE technologies, offering a detailed and easy-to-follow guide adaptable to different environments. Specifically, it explains the logic used to download data from multiple sources, transform it into a common language using Smart Data Models, and make it available in two complementary ways:

  • Airflow Workflows: Python scripts that use Airflow logic to automate the execution of data retrieval and transformation tasks, ensuring that climate predictions are periodically processed and ready to use.

  • FastAPI Services: Python scripts built with FastAPI that expose the processed data through RESTful APIs. These services feed a dashboard where users can select the data sources and points of interest required to run the TETIS hydrological model. Once the user makes a selection, TETIS automatically triggers the corresponding FastAPI Python scripts described in this section to retrieve and process the climate predictions, which are then used as input for the model’s execution.

This section specifically describes the Python scripts used to:

  • Retrieve data from multiple climate data sources such as AEMET, CHJ, Open-Meteo, and Copernicus.

  • Convert the raw data into FIWARE Smart Data Models to standardize the format.

  • The creation of automated Airflow DAGs for pipeline execution.

  • The creation of FastAPI scripts to provide data as a service neede to provide a dashboard where users select the data sources to run TETIS model (Hydrological model from IIAMA-UPV)

All the python files are in the SmartFlow Github Repository

Built With

The project is built using the following main components:
Python Airflow Smart-data-models Docker

Getting Started

To get a local copy up and running follow these simple steps in ubuntu command line:

  1. Clone the repo and navigate to the project folder
git clone https://github.com/PGTEC-VRAIN/SmartFlow
cd SmartFlow
  1. Initialize docker:
sudo systemctl start docker
  1. Initialize docker containers 
    docker compose up --build -d

Airflow Data Sources Overview

This section contains links to detailed explanations of the Python scripts that programmatically download forecasts from different models using Airflow. Specifically, each script description includes:

  • The name of the Python file in the SmartFlow Github Repository

  • The data source accessed and the variables of interest

  • The Smart Data Model used to standardize the data

  • The required API key (if applicable)

  • The suggested execution frequency to keep the data up to date

Usage

This section describes how to run and test the main components of the PGTEC platform — the Airflow workflows for data processing and the FastAPI services for data delivery and dashboard integration. Both components can be deployed together using the provided Docker Compose environment.

It is supposed that the enviroment has been cloned following the instructions of Getting Started section.

1. Running Airflow Workflows

The Airflow DAGs automate the process of retrieving and transforming climate and hydrological data from different sources (AEMET, CHJ, Open-Meteo, Copernicus...).

Steps

1.1 Start the containers:

docker-compose up -d

The -d option runs the containers in detached mode, hiding Airflow logs and keeping the terminal clean.

1.1. Access the Airflow web interface:

http://localhost:8080

1.2. Enable the DAGs:

Inside the Airflow UI, activate the desired workflows (e.g., AEMET_HARMONIE_AROME, AIFS_ECMWF, etc.).

1.3. Monitor execution:

You can visualize the data extraction and transformation progress directly in the DAG view.

Below is an example of a Python script running in the Airflow user interface, showing the logs of its execution:

Airflow DAG example

In the screenshot, the DWD_ICON.py workflow is being executed. In just 4.04 seconds, it retrieves several points of interest from the Valencian Community and stores them using the WeatherForecastSeries Smart Data Model format.

2. Running FastAPI scripts.

The FastAPI services expose the processed data as REST APIs, allowing other applications — such as the TETIS dashboard — to access the latest standardized data stored in the FIWARE Context Broker.

Each service corresponds to a specific data source or Smart Data Model and can be easily extended to include new ones.

Steps

2.1. Make sure the Docker environment is running:
docker-compose up -d
2.2. Access the FastAPI documentation:

http://localhost:8000/docs

This interface allows you to explore and test all available endpoints interactively.

2.3. Test an endpoint:

For example, you can retrieve the latest weather forecast data by calling:

GET /weather?source=AEMET&variable=temperature
2.4 Integration with the TETIS dashboard:

The FastAPI services feed the TETIS dashboard, allowing users to select:

  • The data sources (e.g., AEMET, DWD, ECMWF)

  • The points of interest (catchments, stations, or coordinates)

Once the user selects these options, TETIS automatically triggers the corresponding API calls — which execute the same Python scripts described in the SmartFlow section — to retrieve and process the forecast data used as inputs for hydrological simulations.

Example folder structure

SmartFlow/FastAPI/
├── main.py           # FastAPI entry point
├── routes/
│   ├── AEMET.py      # Endpoints for WeatherObserved / Forecast data
│   ├── DWD_ICON.py   # Endpoints for CHJ flow data
│   └── ...
└── models/
    ├── AEMET.py
    └── DWD_ICON.py

New endpoints

You can add new endpoints easily by creating a new Python file in the routes/ folder and registering it in main.py.

License

Distributed under the AGPL-3.0 License. See LICENSE for more information.

Contact

Project Link: https://github.com/PGTEC-VRAIN

References