How data flow and components work

In this article, we explain how the folder structure and execution model works in Meiro Integrations.

In Meiro Integrations, each workspace has its own data flow. The data flow displays a visual representation of the data loaded into Meiro Integrations, processed through processors and finally loaded to a data warehouse or optimised for a BI tool. Each data flow is presented as a DAG (Directed Acyclic Graph).

When creating the data flow, use components (nodes in the graph)  that either connect to the data source (connectors), process the data (processors) or load data to your chosen data destination (loaders). Arrows between the components show which direction the data is flowing and how components are connected to each other. 

 

You can read more about connectors, processors, loaders, components in the articles available online. 

In the example below, we will demonstrate a data flow that:

1. Downloads the table(s) from the AWS S3 bucket.
2. Uncompresses the data and moves it to the `tables` folder.
3. Filters the data.
4. Uploads the data to a Google Spreadsheet.