Each configuration in Meiro Integrations is an independently running Docker container with a predefined folder structure. All files with data and files created in the data flow are saved in specific folders.
The structure of all directories in your configuration:
/data config.json /in /tables /files /out /tables /files
Depending on the type of component that you are using, the structure can be expanded, as explained below.
/data is a root directory of your configuration and all working files and folders are saved here.
The root directory of Meiro Integrations contains:
- A JSON file
config.json.containt all the parameters of your configuration.
Refer: to this article to learn more.
/infolder ( an input bucket for the configuration).
/outfolder ( an output bucket for the configuration).
Optional files and folders
.rscript files in Python processor or R processor configurations respectively. Meiro Integrations keeps code from the script field in the configuration in the file
For Python and R processors folder structure would look like:
/data config.json script.py (or script.r) /in /tables /files /out /tables /files
/repositoryfolder, where Meiro Integrations clones git repository if you are using Python Git processor or R Git processor components.
For Python from GIT and R from GIT processors folder structure would look like:
/data config.json /in /tables /files /out /tables /files /repository
/data/in directory, with its subfolders, is an input bucket containing input files for the configuration.
When using a processor or a loader, you will notice that all output files from the previous configuration in your data flow that are connected to your processor or loader are automatically saved in the bucket
data/in and the corresponding subfolders.
For connectors, this folder will remain empty.
/data/out directory, with its subfolders, is an output bucket for output files of the configuration.
All files from the
/data/out folder and its subfolders will be saved in the input bucket (and its corresponding subfolders) of the next component of your data flow.
Data extracted in the connector components will be saved in the
/data/out folder once the configuration is run.
Data transformed inside the processor components with the use of scripts will be saved in the
For loader components, the data will be exported to a third-party application or a database and therefore the folder
data/out will remain empty.
Files and Tables
Both Data In and Data Out buckets consist of two subfolders:
tables is used for CSV tables,
files is used for other types of files.
However, file storage may vary depending on the connectors and loaders that you choose. For example, the AWS S3 connector component saves all the files regardless of their type into the
/data/out/files and the AWS S3 loader requires to have all files and tables in the folder
Depending on the component, it is possible to save the data in subfolders of
tables folders as
Names of the tables and files should contain only alphanumeric characters (`a-z`, `A-Z`, `0-9`) and `_`, `-` symbols.
The process of creating subfolders varies from component to component. In order to learn more, please refer to the particular article from the Meiro Integrations user documentation.