File Write Adaptor

Creating a File Write Adaptor

The File Write Adaptor writes contents from a graph to an object-based file.

Currently, only Amazon Web Services Simple Storage System (AWS S3) is supported, but it is planned to expand to other types (for example, Azure blob storage)

• AWS S3 is an object-based storage scheme that presents a file-system view of the objects that it stores; folders are termed buckets, and files are objects. For all intents and purposes, it appears like a web-hosted file system. The ability to read and write data from S3 buckets helps integrate with other systems that can only handle file-based integrations, such as DMPs. 
• To create a new item via the Create menu option in the project editor, select the File adaptor, and give the adaptor a name. 
• Choose the connection from the dropdown menu, and choose the file name and file contents destination.
• File Name: The File Name value can be provided by a literal, schema value, or public variables

• For S3, this name should include any path outside of the top-level bucket.
• A JavaScript Node can be used to generate a file name that uses a timestamp and a unique identifier to ensure that files are ordered by creation date and also that there is no clash or accidental overwriting if multiple engines are processing and writing files simultaneously. 

File Contents To Write: The contents to write to the file, which could be any part of schema value (recommended) or a public variable

• The file is created on each write and will overwrite any existing file without creating an error. It is, therefore, suggested that a unique identifier is used to generate the filename.
• For S3, buckets often contain many thousands of small objects rather than one large file - most third-party data loading processes are happy to process a large number of small files.

Validation Warnings

: All the invalid notifications will be displayed after expanding the Invalid Dropdown.

Related Pages:

• File Read Adaptor