Data Format

Once the File Format is selected, all the related configurations will appear.

Parquet

A parquet file is a type of column-oriented file format where the data in each column is stored independently. It contains a table-type format to store the data.

Delimited

A delimited file is a text file format where a delimiter, such as a comma, tab, pipe, etc, separates the data in each column.

Selection of Custom enables the Custom Delimiter.

Custom Delimiter: Allows to define a new custom delimiter for the data.

Escape Character

An Escape character and a Text Qualifier create a sequence recognized and ignored during parsing. Its purpose is to allow the use of escape sequences in the data that would otherwise be seen as delimiter occurrences. 

The character preceded by a backslash (\) is known as an escape character.

Custom Delimiter: Allows to define a new custom delimiter for the data.

Text Qualifier

Text qualifier is used if delimiters are contained within the row cell. If the cell contains a delimiter and a text qualifier is not used, then the data that occurs after the delimiter will spill into the next column.

The Text Qualifiers are single (‘‘) or double (““) quotation marks around data elements to identify the element as text- or character-based data.

A text qualifier is a character used at the beginning and end of a field value. If the data has single or double quotation marks used to enclose the delimiter character or any special character, the same must be defined in the text qualifier.

JSON

A plain text file is written in the syntax of JSON

Not Supported Structure

1. JSON files with formatted records (that span over multiple rows).

   {
      "first_name":"Rachel",
      "priority":2619,
      "subscribe":false,
      "income":435324.12,
      "address":{
         "City":"Smithstad",
         "State":"Michigan"
      },
      "countries_visited":[
         "Belize",
         "Eritrea",
         "Egypt"
      ],
      "date_of_birth":"1976-06-19 00:00:00",
      "null_key":null
   }

2. JSON contains data in arrays.

   [{"id":1,"name":"John Doe","email":"john.doe@example.com"},{"id":2,"name":"Jane Doe","email":"jane.doe@example.com"},{"id":3,"name":"Mike Smith","email":"mike.smith@example.com"}]
   [{"id":7,"name":"Peter Green","email":"peter.green@example.com"},{"id":8,"name":"Susan Black","email":"susan.black@example.com"},{"id":9,"name":"Michael White","email":"michael.white@example.com"},{"id":10,"name":"Jessica Green","email":"jessica.green@example.com"}]

The View Batch Schema button gets enabled after uploading the selected file.

View Source Schema: Allows you to view and modify the selection of source schema.

Source Data Format

All the available parent column names in the uploaded data file will appear here.

Filter: It allows you to search and filter the specific columns here. Scroll down manually to see the entire list.

Column list: As required, you can select or deselect the required columns of the left-hand side drawer. All the columns available in the source data format are selected by default.

JSON: JSON schema will update dynamically when columns/keys are selected or deselected from the left-hand side drawer. It displays the standardized schema structure that needs to be mapped in the project schema.

Users can only view the standardized schema structure.

• After changes, copy the created entire schema and paste it into the defined project schema at the right-hand side drawer.
• Click Save Changes to save the configuration.

Related Topics

• Batch Listener
• Path Pattern (Regex)
• Data Format
• Interval 
• Output