Configuring Data Archiving Jobs¶

EnOS Data Archiving supports archiving and storing data from the real-time message channel, data from the offline message channels, and real-time alert records, and provides flexible storage configurations to reduce your data storage costs.

Prerequisites¶

Authorization to the Data Archiving module
OU has requested data archiving resource

Creating a Data Archiving Job¶

For archiving and storing data, you need to create data archiving jobs for asset models.

Log in the EnOS Console, and select Data Archiving under the Common Resource section.
If your OU has not requested data archiving resource, go to EnOS Console > Resource Management.
Click New Job, select the source of data to be archived, enter a name and description for the data archiving job.
Click OK to complete the basic setting and open the detailed configuration page of the data archiving job.

Note

At most 5 data archiving jobs can be created for an organization.

Basic Information¶

Edit the basic information of the data archiving job if needed:

Name: Name of the data archiving job, where Chinese characters, upper cases and lower cases, numbers, underline are supported, with a length limit of 50 characters.
Description: Description of the data archiving jobs, with a length limit of 100 characters.

Storage Configuration¶

Complete the data storage configuration, including configuring the storage resource, setting properties of the archived file, etc.

Select Resource Type and specify the target storage system (BLOB or HDFS) for synchronizing the archived files.
From the Storage Resource drop-down list, select the BLOB data source that has been registered through Data Connection or the HDFS storage that has been requested through Resource Management.
Input the Storage Path where the archive files are located in the storage system and select the time partition format for the storage path root directory. The path must start and end with “/”. For detailed information about time partition format, see Storage Path Partition Parameters.
Select whether to generate path by event time or by system time. For detailed description about how to generate directory by different time, see Data Archiving Logic.

Note

For archiving data from offline message channel and real-time alert records, it is supported to generate path by event time only.
Input the File Name of the archived file, where upper cases, lower cases, numbers, and hyphens are supported, with a length limit of 50 characters. Once a file is generated, the system will append a time stamp suffix “_UTC” after the file name. When “Generate path by event time” is selected, the time stamp suffix is UTC+0 corresponding to the data event time; when “Generate path by system time” is selected, the time stamp suffix is UTC+0 corresponding to the system time.
Select File Type, where only TEXTFILE format (.csv) is supported currently.
Select the Encoding for archived file, where the default format is UTF-8.
Select Column Delimiter, where the default delimiter is comma.
Select the Compression format for the archived file, where the default setting is “not compressed”.
Select the File Size for the archived file, that is, the file size limit before compression. If the size of a file exceeds the upper limit, the system will slice it for storage. The sliced files will be named in the form of filename_UTC_n, where n is a 6-digit random string.

Note

Once the data archiving job is submitted, the file type, encoding format, column delimiter, and compression format cannot be modified.

Archive Configuration¶

Select the Archive Cycle for the data archiving job. Currently, data archiving by every 1 hour is supported. If the archived data falls in the same archiving cycle, the data will be saved in 1 file and sliced by the specified file size.

Data Configuration¶

Select the Model to which the data to be archived belongs.

For real-time alert record archiving job:

Select the type of alert records to be archived, history alerts or active alerts. History alerts are recovered alert records, and active alerts are records not recovered yet.
Select the fields of alert records to be archived.

Submitting the Archiving Job¶

After completing the above configuration, click OK to submit the data archiving job. The job configuration will take effect immediately once it is submitted.

When the data archiving job is submitted, the system starts reading data from the specified message channel.
If the current data archiving job is not completed in the current archiving cycle, it will restart immediately per the newly-submitted configuration. Archived files will not be impacted. Data of the added models will also be archived immediately when the arching job is submitted.
For more information about how archived files are generated, see Generation of Archived Files.