• Documentation
  • Synchronizing Data Assets

Synchronizing Data Assets

Massive data is the primary element composing the Data Catalog service. With the Sync Task function, users can synchronize data from the access side as well as and multiple other data sources to the Data Catalog, from where the data can then be managed.

Create Sync Task

The Data Catalog service allows users to create the following data sync tasks:

  • System Synchronization Tasks: Provides built-in data sync task types, including common data sync tasks such as synchronizing data sources, assets, Hive tables, and measurements. Users can choose to create one or more system sync tasks as needed.

  • User Synchronization Tasks: Users can configure their own data sync tasks through Open API. This gives users a higher degree of freedom to select data sources for sync.

Creating a System Sync Task

  1. In the left navigation bar of the Data Catalog, click Synchronization Tasks.

  2. Click Create Task> System Synchronization Task.

  3. Select a system sync task from the options provided:

    • Sync Data Source: Synchronize data from different data sources to the Data Catalog. Internal data sources (Hive/HDFS/TSDB) are automatically obtained by the system, while external data sources must be registered by the user. For related documents about registering external data sources, see Data Source Registration.

    • Sync Tables from EnOS Hive: Synchronizes data from Hive data sources in the EnOS system to the Data Catalog.

    • Sync Points from DCM: Synchronizes measurement point data from the IoT Hub to the Data Catalog. If you choose this option, you will need to then select the model to be synchronized.

    • Sync Device Asset from DCM: Synchronizes device data from the IoT Hub to the Data Catalog.

    • Sync Asset Tree Node from DCM: Synchronizes asset tree node data from the IoT Hub to the Data Catalog.

  4. Click OK to create the system sync task.

Once the system sync task is created, the new system sync task will appear on the Sync Task function homepage.

Note

When the system sync task synchronizes data from the IoT Hub, if the data has its own tags, the data tags will be synchronized to the Data Catalog and automatically classified according to the tag group. For a description of system tag groups, see System Tag Groups.

Creating a User Synchronization Task

Note

Before using the User Synchronization Task, please make sure that the relevant data source has been registered using the Data Source Registration service. For more on data source registration, see Data Source Registration.

  1. In the left navigation bar of the Data Catalog, click Synchronization Task.

  2. Click New Task > User Synchronization Task.

  3. On the Create User Synchronization Task page, enter the following information:

    • Synchronization Task name: enter the sync task name, no more than 100 characters.

    • Data Source Type: select a data source type.

      Note

      Currently only Open API types are supported. Therefore, before using the User Synchronization Task function, please make sure that the Open API data source has been registered using the Data Source Registration service. For more on Open API data source registration, see Registering an Open API Data Source.

    • Data Source*: select a data source (the options are the data sources configured through the **Data Source Registration service)

    • Data Sync Method: tick the data acquisition method.

      • Full-load DataAll data will be synchronized every time and completely overwrite existing data

      • Incremental DataOnly the portions different from existing data will be synchronized each time

    • Request Parameters: If the request parameters have been configured for the API in the Data Source Registration, fill in the corresponding request parameters in this item. If not configured, this item is displayed as empty.

    • Content to Sync: select content to synchronize

      • If you select entity synchronization, you also need to select a master data object type

        • Info Mapping: fill in the returned field in the master data object’s attribute field

        • Data Checktick whether to start synchronous real-time data verification.

          If enabled, during the sync process, the data will be verified by the verification rules set when the object type is created. Alerts will be sent when data that does not meet the verification rules is found (verification rule alerts are turned off by default, and need to be enabled in the sync task’s alert settings).

          For more on verification rule configuration, see Object Type Creation - Verification Rules.

          For more on enabling and configuring verification rule alerts, see Sync Task Configuration - Alert Configuration.

      • If you select entity relationship synchronization, you also need to select an object relationship type

        • Content Mapping: fill in the returned fields of the object model and object primary key

  4. Click OK to create the User Synchronization Task.

Once the User Synchronization Task is created, the new User Synchronization Task will appear on the Sync Task function homepage.

Note

You need to manually start the sync task before the data will start to be synchronized. For how to start sync tasks, see Managing Task Status.

Sync Task Configuration

When creating a sync task, the Data Catalog service is set by default to open running configuration for sync tasks, and alerts are turned off by default. Users can change the running configuration and alert configuration as needed.

Running Configuration

For running configuration, the sync interval and running resources of the sync task can be configured.

  1. In the data sync task list on the Synchronization Tasks function homepage, click the ellipsis icon more_icon at the end of the sync task row to start running configuration.

  2. Click Configure Resource to enter the task’s running configuration page.

    • Sync Interval: enter a positive integer

    • Resource: enter the running resource value (cannot exceed the remaining resource size stated in the text), the unit is CU.

    Note

    The total running resources for all running sync tasks and tag rules cannot exceed the total amount of resources configured for Data Synchronization. Under the Data Catalog service, the resources configured for Data Synchronization can be viewed in the Resource Management service, as shown in the following figure:

    ../_images/data_sync_resource.png

    For resource configuration of tag rules, see Tag Rules Running Configuration.

    For more on data catalog resources, see Data Catalog Resource Specification.

  3. Click OK to complete running configuration.

Alert Configuration

Alert configuration allows you to monitor the running status of the sync task. If an abnormality occurs during a sync task, the relevant users will be notified by email and SMS.

Note

For data sync tasks that start data verification when creating a User Synchronization Task, after alerts are enabled, alerts will also be sent for data that does not meet verification rules.

  1. In the data sync task list on the Synchronization Tasks function homepage, click the ellipsis icon more_icon at the end of the sync task row to enter alert configuration.

  2. Click Set Alarm to go to the task’s alarm configuration page.

    • Switch: click the switch icon switch_icon to set it to enabled slide_start_icon.

    • Alarm Mode: Mail is ticked by default and cannot be changed, but can be modified by ticking SMS.

    • Receiver: select the alert receiver (more than one can be selected). Click the small cross cross_icon next to the selected receiver’s name or click the receiver’s name again in the receiver drop-down option to untick the receiver.

    • Status Alarm: select the sync task status for which you need alerts (more than one can be selected). Click the small cross cross_icon next to the selected task status or click the task status again in the task status drop-down options to untick the task status.

    • Message Push:

      • To App: Select a push application, which can be configured under Notification Management > Quota. For more information on quota management, see Quota Management.

      • Email Template: click on the name of the email template to go to the corresponding template

      • SMS Template: click on the name of the SMS template to go to the corresponding template

  3. Click OK to complete alarm configuration.

Managing Sync Tasks

For the various sync tasks in the data sync task list, users can manage their running status, as well as view, edit, and delete them.

Managing Task Status

  • Starting Sync Tasks: In the data sync task list on the Synchronization Tasks function homepage, click Start start_icon at the end of the sync task row to start the data sync task.

  • Pausing Sync Tasks: In the data sync task list on the Synchronization Tasks function homepage, click Pause pause_icon at the end of the sync task row to pause the data sync task.

  • Stopping Sync Tasks: In the data sync task list on the Synchronization Tasks function homepage, click Stop end_icon at the end of the sync task row to stop the data sync task.

Viewing Sync Tasks

In the data sync task list on the Data Sync function homepage, you can view key information such as the name, type, data source, sync method, running configuration, and status of the sync task.

Click the name of a specific sync task to view its basic information.

Note

In the current version, only viewing Sync Points from DCM and Sync Device Asset from DCM are supported in User Synchronization Tasks and system sync tasks.

Editing Sync Tasks

Note

Editing is supported only for User Synchronization Tasks.

  1. In the data sync task list on the Synchronization Tasks function homepage, click the ellipsis icon more_icon at the end of the sync task row to enter editing.

  2. Click Edit to enter the sync task’s editing page.

  3. In the User Synchronization Task edit page, the Synchronization Task Name, Data Source Type, and Data Source cannot be edited, but you can edit and modify the data acquisition method, request input parameters, sync settings, etc.

  4. Click OK to complete editing of the User Synchronization Task.

Deleting Sync Tasks

Note

Only sync tasks with a status of Stopped and Paused can be deleted.

  1. In the data sync task list on the Synchronization Tasks function homepage, click the ellipsis icon more_icon at the end of the sync task row to enter deletion.

  2. Click Delete > OK to delete the task.