Enterprise Data Platform¶
In EnOS 2.2 Release, we renamed some of the products and services, and added new functions and features for Data Asset Catalog, Data Federation,Time Series Data Management, Stream Processing, and Batch Processing. Detailed information is as follows:
Product/Service Renaming¶
Starting from EnOS 2.2 Release, the following products of Enterprise Data Platform are renamed:
| Before | After |
|---|---|
| 数据联邦 -> 通道管理 / Data Federation -> Channel Manager | 数据联邦 -> 联邦通道管理 / Data Federation -> Federation Channels |
| 时序数据管理 -> 存储策略 / Time Series Data Management -> Storage Policy | 时序数据管理 -> 存储策略 / Time Series Data Management -> Storage Policies |
| 流数据处理 -> 流开发 / Stream Processsing -> Stream Development | 流数据处理 -> 流开发 / Stream Processsing -> Pipeline Designer |
| 流数据处理 -> 流运维 / Stream Processsing -> Stream Operation | 流数据处理 -> 流运维 / Stream Processsing -> Pipeline Operation |
| 流数据处理 -> 包管理 / Stream Processsing -> Streamsets Libs | 流数据处理 -> 流运维 / Stream Processsing -> Pipeline Library |
| 批数据处理 -> 数据开发 / Batch Processing -> Data Development | 批数据处理 -> 任务流开发 / Batch Processing -> Workflow Development |
| 批数据处理 -> 任务资源 / Batch Processing -> Job Resource | 批数据处理 -> 任务资源 / Batch Processing -> Task Resources |
| 批数据处理 -> 任务运维 / Batch Processing -> Workflow Operation | 批数据处理 -> 任务流运维 / Batch Processing -> Workflow Operation |
Data Asset Catalog (Preview)¶
The Data Asset Catalog product is redesigned, with a new feature “Metadata Synchronization” added.
For more information, see Data Asset Catalog.
Stream Processing¶
In EnOS 2.2 Release, Stream Processing service added support of configuring advanced parameters in Stream Operation, the V0.2.0 Streaming Calculator Library, cluster monitoring, and template version upgrade. Details are as follows:
Advanced Parameter Configuration in Stream Operation¶
Added support for the configuring of advanced parameters for running stream processing jobs by Cluster Mode. With the advanced features, you can now better control the running efficiency of stream processing jobs.
For more information, see Configuring Advanced Parameters.
Streaming Calculator Library¶
Added support of the V0.2.0 Streaming Calculator Library, which contains all the calculators in the V0.1.0 library and provides the following features:
- The JavaScript calculator supports more functions.
- The Internal HTTP Client calculator supports response parsing.
For more information, see Calculator Library 0.2.0 Documentation.
Cluster Monitoring¶
Added support for monitoring the usage of cluster resources. With the Cluster Monitoring feature, you can view the real-time usage of cluster resources by the stream processing service, which can be your reference when requesting computing resources for stream data processing.
For more information, see Monitoring Cluster Resources.
Template Version Update¶
Added support for updating the stream processing job templates. When the Stream Processing service provides new versions of calculator libraries or stream processing templates, you can update the templates that are used by existing stream processing jobs online. This can improve the running efficiency of stream processing jobs.
For more information, see Updating Stream Processing Template Version.
Time Series Data Management¶
In EnOS 2.2 Release, Time Series Data Management added support of lossy data compression, downloading and uploading point storage configuration, and configuring stored data modifiable attribute. Details are as follows:
Lossy Compression for TSDB Data Storage¶
Added support for the configuring of lossy compression for measurement point data that is stored in TSDB. When configuring storage policy for measurement points, you can choose to store the point data after compression to save storage resource.
For more information, see Compressing Stored Data.
Downloading and Uploading Point Storage Configuration¶
Added support for the downloading and uploading of the storage policy configuration of measurement points. When you need to migrate or update TSDB storage policy configuration for measurement points, you can download the existing TSDB storage configuration, update the downloaded Excel file, and then upload the edited file to update the configuration quickly.
For more information, see Downloading and Uploading Storage Configuration.
Configuring Stored Data Modifiable Attribute¶
Added support for disabling the default configuration of overwriting data records that are stored in TSDB with duplicate data records. If the business requires that the data saved in TSDB cannot be modified, you can specify that duplicate data records cannot overwrite the saved data records when configuring the storage policy of measurement points. When there are duplicate data records, the duplicate data will be discarded.
For more information, see Configuring Stored Data Modifiable Attribute.
Data Federation¶
In EnOS 2.2 Release, the Data Federation service added supported of visualized federation query, federation download, and cross-source analysis functions. Details are as follows:
Federation Query¶
Added a visual data query interface for querying data from various data sources through the created read channels. Deverlopers can use the visual interface to query data using SQL statements for application debugging and data analysis.
For more information, see Querying Data Online.
Federation Download¶
Application developers and data analysts can complete task configuration in the visual interface and download files from data sources through authorized channels.
For more information, see Downloading Files.
Cross-source Analysis¶
When starting a read channel, you can choose whether to enable the cross-source analysis function. When enabled, you can use unified SQL statements to perform cross-source queries through multiple data sources.
For more information, see Starting Channel.
Data Subscription¶
In EnOS 2.2 Release, we added support for asset OTA event data subscription, including events about OTA firmware files and OTA jobs. When creating a data subscription job for asset event data, you can now configure the job for subscribing to OTA event data.
For more information, see Developing Data Subscription Jobs.
Data Synchronization¶
In EnOS 2.2 Release, we added support of selecting data mapping methods when synchronizing data from BLOB, FTP, SFTP, or S3 to EnOS Hive. Available options are mapping data by column ID or column name. Mapping data by column name is applicable to the case that the synchronized files have new data fields, so the uploaded file must have table header.
For more information, see Synchronizing Data from External Data Source to Hive.
Batch Processing¶
In EnOS 2.2 Release, the Batch Processing service added supported of script development and workflow task node monitoring functions. Details are as follows:
Script Development¶
Added support of the script development feature for users to edit, debug, and manage versions of Hive SQL or Spark SQL scripts. Developed scripts can be used as task resources for running PYTHON task nodes in workflows.
For more information, see Script Development.
Workflow Task Node Monitoring¶
Added support of monitoring task nodes in Workflow Operation. By monitoring the resource usage of task nodes, users can dynamically change the resource configuration for workflow task nodes.
For more information, see Monitoring Task Nodes.