Stream Processing Resource¶
Note
This section refers to Enterprise Data Platform > Stream Processing. If you are using an older version of this service, see Stream Processing Resource (Old Version)
The stream processing service can fully satisfy the processing needs of the asset and device real-time data as well as the historical data that is integrated through offline message channels.
Based on Apache Spark™ Streaming, customized and optimized by Envision, the EnOS stream processing service has high scalability, high throughput, and high fault tolerance. EnOS is also committed to adopting common algorithms for streaming processing in the IoT field, enabling developers to complete the development of streaming processing tasks through simple template configurations.
In addition, the streaming processing service has adopted multiple sets of calculation templates and general operators in the energy field, helping developers to develop data processing solutions quickly without the need to code, thereby greatly improving data development efficiency and lowering the development threshold.
For more information, see Stream Processing.
Resource Application Scenario¶
Before installing the stream processing templates and StreamSets calculator libraries, or before configuring stream processing jobs, you need to apply for the Stream Processing resource. There are three resource types for Stream Processing resources, namely: Stream Designing, Cluster Processing, and Standalone Processing.
Stream Designing: Stream Designing resources are used to create and design stream processing tasks. When performing native drag-and-drop stream task designing and debugging, you need to temporarily run related tasks or install the related library packages.
Cluster Processing: When the amount of stream set is large and the processing performance is high, you can use the Cluster Processing resource.
Standalone Processing: When the amount of stream set is small and the cost control is strict, you can use the Standalone Processing resource.
The stream processing resources can be requested based on the computing unit (CU). Different specifications under different resource types correspond to different data processing capabilities. In the same resource type, the higher the specification, the higher the processing efficiency, and the larger the amount of data processed per unit time. The resource specifications and corresponding data processing capabilities are as per the below.
Stream Designing¶
Specification |
Computing Capability |
Reference |
|---|---|---|
Standard |
1CU |
1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 3 lib packages. |
Standard X2 |
2CU |
1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 6 lib packages. |
Standard X3 |
3CU |
1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 9 lib packages. |
Standard X4 |
4CU |
1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 12 lib packages. |
Cluster Processing¶
Specification |
Computing Capability |
|---|---|
Standard |
|
Standard X 2 |
|
Standard X 3 |
|
Standard X 4 |
|
Standard X 5 |
|
Standard X 6 |
|
Standard X 7 |
|
Standard X 8 |
|
Note
Reference: 1CU = 1 core CPU + 2GB Memory. For simple data processing jobs such as single-pipeline filtering and string conversion, 1CU cluster processing resource can process 4,000 - 11,000 records per second. 1CU cluster processing resource can run 1 pipeline.
Standalone Processing¶
Specification |
Computing Capability |
|---|---|
Standard |
4CU |
Standard X 2 |
8CU |
Standard X 3 |
12CU |
Standard X 4 |
16CU |
Standard X 5 |
20CU |
Standard X 6 |
24CU |
Standard X 7 |
28CU |
Standard X 8 |
32CU |
Note
Reference: 1CU = 1 core CPU + 2GB Memory. For simple data processing jobs such as single-pipeline filtering and string conversion, 1CU standalone processing resource can process 4,000 - 11,000 records per second. 1CU standalone processing resource can run 2 pipelines.