| When a stream pipeline keeps restarting due to issues with resource configuration and so on, the stream pipeline will be in the UNKOWN status. By default, in the UNKOWN status, any action including state restoration is not supported. |
Added PAUSE and STOP functions in the UNKOWN status for stream operation, enabling users and operators to reset the stream status. |
- During the upgrade process, the user pipeline and pipeline design will be restarted to adapt to the relevant changes, which is expected to last for 15-30 minutes.
- After installation, there is no direct impact on the system.
|
| High-level stream supports users to select input and output topics that they defined, but when importing across OUs, the OU ID suffix of the topic will not be automatically modified. Users must edit the stream manually before publishing, which is inconvenient. |
The topic selected in the high-level stream supported automatic replacement of the OU ID suffix when importing across OUs. |
- During the upgrade process, the user pipeline and pipeline design will be restarted to adapt to the relevant changes, which is expected to last for 15-30 minutes.
- After installation, there is no direct impact on the system.
|
| When high-level stream users select to output to CAL topic, the message partition expression of the original default KafkaProducer cannot adapt to the CAL topic message format, resulting in messages aggregating in one partition. |
Modified the default configuration of the partition expression of the high-level stream KafkaProducer to support partitioning by the assetId of the message format in INTERNAL and CAL topics, and enabled users to modify the partition expression configuration for the high-level stream KafkaProducer. |
- During the upgrade process, the user pipeline and pipeline design will be restarted to adapt to the relevant changes, which is expected to last for 15-30 minutes.
- After installation, there is no direct impact on the system.
|
| By default, the stream caches all model configurations of the OU to improve processing efficiency. However, when there is a large amount of data, there are many irrelevant models that take up a lot of memory and affect the stability of stream pipeline. |
Upgraded the DCM model service SDK in all operator versions, filtered irrelevant measurement points, and improved the performance of the operator model cache when there is a large amount of data. |
- During the upgrade process, the user pipeline and pipeline design will be restarted to adapt to the relevant changes, which is expected to last for 15-30 minutes.
- After installation, there is no direct impact on the system.
|
| If a cluster task is not configured correctly, the out of memory (OOM) executor will appear intermittently. By default, these executors are not removed until the task stops. For long-running tasks, the large amount of accumulated OOM executors will affect the stability of the cluster nodes. |
Upgraded the Spark mirror in StreamSets, supported automatic cleaning of OOM executors, and adjusted the number of garbage collection (GC) threads of Driver to improve the stability of cluster task operation in some scenarios. |
- During the upgrade process, the user pipeline and pipeline design will be restarted to adapt to the relevant changes, which is expected to last for 15-30 minutes.
- After installation, there is no direct impact on the system.
|
| For complex stream pipeline, when performance bottleneck occurs due to some uncertain reasons, it is difficult to determine which operators in the stream caused the bottleneck, making it hard to troubleshoot the problem. |
Increased the collection of indicators, such as the execution time of a single operator in the stream pipeline, to Prometheus to assist operators and developers in troubleshooting the performance problems. |
- During the upgrade process, the user pipeline and pipeline design will be restarted to adapt to the relevant changes, which is expected to last for 15-30 minutes.
- After installation, there is no direct impact on the system.
|