Data Backfill

Edge scenarios often face disconnection and weak network problems, which will cause communication obstruction, data backlog or loss. EnOS Edge supports data backfill in various scenarios, that is, Edge caches data in a network disconnection and continues to transmit cached historical data after communication resumes. Taking the data backfill between EnOS Edge and EnOS cloud as an example, data backfill can avoid data loss caused by communication failures, and ensure that applications in EnOS cloud can calculate and analyze historical data over a period of time. The data processing method is shown in the following figure.

../_images/backfill.png

Application

It supports data backfill between the following systems:

  • EnOS Edge - EnOS Cloud

  • EnOS Edge - EnOS Edge

  • Edge SCADA - EnOS Edge

  • Third Party SCADA - EnOS Edge

  • EnOS Edge - Third Party SCADA

Features

EnOS Edge data backfill has the following features:

  • Provide measurement tags to process historical data ( discard, re-filled through real-time topic or re-filled through historical topic)

  • Provide data compression function to restore more data

  • Support data feed strategy to insure real-time data priority

Configuration Items

The default values of configuration items for data backfill can meet the requirements of common scenarios. If there are additional usage requirements, the following items can be changed during the deployment process.

Item

Description

Default Value

config-manager.box.data.history.switch

Whether to enable history channel (enabled by default after v2.3.0)

True

config-manager.box.break.point.realtime.data.topic.capacity

The capacity of real-time Topic is used to set the storage capacity

100 GB

config-manager.box.break.point.history.data.topic.capacity

The capacity of historical Topic is used to set the storage capacity

100 GB

config-manager.box.break.point.topic.timeout

The maximum cache time (historical and real-time data retention time when resuming a breakpoint）is used to set the storage capacity

720 hours

Note

It is recommended to set the storage capacity according to your actual situations. The upper limit of the actual storage Topic capacity takes the smallest value among the Topic capacity, disk size, and the maximum cache time.