Data Format Specification¶

The main change in this operator version update is the addition of support for new data formats.

Old data format:
New data format:

As shown in the figure, the most major differences between the old and new data formats are:

In the old data format, the operator usually combined the fields modelId and pointId in the data to filter the data.
The new data format has a separate measurementId field that can be used to filter the data.

This measurementId field corresponds to the Measurement metadata in Data Catalog service. You can Create a Measurement and get measurementId in Data Catalog service.

For the Measurement metadata, you can model directly in the Data Catalog service without device modeling in IoT Hub. The data accessed to the platform can be consumed by other data products according to the specified data format.

In combination with the new Record Formatter operator, tagging of measurement points and assets is supported. Further, with the new Record Filter operator, expressions can be used to filter the data more flexibly based on the tag information of data.

Tens of old operators were adapted in order to support the new data format throughout the operator functionality.

The name of the adapted operator is the same as the old one, and the icon of the adapted operator in the new format is marked with a * in the upper right corner (the adapted operator name is also marked with a * in this document to make it easier to distinguish).

In terms of calculation logic, the operator remains basically unchanged after adaptation, except that the input/output configuration (Input/Output) uses measurementId instead of modelId::pointId, and for more details, please refer to the corresponding operator documentation.

Below we will further display the differences between the two old and new formats before.

Old Data Format (ModelId::PointId)¶

{
   "orgId":"o1234567890",
   "modelId":"inverter",
   "modelIdPath":"/rootModel/inverter",
   "pointId":"inverter",
   "assetId":"zabPDuHq",
   "time":1542609276270,
   "value":23.4,
   "quality":0,
   "dq":0,
   "attr": {}
}

Fields	Description
orgId	Organization ID
modelId	Model ID
modelIdPath	The full path of Model
pointId	Measurement points ID on the model
assetId	Asset ID
time	Timestamp
value	Measurement point value
quality	The quality point on the measurement point, without a quality point the value defaults to 0
dq	Optional field, Quality Filter use for data quality operators and other operators
attr	Used to store information such as operator calculation results, default is empty JSON Object

New Data Format (MeasurementId)¶

{
   "orgId":"o1234567890",
   "modelId":"legacyModelId",
   "modelIdPath":"/legacyModelId",
   "pointId":"legacyPointId",
   "assetId":"zabPDuHq",
   "assetTags": {"DCMModel":{"DCMModel::ModelA":{}}},
   "measurementId":"measurement1234",
   "measurementTags": {"Haystack":{"Haystack::temp":{"temp-attr":"dummy"}}},
   "time":1542609276270,
   "value":23.4,
   "quality":0,
   "dq":0,
   "attr": {}
}

Fields	Description
orgId	Organization ID
modelId	Model ID，Legacy field. The field is no longer used by operators adapted to the new format, except for a few operators such as Record Formatter, where the correctness of the field value is guaranteed by the user.
modelIdPath	The full path to the Model ID. Legacy field, adapted to the new format of the operator, which is no longer used except for a few operators such as Record Formatter, where the correctness of the field value is guaranteed by the user.
pointId	Measurement point ID on Model, legacy field. This field is no longer used by operators that are adapted to the new format, except for a few operators such as Point Lookup*, where the correctness of the field value is guaranteed by the user.
assetId	Asset ID
assetTags	Tag associated with asset ID, optional field, tag found by Record Formatter operator
measurementId	Measurement point ID, which can be associated with Measurement metadata in the Data Asset Catalog
measurementTags	Tag associated with the measurement point ID, optional field, the tag is found by the Record Formatter operator.
time	Timestamp
value	Measurement point value
quality	The quality point on the measurement point, without a quality point the value defaults to 0
dq	Optional field, Quality Filter use for data quality operators and other operators
attr	Used to store information such as operator calculation results, default is empty JSON Object

Operators List¶

Operator Categories	Operator Name	Support for old data formats（ModelId::PointId）	Support for new data formats（MeasurementId）
Data Source Operators	EDH Kafka Consumer	o	o
	Dev Raw Data Source	o	o
Asset Metadata Operators	Asset Lookup	o
	Child Asset Lookup	o
	Parent Asset Lookup	o
	Point Lookup	o
	Asset Lookup*		o
	Child Asset Lookup*		o
	Parent Asset Lookup*		o
	Point Lookup*		o
Data Processing Operators	Record Sorter By Time	o	o
	Normalizer	o
	Point Selector	o
	Record Generator	o
	Virtual Record Generator	o
	Related Record Obtainer	o
	Partitioner	o
	Fixed Time Window Aggregator	o
	Sliding Time Window Aggregator	o
	Batch Merger	o
	Last Changed Record Appender	o
	Latest Record Merger	o
	Record Capturer	o
	Record Restorer	o
	Point Selector*		o
	Fixed Time Window Aggregator*		o
	Sliding Time Window Aggregator*		o
	Batch Merger*		o
	Last Changed Record Appender*		o
	Latest Record Merger*		o
	Record Capturer*		o
	Record Restorer*		o
Other Data Processing Operators	Python Evaluator	o
	JavaScript	o
	Internal HTTP Client	o
	Python Evaluator*		o
	JavaScript*		o
	Internal HTTP Client*		o
	Record Filter		o
	Record Formatter		o
Data Quality Operators	Late Point Tagger	o
	Off Limit Tagger	o
	Late Point Tagger*		o
	Off Limit Tagger*		o
Power Calculation Operators	Last Record Appender	o
	Cumulant Decomposer	o
	Simplified Time Window Aggregator	o
	Last Record Appender*		o
	Cumulant Decomposer*		o
	Simplified Time Window Aggregator*		o
Target Operators	EDH Kafka Producer	o	o
	Trash	o	o