Internal HTTP Client¶

This stage supports HTTP requests through GET, POST, or PUT, and supports caching (generating keys with URL and request parameters) for further queries.

Configuration¶

The configuration tabs for this stage are General, Basic, Input/Output, Request, and QueryConfig.

General¶

Name	Required?	Description
Name	Yes	The name of the stage.
Description	No	The description of the stage.
Stage Library	Yes	The streaming operator library to which the stage belongs.
Required Fields	No	The fields that the data records must contain. If the specified fields are not included, the record will be filtered out.
Preconditions	No	The conditions that must be satisfied by the data records. Records that do not meet the conditions will be filtered out. For example, `${record:value('/value') > 0}`. For the syntax of EL expressions, see `Expression Language https://docs.streamsets.com/portal/datacollector/latest/help/datacollector/UserGuide/Expression_Language/ExpressionLanguage_overview.html>`__.
On Record Error	Yes	The processing method for error data. Discard: Error data will be discarded and ignored Send to Error: Error messages will be reported Stop Pipeline: The pipeline will be stopped

Basic¶

Name	Required?	Description
Enable Cache Fields	No	Specify whether to enable caching. Local cache enables fast data processing. When the pipeline is paused, restarted, or retried, the cached data will be lost.
ExpireRule	No	Select the expiring time of the cache.
Quality Filter	No	Filter the data according to the data quality. Only records that meet the quality conditions will be processed by this stage.

Input/Output¶

Name	Required?	Description
Input Point	Yes	Specify the input point of the records, using the format {modelId}::{pointId}.
Output Point	No	Specify the output point of the records, using the format {modelId}::{pointId}.

Request¶

Name	Required?	Description
Request Method	Yes	Select the API request method. + Get + Post + Put
Request URL	Yes	Specify the API request URL.
Param Generate: Position	Yes	Specify the position of the parameters and values that are required by the API request. + URL Path + URL Query + Request Body + Header
Param Generate: Param Key	Yes	Specify the parameter name.
Param Generate: URLParam Type	Yes	Select the parameter type. + ByField. If you choose byField, complex formats such as List/Map are not supported, and the value type of the passed parameter will be forced to String. + ByInput
Param Generate: Param Value	Yes	Specify or select the value of the parameter.
Access Key	Yes	Enter the access key of the application, which is used for API request authentication (the access key can be generated by registering the application on EnOS Management Console).
Secret Key	Yes	Enter the secret key of the application.

QueryConfig¶

Name

Required?

Description

Parallel Number

Yes

Select the method of parallel requests. Based on the limit of API service, options are System Default and User Defined.

System Default: The default number of parallel requests that are supported by the system.
User Defined: Customized number of parallel requests. Enter a specific number in the UserDefined field (If the provided number exceeds the limit of the stage configuration, an error will be reported when starting the pipeline).

Enable Retry

No

Specify whether to retry upon request failure, and enter a specific number of retry attempts in the Retry Attempts field.

Cache Filter

No

When cache is enabled, specify conditions for filtering the cache. The default value is true. Aviator expressions are supported. For example:

code == 200
(code != 500) && (msg == "ok")

Output Results¶

The output results of this stage are included in the lookupTag field of the attr struct. The description of the fields are as follows:

Name	Data Type	Description
hasData	Boolean	Indicates whether the API request is successful.
checkSuccess	Boolean	Indicates whether the response matches the expression specified in Cache Filter.
response	Object	If the request is successful, displays the response data. The returnBy field indicates if the response is returned by Redis or HTTP.
cacheKey	String	The request URL composed of `Request URL + Param Generate configuration`.

Output Example¶

../../../_images/http_lookup_result1.png