Skip to content

Commit 9de8efe

Browse files
Linh NguyenGitHub Enterprise
authored andcommitted
Merge pull request #6524 from AdobeDocs/PLAT-244435/How-to-Enable-Partial-Ingestion-and-Error-Diagnostics-for-an-Existing-Dataflow
PLAT-244435/How-to-Enable-Partial-Ingestion-and-Error-Diagnostics-for-an-Existing-Dataflow[READY]
2 parents 22fd149 + b2ec976 commit 9de8efe

File tree

1 file changed

+100
-0
lines changed

1 file changed

+100
-0
lines changed

help/ingestion/batch-ingestion/partial.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,106 @@ The **[!UICONTROL Error diagnostics]** toggle only appears when the **[!UICONTRO
115115

116116
**[!UICONTROL Error threshold]** allows you to set the percentage of acceptable errors before the entire batch will fail. By default, this value is set to 5%.
117117

118+
## Enable partial ingestion and error diagnostics for an existing dataflow
119+
120+
If a dataflow in Experience Platform was created without enabling partial ingestion or error diagnostics, you can still enable these features without recreating the flow. By enabling partial ingestion and robust error diagnostics, you can greatly enhance the reliability and ease of troubleshooting in your data ingestion workflows. Read the sections below to learn how to enable partial ingestion and error diagnostics for an existing dataflow using the [!DNL Flow Service] API.
121+
122+
By default, dataflows may not have partial ingestion or error diagnostics enabled. These features are helpful for identifying and isolating issues during data ingestion. Using the [!DNL Flow Service] API, you can retrieve your current dataflow configuration and apply the necessary changes using a PATCH request.
123+
124+
Follow the steps below to enable partial ingestion and error diagnostics for an existing dataflow.
125+
126+
### Retrieve flow details
127+
128+
To retrieve your dataflow configurations, make a GET request to the `/flows/{FLOW_ID}` endpoint and provide the ID of your dataflow. For more information on retrieving dataflow details, refer to the [Update dataflows using the [!DNL Flow Service] API](../../sources/tutorials/api/update-dataflows.md) guide.
129+
130+
Make sure to save the value of the `etag` field returned in the response. This is necessary for the update request to ensure version consistency.
131+
132+
### Update flow configuration
133+
134+
Next, make a PATCH request to the `/flows/` endpoint and provide the ID of the dataflow that you want to enable partial ingestion and error diagnostics for.
135+
136+
>[!IMPORTANT]
137+
>
138+
>- Include the previously saved `etag` value in the request header using the If-Match key.
139+
>- You can modify the `partialIngestionPercent` value to suit your specific needs.
140+
141+
**API format**
142+
143+
```http
144+
PATCH /flows/{FLOW_ID}
145+
```
146+
147+
**Request**
148+
149+
```shell
150+
curl -X PATCH \
151+
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
152+
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
153+
-H 'x-api-key: {API_KEY}' \
154+
-H 'x-gw-ims-org-id: {ORG_ID}' \
155+
-H 'x-sandbox-name: {SANDBOX_NAME}'
156+
-H 'If-Match: "1a0037e4-0000-0200-0000-602e06f60000"' \
157+
-d '[
158+
{
159+
"op": "add",
160+
"path": "/options",
161+
"value": {
162+
"partialIngestionPercent": "10"
163+
}
164+
},
165+
{
166+
"op": "add",
167+
"path": "/options/errorDiagnosticsEnabled",
168+
"value": true
169+
}
170+
]'
171+
```
172+
173+
**Response**
174+
175+
A successful response returns your dataflow's `id` and an updated `etag`.
176+
177+
```json
178+
{
179+
"id": "2edc08ac-4df5-4fe6-936f-81a19ce92f5c",
180+
"etag": "\"2c000802-0000-0200-0000-613976440000\""
181+
}
182+
```
183+
184+
### Verify the update
185+
186+
After the PATCH is complete, make a GET request and retrieve your dataflow to verify that the changes were successfully completed.
187+
188+
**API format**
189+
190+
```http
191+
GET /flows/{FLOW_ID}
192+
```
193+
194+
**Request**
195+
196+
The following request retrieves updated information regarding your flow ID.
197+
198+
```shell
199+
curl -X GET \
200+
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
201+
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
202+
-H 'x-api-key: {API_KEY}' \
203+
-H 'x-gw-ims-org-id: {ORG_ID}' \
204+
-H 'x-sandbox-name: {SANDBOX_NAME}'
205+
```
206+
207+
**Response**
208+
209+
A successful response returns your dataflow details, confirming that partial ingestion and error diagnostics are now enabled in the `options` section.
210+
211+
```json
212+
"options": {
213+
"partialIngestionPercent": 10,
214+
"errorDiagnosticsEnabled": true
215+
}
216+
```
217+
118218
## Next steps {#next-steps}
119219

120220
This tutorial covered how to create or modify a dataset to enable partial batch ingestion. For more information on batch ingestion, please read the [batch ingestion developer guide](./api-overview.md).

0 commit comments

Comments
 (0)