logs_pipelines_order: how to use?

[thomas-riccardi]

Overview of issue:
(I'm the customer that asked support for logs_pipelines_order, thanks for the work.)

We got some issues using the new logs_pipelines_order resource, this issue is there to help future users, maybe improve documentation?

when running a diffs before the first sync, we get irrelevant warning

2023-10-09 12:07:49,477 - INFO - Skipping resource: logs_pipelines_order with ID: logs-pipeline-order. Failed to connect resource. {'logs_pipelines': ['jVAX8t9cSTyi_d0-MUh1Jg', 'Sa3WHk3XTwmYYZ0OIR7nuQ', 'IG8_j7YlRfKx0SEvpmjVwg', 'fX2bzl2GQU6EZ0OQ4Jagsg', 'ztJVNwC1S9aBUpqrhUipoA', '6P7jGlyzQqmWYUyn_-F1lw', 'niwmTpGBRIa0SVWa4fU5_w', 'TNDMVe05RsiJmoHgFFbWCQ', 'I-8dmnN5Sk6hEUZdoT8Pdw', 'pAlQHQrvTGOTnrMdHsMVAA', '5At6jBy3QdiV_TSkzNA1kg', 'qZ9ZFLDfTMqGt19AWMxT9g', 'c02R-uDlSUqtNFN4yZcc3g', 'KXPC9KXYTjOV-an6gOhxsw', '52UKFMI5T8WePSxnzyBYHg', 'flyg8Z2iQ3qX3xilMLkaVw', '-eC5_z7ASnyRY74WJ8Zp7g', 'fPxzejH1T_qYqyYPEQqppw', 'tdAfnrdtRWmSYgSm1V_ewA', 'dNCvmHB2QUq6wT0tsinwYg', '763j0DksQ8qftv17JFORoQ', 'CIpxRivjQ2GG-rsunYLZgA', 'T2ZbdeZiSN64rGcx_hMj3A', 'jl9IrjFLTYeWytPhKHQY5Q', '7cKr42xYRaSKJEQ_Tv1N9w', 'ngPqhHLcT3CzlQnVpzLHZA', 'ku7hXF7cSnSlIhLTFw667g', 'wJQRgMvmRGyJl30P78aPTw', 'g5Cs5sr2SemXtMu9Kx87mQ', 'v8w8EypcQf6Egrubqsk5mQ', '4zK7IVI7SbOF5CVgKNgWSA', 'm1EY3k75R6W_DjRPVrA-SQ', 'e1aJMuvWQDC6cjfLILdlFA', 'ulZwKxd3TJGK4F17V3HiJQ', 'GcuKoWxhRJeJqEMEzsyZdA', 'OAusavmVRo2MDS2LwvBHFQ', 'gjs9S_ToRPeEKsT3I4rd9g', 'bxzFAkgeR3GYupJiC9C0Gw', 'CIgFm_mdT5uzKUROm0K6vQ', 'sSQhKUSxReO_ypgLSVvf2w', 'JyPx6rEFRkmKAFMBTAeMeA']}

indeed, after a first sync (of logs_pipelines,logs_pipelines_order), we only get a smaller list:

2023-10-09 12:13:00,580 - INFO - Skipping resource: logs_pipelines_order with ID: logs-pipeline-order. Failed to connect resource. {'logs_pipelines': ['6P7jGlyzQqmWYUyn_-F1lw', 'TNDMVe05RsiJmoHgFFbWCQ', 'I-8dmnN5Sk6hEUZdoT8Pdw', 'pAlQHQrvTGOTnrMdHsMVAA', 'c02R-uDlSUqtNFN4yZcc3g', '-eC5_z7ASnyRY74WJ8Zp7g', 'tdAfnrdtRWmSYgSm1V_ewA', '763j0DksQ8qftv17JFORoQ', 'CIpxRivjQ2GG-rsunYLZgA', 'T2ZbdeZiSN64rGcx_hMj3A', 'jl9IrjFLTYeWytPhKHQY5Q', '7cKr42xYRaSKJEQ_Tv1N9w', 'wJQRgMvmRGyJl30P78aPTw', 'g5Cs5sr2SemXtMu9Kx87mQ', '4zK7IVI7SbOF5CVgKNgWSA', 'm1EY3k75R6W_DjRPVrA-SQ', 'e1aJMuvWQDC6cjfLILdlFA', 'ulZwKxd3TJGK4F17V3HiJQ', 'GcuKoWxhRJeJqEMEzsyZdA', 'OAusavmVRo2MDS2LwvBHFQ', 'gjs9S_ToRPeEKsT3I4rd9g', 'bxzFAkgeR3GYupJiC9C0Gw', 'CIgFm_mdT5uzKUROm0K6vQ', 'JyPx6rEFRkmKAFMBTAeMeA']}

the diff between the two are the logs_pipelines created during the sync
=> diff is not relevant

the logs_pipelines_order resource is skipped

We did a first sync for logs_pipelines, then here logs_pipelines_order:

$ datadog-sync sync --config config --max-workers 1 --verbose --resources="logs_pipelines_order"
2023-10-09 12:48:56,885 - DEBUG - Starting new HTTPS connection (1): api.datadoghq.eu:443
2023-10-09 12:48:56,960 - DEBUG - https://api.datadoghq.eu:443 "GET /api/v1/validate HTTP/1.1" 200 15
2023-10-09 12:48:56,961 - INFO - clients validated successfully
2023-10-09 12:48:57,090 - INFO - Starting sync...
2023-10-09 12:48:57,155 - DEBUG - https://api.datadoghq.eu:443 "GET /api/v1/logs/config/pipeline-order HTTP/1.1" 200 None
2023-10-09 12:48:57,157 - INFO - Skipping resource: logs_pipelines_order with ID: logs-pipeline-order. Failed to connect resource. {'logs_pipelines': ['6P7jGlyzQqmWYUyn_-F1lw', 'TNDMVe05RsiJmoHgFFbWCQ', 'I-8dmnN5Sk6hEUZdoT8Pdw', 'pAlQHQrvTGOTnrMdHsMVAA', 'c02R-uDlSUqtNFN4yZcc3g', '-eC5_z7ASnyRY74WJ8Zp7g', 'tdAfnrdtRWmSYgSm1V_ewA', '763j0DksQ8qftv17JFORoQ', 'CIpxRivjQ2GG-rsunYLZgA', 'T2ZbdeZiSN64rGcx_hMj3A', 'jl9IrjFLTYeWytPhKHQY5Q', '7cKr42xYRaSKJEQ_Tv1N9w', 'wJQRgMvmRGyJl30P78aPTw', 'g5Cs5sr2SemXtMu9Kx87mQ', '4zK7IVI7SbOF5CVgKNgWSA', 'm1EY3k75R6W_DjRPVrA-SQ', 'e1aJMuvWQDC6cjfLILdlFA', 'ulZwKxd3TJGK4F17V3HiJQ', 'GcuKoWxhRJeJqEMEzsyZdA', 'OAusavmVRo2MDS2LwvBHFQ', 'gjs9S_ToRPeEKsT3I4rd9g', 'bxzFAkgeR3GYupJiC9C0Gw', 'CIgFm_mdT5uzKUROm0K6vQ', 'JyPx6rEFRkmKAFMBTAeMeA']}
2023-10-09 12:48:57,162 - INFO - Finished sync: 0 successes, 0 errors

=> no sync.

After digging into the code, I understand we need to run sync with --skip-failed-resource-connections=false:

those pipelines are out-of-the-box pipelines in source organization that are not present in destination organisation (because logs for those services have not (yet) been emitted on the destination organization): they are skipped by logs_pipelines because they cannot be created by API. logs_pipelines_order should also skip them.
by default datadog-sync-cli skips the resource in case of resource connection failure; for this one special case/resource we want to continue (maybe force an override in the code?)

$ datadog-sync sync --config config.real --max-workers 1 --verbose --resources="logs_pipelines_order" --skip-failed-resource-connections=false
2023-10-09 12:51:56,084 - DEBUG - Starting new HTTPS connection (1): api.datadoghq.eu:443
2023-10-09 12:51:56,169 - DEBUG - https://api.datadoghq.eu:443 "GET /api/v1/validate HTTP/1.1" 200 15
2023-10-09 12:51:56,169 - INFO - clients validated successfully
2023-10-09 12:51:56,285 - INFO - Starting sync...
2023-10-09 12:51:56,354 - DEBUG - https://api.datadoghq.eu:443 "GET /api/v1/logs/config/pipeline-order HTTP/1.1" 200 None
2023-10-09 12:51:56,355 - WARNING - logs_pipelines_order with ID: logs-pipeline-order. Failed to connect resource. {'logs_pipelines': ['6P7jGlyzQqmWYUyn_-F1lw', 'TNDMVe05RsiJmoHgFFbWCQ', 'I-8dmnN5Sk6hEUZdoT8Pdw', 'pAlQHQrvTGOTnrMdHsMVAA', 'c02R-uDlSUqtNFN4yZcc3g', '-eC5_z7ASnyRY74WJ8Zp7g', 'tdAfnrdtRWmSYgSm1V_ewA', '763j0DksQ8qftv17JFORoQ', 'CIpxRivjQ2GG-rsunYLZgA', 'T2ZbdeZiSN64rGcx_hMj3A', 'jl9IrjFLTYeWytPhKHQY5Q', '7cKr42xYRaSKJEQ_Tv1N9w', 'wJQRgMvmRGyJl30P78aPTw', 'g5Cs5sr2SemXtMu9Kx87mQ', '4zK7IVI7SbOF5CVgKNgWSA', 'm1EY3k75R6W_DjRPVrA-SQ', 'e1aJMuvWQDC6cjfLILdlFA', 'ulZwKxd3TJGK4F17V3HiJQ', 'GcuKoWxhRJeJqEMEzsyZdA', 'OAusavmVRo2MDS2LwvBHFQ', 'gjs9S_ToRPeEKsT3I4rd9g', 'bxzFAkgeR3GYupJiC9C0Gw', 'CIgFm_mdT5uzKUROm0K6vQ', 'JyPx6rEFRkmKAFMBTAeMeA']}
2023-10-09 12:51:56,374 - INFO - Running update for logs_pipelines_order with logs-pipeline-order
2023-10-09 12:51:56,601 - DEBUG - https://api.datadoghq.eu:443 "PUT /api/v1/logs/config/pipeline-order HTTP/1.1" 200 None
2023-10-09 12:51:56,617 - INFO - Finished update for logs_pipelines_order with logs-pipeline-order
2023-10-09 12:51:56,622 - INFO - Finished sync: 1 successes, 0 errors

=> sync OK (although the WARNING is still there when re-running a diffs or sync afterward)