-
Notifications
You must be signed in to change notification settings - Fork 45
Feature/add ingestion source blob #440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
tanmaya-panda1
wants to merge
30
commits into
feature/IngestV2
Choose a base branch
from
feature/add-Ingestion-Source-Blob
base: feature/IngestV2
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 27 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
2ad23f4
* Add boilerplate
ag-ramachandran 8a031a1
* Move code forward
ag-ramachandran 6e6998c
Add IngestionSource Blob type and QueuedIngest
tanmaya-panda1 debcb10
removed unnecessary methods
tanmaya-panda1 b60ea4f
* Reformat code
ag-ramachandran a548559
* Update POM
ag-ramachandran fb68d1c
* Remove gitignore (redundant)
ag-ramachandran e0fc2c8
* Add a todo for fixing
ag-ramachandran 7dfd173
* Add some more tests
ag-ramachandran c261be7
* WIP with failing tests
ag-ramachandran c2b83ed
* WIP with failing tests
ag-ramachandran 9db6537
* WIP with failing tests
ag-ramachandran be86fe3
* WIP add additional tests
ag-ramachandran d7819c5
* WIP: Fix review comments
ag-ramachandran df44c94
* WIP: Fix review comments
ag-ramachandran 3d51aa4
* Add additional tests
ag-ramachandran 2a1c55c
* Add support for streaming ingest
ag-ramachandran b614009
fixed streaming ingestion binary kusto format and added happy flow te…
tanmaya-panda1 ea95002
* Format changes
ag-ramachandran a8fa935
* Refactor tests
ag-ramachandran fc24849
* Add tests for ingestion mapping inline
ag-ramachandran 5eebd47
* Refactor for inline mapping
ag-ramachandran 3bf0e6d
added bloburl to streaming
tanmaya-panda1 94e92f1
refactored Streaming ingestion tests
tanmaya-panda1 1b946e0
* rebase changes
ag-ramachandran ffe3cda
* rebase changes
ag-ramachandran 2de495c
* Add TODO for OneApiException scenario
ag-ramachandran 94eb3a6
* Add TODO for OneApiException scenario
ag-ramachandran abc6df8
* Fix review comments
ag-ramachandran 7a9e8e2
*Address some more review comments
ag-ramachandran File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
46 changes: 0 additions & 46 deletions
46
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/ConfigurationApiWrapper.kt
This file was deleted.
Oops, something went wrong.
92 changes: 92 additions & 0 deletions
92
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/ConfigurationClient.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| // Copyright (c) Microsoft Corporation. | ||
| // Licensed under the MIT License. | ||
| package com.microsoft.azure.kusto.ingest.v2 | ||
|
|
||
| import com.azure.core.credential.TokenCredential | ||
| import com.microsoft.azure.kusto.ingest.v2.common.exceptions.IngestException | ||
| import com.microsoft.azure.kusto.ingest.v2.infrastructure.HttpResponse | ||
| import com.microsoft.azure.kusto.ingest.v2.models.ConfigurationResponse | ||
| import io.ktor.http.HttpStatusCode | ||
| import org.slf4j.LoggerFactory | ||
| import java.net.ConnectException | ||
|
|
||
| class ConfigurationClient( | ||
| override val dmUrl: String, | ||
| override val tokenCredential: TokenCredential, | ||
| override val skipSecurityChecks: Boolean = false, | ||
| ) : KustoBaseApiClient(dmUrl, tokenCredential, skipSecurityChecks) { | ||
| private val logger = | ||
| LoggerFactory.getLogger(ConfigurationClient::class.java) | ||
| private val baseUrl = "$dmUrl/v1/rest/ingestion/configuration" | ||
|
|
||
| suspend fun getConfigurationDetails(): ConfigurationResponse { | ||
| try { | ||
| val configurationHttpResponse: HttpResponse<ConfigurationResponse> = | ||
| api.getIngestConfiguration() | ||
| if (configurationHttpResponse.success) { | ||
| logger.info( | ||
| "Successfully retrieved configuration details from $dmUrl with status: ${configurationHttpResponse.status}", | ||
| ) | ||
| logger.debug( | ||
| "Configuration details: {}", | ||
| configurationHttpResponse.body(), | ||
| ) | ||
| return configurationHttpResponse.body() | ||
| } else if ( | ||
| configurationHttpResponse.status == | ||
| HttpStatusCode.NotFound.value | ||
| ) { | ||
| /* | ||
| 404 is a special case - it indicates that the endpoint is not found. This may be a transient | ||
| network issue | ||
| */ | ||
| val message = | ||
| "Endpoint $dmUrl not found. Please ensure the cluster supports queued ingestion." | ||
| logger.error( | ||
| "{}. Status: {}", | ||
| message, | ||
| configurationHttpResponse.status, | ||
| ) | ||
| throw IngestException( | ||
| message = message, | ||
| cause = ConnectException(message), | ||
| failureCode = configurationHttpResponse.status, | ||
| failureSubCode = "", | ||
| isPermanent = false, | ||
| ) | ||
| } else { | ||
| val configurationResponseBody = configurationHttpResponse.body() | ||
| val message = | ||
| "Failed to retrieve configuration details from $baseUrl.Status: ${configurationHttpResponse.status}, " + | ||
| "Body: $configurationResponseBody" | ||
| logger.error("{}", message) | ||
| throw IngestException( | ||
| message = message, | ||
| failureCode = configurationHttpResponse.status, | ||
| ) | ||
| } | ||
| } catch (notAbleToReachHost: ConnectException) { | ||
| val message = | ||
| "Failed to reach $baseUrl. Please ensure the cluster address is correct and the cluster is reachable." | ||
| throw IngestException( | ||
| message = message, | ||
| cause = notAbleToReachHost, | ||
| failureCode = HttpStatusCode.NotFound.value, | ||
| failureSubCode = "", | ||
| isPermanent = false, | ||
| ) | ||
| } catch (ex: Exception) { | ||
| if (ex is IngestException) throw ex | ||
| val message = | ||
| "An unexpected error occurred while trying to reach $baseUrl" | ||
| throw IngestException( | ||
| message = message, | ||
| cause = ex, | ||
| // Mark this as a 5xx series error | ||
| failureCode = HttpStatusCode.InternalServerError.value, | ||
| failureSubCode = "", | ||
| isPermanent = true, | ||
| ) | ||
| } | ||
| } | ||
| } |
76 changes: 76 additions & 0 deletions
76
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/IngestClient.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| // Copyright (c) Microsoft Corporation. | ||
| // Licensed under the MIT License. | ||
| package com.microsoft.azure.kusto.ingest.v2 | ||
|
|
||
| import com.microsoft.azure.kusto.ingest.v2.common.exceptions.IngestException | ||
| import com.microsoft.azure.kusto.ingest.v2.infrastructure.HttpResponse | ||
| import com.microsoft.azure.kusto.ingest.v2.models.IngestResponse | ||
| import io.ktor.http.HttpStatusCode | ||
| import org.slf4j.Logger | ||
| import org.slf4j.LoggerFactory | ||
| import java.net.ConnectException | ||
|
|
||
| interface IngestClient { | ||
| val logger: Logger | ||
| get() = LoggerFactory.getLogger(IngestClient::class.java) | ||
|
|
||
| // Common way to parse ingestion response for both Streaming and Queued ingestion | ||
|
|
||
| suspend fun <T : Any> handleIngestResponse( | ||
| response: HttpResponse<T>, | ||
| database: String, | ||
| table: String, | ||
| dmUrl: String, | ||
| endpointType: String, | ||
| ): T { | ||
| if (response.success) { | ||
| val ingestResponseBody = response.body() | ||
| return ingestResponseBody | ||
| } else { | ||
| if (response.status == HttpStatusCode.NotFound.value) { | ||
| val message = | ||
| "Endpoint $dmUrl not found. Please ensure the cluster supports $endpointType ingestion." | ||
| logger.error( | ||
| "$endpointType ingestion endpoint not found. Please ensure that the target cluster supports $endpointType ingestion and that the endpoint URL is correct.", | ||
| ) | ||
| throw IngestException( | ||
| message = message, | ||
| cause = ConnectException(message), | ||
| failureCode = response.status, | ||
| failureSubCode = "", | ||
| isPermanent = false, | ||
| ) | ||
| } | ||
| val nonSuccessResponseBody: T = response.body() | ||
| val ingestResponseOperationId = | ||
| if (nonSuccessResponseBody is IngestResponse) { | ||
| if ( | ||
| (nonSuccessResponseBody as IngestResponse) | ||
| .ingestionOperationId != null | ||
| ) { | ||
| logger.info( | ||
| "Ingestion Operation ID: ${(nonSuccessResponseBody as IngestResponse).ingestionOperationId}", | ||
| ) | ||
| nonSuccessResponseBody.ingestionOperationId | ||
| } else { | ||
| "N/A" | ||
| } | ||
| } else { | ||
| "N/A" | ||
| } | ||
| val errorMessage = | ||
| "Failed to submit $endpointType ingestion to $database.$table. " + | ||
| "Status: ${response.status}, Body: $nonSuccessResponseBody. " + | ||
| "OperationId $ingestResponseOperationId" | ||
| logger.error( | ||
| "$endpointType ingestion failed with response: {}", | ||
| errorMessage, | ||
| ) | ||
| throw IngestException( | ||
| message = errorMessage, | ||
| cause = RuntimeException(errorMessage), | ||
| isPermanent = true, | ||
| ) | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.