Skip to content

Conversation

@tanmaya-panda1
Copy link
Collaborator

add ingestion source blobtype to QueuedIngest

@github-actions
Copy link

github-actions bot commented Sep 13, 2025

Test Results

529 tests  ±0   520 ✅ ±0   3m 10s ⏱️ -31s
 31 suites ±0     9 💤 ±0 
 31 files   ±0     0 ❌ ±0 

Results for commit 7a9e8e2. ± Comparison against base commit 31ff143.

♻️ This comment has been updated with latest results.

@AsafMah AsafMah self-requested a review September 18, 2025 09:24
@ag-ramachandran ag-ramachandran force-pushed the feature/add-Ingestion-Source-Blob branch from 8239293 to 058226c Compare September 27, 2025 12:22
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements Azure Kusto ingestion client v2 with comprehensive testing infrastructure. It introduces new client classes for streaming and queued ingestion operations, configuration management, and supporting utilities for data ingestion to Azure Data Explorer.

  • Adds streaming and queued ingestion clients with blob-based and direct ingestion support
  • Implements configuration client for cluster configuration retrieval
  • Creates comprehensive test suites for all client types with end-to-end validation

Reviewed Changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
StreamingIngestClient.kt New client for direct and blob-based streaming ingestion operations
QueuedIngestionClient.kt New client for queued ingestion with polling and status tracking
ConfigurationClient.kt Replaces ConfigurationApiWrapper with improved error handling
ConfigurationClientTest.kt Updated test with parallel execution support and better test structure
StreamingIngestClientTest.kt New test suite for streaming ingestion with parameterized test cases
QueuedIngestionClientTest.kt New test suite for queued ingestion with mapping validation
IngestV2TestBase.kt New base test class with common setup for E2E tests
IngestClient.kt New interface with common response handling for ingestion clients
KustoBaseApiClient.kt Enhanced with serialization support, timeout configuration, and improved error handling
openapi.yaml Updated API specification with additional headers and renamed mapping properties
pom.xml Updated Kotlin version and added test dependencies
logback.xml, application.yaml New configuration files for logging and application settings
FormatUtil.kt New utility for format detection
IngestionResultUtils.kt New utility for result status checking
OffsetDateTimeSerializer.kt New custom serializer for date-time handling
mapping/*.kt New data models for ingestion mapping configurations
ContainerBase.kt New interface for container abstraction
LocalSource.kt, IngestionSource.kt Removed unused source classes
ConfigurationApiWrapper.kt Removed and replaced by ConfigurationClient

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

if (blobUrl != null) {
// Blob-based streaming
val requestBody = StreamFromBlobRequestBody(sourceUri = blobUrl)
bodyContent = Json.encodeToString(requestBody).toByteArray()
Copy link

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing serializer specification for Json.encodeToString(). The method requires an explicit serializer parameter: Json.encodeToString(StreamFromBlobRequestBody.serializer(), requestBody) to properly serialize the StreamFromBlobRequestBody data class.

Suggested change
bodyContent = Json.encodeToString(requestBody).toByteArray()
bodyContent = Json.encodeToString(StreamFromBlobRequestBody.serializer(), requestBody).toByteArray()

Copilot uses AI. Check for mistakes.
@AfterAll
fun dropTables() {
val dropTableScript = ".drop table $targetTable ifexists"
logger.error("Dropping table $targetTable")
Copy link

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using logger.error() for normal cleanup operations is incorrect. This should be logger.info() since dropping tables in test teardown is expected behavior, not an error condition.

Suggested change
logger.error("Dropping table $targetTable")
logger.info("Dropping table $targetTable")

Copilot uses AI. Check for mistakes.
required: true
schema:
type: string
example: "gzip"
Copy link

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example value for the 'Host' header parameter is set to 'gzip', which is incorrect. The Host header should contain a hostname (e.g., 'example.kusto.windows.net'), not an encoding type.

Suggested change
example: "gzip"
example: "ingest-mycluster.swedencentral.kusto.windows.net"

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants