[Cloudflare Logpush] - Added support for Azure Blob Storage input in all data streams (elastic#15440)

Added support for Azure Blob Storage input in all data streams along with
relevant system, policy tests and updated documentation.

Author: ShourieG

packages/cloudflare_logpush/_dev/build/docs/README.md

@@ -4,10 +4,12 @@
 
 The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Access Request, Audit, CASB, Device Posture, DNS, DNS Firewall, Firewall Event, Gateway DNS, Gateway HTTP, Gateway Network, HTTP Request, Magic IDS, NEL Report, Network Analytics, Sinkhole HTTP, Spectrum Event, Network Session and Workers Trace Events logs. Cloudflare is a content delivery network and DDoS mitigation company. Cloudflare provides a network designed to make everything you connect to the Internet secure, private, fast, and reliable; secure your websites, APIs, and Internet applications; protect corporate networks, employees, and devices; and write and deploy code that runs on the network edge.
 
-The Cloudflare Logpush integration can be used in three different modes to collect data:
+The Cloudflare Logpush integration can be used in the following modes to collect data:
 - HTTP Endpoint mode - Cloudflare pushes logs directly to an HTTP endpoint hosted by your Elastic Agent.
 - AWS S3 polling mode - Cloudflare writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
 - AWS S3 SQS mode - Cloudflare writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
+- Azure Blob Storage polling mode - Cloudflare writes data to Azure Blob Storage and Elastic Agent polls the Azure Blob Storage containers by listing its contents and reading new files.
+- Google Cloud Storage polling mode - Cloudflare writes data to Google Cloud Storage and Elastic Agent polls the GCS buckets by listing its contents and reading new files.
 
 For example, you could use the data from this integration to know which websites have the highest traffic, which areas have the highest network traffic, or observe mitigation statistics.
 
@@ -136,13 +138,25 @@ When configuring the integration to read from S3-Compatible Buckets such as Clou
 ### Collect data from GCS Buckets
 
 - Configure the [Data Forwarder](https://developers.cloudflare.com/logs/get-started/enable-destinations/google-cloud-storage/) to ingest data into a GCS bucket.
-- Configure the GCS bucket names and credentials along with the required configs under the "Collect Cloudflare Logpush logs via Google Cloud Storage" section. 
+- Configure the GCS bucket names and credentials along with the required configurations under the "Collect Cloudflare Logpush logs via Google Cloud Storage" section. 
 - Make sure the service account and authentication being used, has proper levels of access to the GCS bucket [Manage Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys/)
 
 **Note**:
 - The GCS input currently does not support fetching of buckets using bucket prefixes, so the bucket names have to be configured manually for each data stream.
-- The GCS input currently only accepts a service account JSON key or a service account JSON file for authentication.
-- The GCS input currently only supports json data.
+- The GCS input accepts a service account JSON key or a service account JSON file for authentication.
+- The GCS input supports JSON/NDJSON data.
+
+### Collect data from Azure Blob Storage
+
+- [Enable Microsoft Azure](https://developers.cloudflare.com/logs/logpush/logpush-job/enable-destinations/azure/) to ingest data into Azure Blob Storage containers.
+- Configure Azure Blob Storage container names and credentials along with the required configurations under the "Collect Cloudflare Logpush logs via Azure Blob Storage" section. 
+- Make sure the storage account and authentication being used, has proper levels of access to the Azure Blob Storage Container. Please follow the documentation [here](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-data-operations-portal) for more details.
+- If you want to use RBAC for your account please follow the documentation [here](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory).
+
+**Note**:
+- The Azure Blob Storage input does not support fetching from containers using container prefixes, so the containers' names must be configured manually for each data stream.
+- The Azure Blob Storage input accepts a service account key (shared credentials key), service account URI (connection string) and OAuth2 credentials for authentication.
+- The Azure Blob Storage input only supports JSON/NDJSON data.
 
 ### Collect data from the Cloudflare HTTP Endpoint
 

packages/cloudflare_logpush/_dev/deploy/docker/Dockerfile

@@ -0,0 +1,20 @@
+FROM node:current-bullseye-slim
+
+# Install dependencies
+RUN apt-get update && apt-get install -y \
+    curl \
+    netcat \
+    ca-certificates \
+    lsb-release \
+    && rm -rf /var/lib/apt/lists/*
+# Install Azurite
+RUN npm install -g azurite
+
+# Install Azure CLI
+RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash
+
+# Copy emulator script
+COPY files/emulator.sh /emulator.sh
+RUN chmod +x /emulator.sh
+
+ENTRYPOINT ["/bin/sh", "/emulator.sh"]

packages/cloudflare_logpush/_dev/deploy/docker/docker-compose.yml

@@ -189,3 +189,14 @@ services:
       - STREAM_WEBHOOK_PROBE=false
       - STREAM_ADDR=http://elastic-agent:9560/cloudflare_logpush/dns_firewall
     command: log --start-signal=SIGHUP --delay=5s /sample_logs/dns_firewall.log
+  
+  azure-blob-storage-emulator:
+    build: .
+    volumes:
+      - ./sample_logs:/sample_logs
+    ports:
+      - "10000/tcp"
+    healthcheck:
+      test: "curl --fail --silent --output /dev/null http://localhost:4443 || exit 1"
+      interval: 10s
+      timeout: 5s

packages/cloudflare_logpush/_dev/deploy/docker/files/emulator.sh

@@ -0,0 +1,35 @@
+#!/bin/sh
+set -e
+
+# Set environment variables for Azure cli
+export AZURE_STORAGE_ACCOUNT=devstoreaccount1
+export AZURE_STORAGE_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
+export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=$AZURE_STORAGE_ACCOUNT;AccountKey=$AZURE_STORAGE_KEY;BlobEndpoint=http://127.0.0.1:10000/$AZURE_STORAGE_ACCOUNT;"
+
+# Start Azurite in background
+azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --skipApiVersionCheck &
+AZURITE_PID=$!
+
+# Wait until Azurite is ready
+while ! nc -z 127.0.0.1 10000; do sleep 0.5; done
+
+# Create container
+az storage container create --name test-container
+
+# Upload all files
+for f in /sample_logs/*; do
+  if [ -f "$f" ]; then
+    az storage blob upload --container-name test-container --file "$f" --name "$(basename "$f")"
+  fi
+done
+
+node -e "
+const http = require('http');
+const server = http.createServer((req, res) => { res.writeHead(200); res.end('OK'); });
+server.listen(4443, () => console.log('Healthcheck API available on port 4443'));
+" &
+
+echo "All files uploaded. Healthcheck API started."
+
+# Keep Azurite running
+wait $AZURITE_PID

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/http_request.log

@@ -1 +1,2 @@
 {"BotDetectionIDs":[7,8,9],"BotScore":20,"BotScoreSrc":"Verified Bot","BotTags":["bing","api"],"CacheCacheStatus":"dynamic","CacheResponseBytes":983828,"CacheResponseStatus":200,"CacheTieredFill":false,"ClientASN":43766,"ClientCountry":"sa","ClientDeviceType":"desktop","ClientIP":"175.16.199.0","ClientIPClass":"noRecord","ClientMTLSAuthCertFingerprint":"Fingerprint","ClientMTLSAuthStatus":"unknown","ClientRequestBytes":5800,"ClientRequestHost":"xyz.example.com","ClientRequestMethod":"POST","ClientRequestPath":"/xyz/checkout","ClientRequestProtocol":"HTTP/1.1","ClientRequestReferer":"https://example.com/s/example/default?sourcerer=(default:(id:!n,selectedPatterns:!(example,%27logs-endpoint.*-example%27,%27logs-system.*-example%27,%27logs-windows.*-example%27)))&timerange=(global:(linkTo:!(),timerange:(from:%272022-05-16T06:26:36.340Z%27,fromStr:now-24h,kind:relative,to:%272022-05-17T06:26:36.340Z%27,toStr:now)),timeline:(linkTo:!(),timerange:(from:%272022-04-17T22:00:00.000Z%27,kind:absolute,to:%272022-04-18T21:59:59.999Z%27)))&timeline=(activeTab:notes,graphEventId:%27%27,id:%279844bdd4-4dd6-5b22-ab40-3cd46fce8d6b%27,isOpen:!t)","ClientRequestScheme":"https","ClientRequestSource":"edgeWorkerFetch","ClientRequestURI":"/s/example/api/telemetry/v2/clusters/_stats","ClientRequestUserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36","ClientSrcPort":0,"ClientSSLCipher":"NONE","ClientSSLProtocol":"TLSv1.2","ClientTCPRTTMs":0,"ClientXRequestedWith":"Request With","Cookies":{"key":"value"},"EdgeCFConnectingO2O":false,"EdgeColoCode":"RUH","EdgeColoID":339,"EdgeEndTimestamp":"2022-05-25T13:25:32Z","EdgePathingOp":"wl","EdgePathingSrc":"macro","EdgePathingStatus":"nr","EdgeRateLimitAction":"unknown","EdgeRateLimitID":0,"EdgeRequestHost":"abc.example.com","EdgeResponseBodyBytes":980397,"EdgeResponseBytes":981308,"EdgeResponseCompressionRatio":0,"EdgeResponseContentType":"application/json","EdgeResponseStatus":200,"EdgeServerIP":"1.128.0.0","EdgeStartTimestamp":"2022-05-25T13:25:26Z","EdgeTimeToFirstByteMs":5333,"OriginDNSResponseTimeMs":3,"OriginIP":"67.43.156.0","OriginRequestHeaderSendDurationMs":0,"OriginResponseBytes":0,"OriginResponseDurationMs":5319,"OriginResponseHeaderReceiveDurationMs":5155,"OriginResponseHTTPExpires":"2022-05-27T13:25:26Z","OriginResponseHTTPLastModified":"2022-05-26T13:25:26Z","OriginResponseStatus":200,"OriginResponseTime":5232000000,"OriginSSLProtocol":"TLSv1.2","OriginTCPHandshakeDurationMs":24,"OriginTLSHandshakeDurationMs":53,"ParentRayID":"710e98d93d50357d","RayID":"710e98d9367f357d","SecurityLevel":"off","SmartRouteColoID":20,"UpperTierColoID":0,"SecurityAction":"unknown","WAFAttackScore":50,"WAFRCEAttackScore":1,"WAFSQLiAttackScore":99,"WAFXSSAttackScore":90,"WAFFlags":"0","WAFMatchedVar":"example","WAFProfile":"unknown","SecurityRuleID":"98d93d5","SecurityRuleDescription":"matchad variable message","WorkerCPUTime":0,"WorkerStatus":"unknown","WorkerSubrequest":true,"WorkerSubrequestCount":0,"ZoneID":393347122,"ZoneName":"example.com"}
+{"ClientASN":8075,"ClientCountry":"NL","ClientDeviceType":"desktop","ClientIP":"89.160.20.156","ClientIPClass":"noRecord","ClientRequestBytes":7495,"ClientRequestHost":"api.deals-referral.com","ClientRequestMethod":"GET","ClientRequestPath":"/referralapi/api/deals/find","ClientRequestProtocol":"HTTP/1.1","ClientRequestReferer":"https://www.main-site.com/user/dashboard","ClientRequestURI":"/referralapi/api/deals/find?pricingCaseId=68c15fccf03482a4cb20460b","ClientRequestUserAgent":"got (https://github.com/sindresorhus/got)","ClientSSLCipher":"AEAD-AES256-GCM-SHA384","ClientSSLProtocol":"TLSv1.3","ClientSrcPort":32462,"ClientXRequestedWith":"XMLHttpRequest","EdgeColoCode":"AMS","EdgeColoID":996,"EdgeEndTimestamp":"2025-09-17T08:37:18.000Z","EdgePathingOp":"wl","EdgePathingSrc":"undef","EdgePathingStatus":"nr","EdgeRateLimitAction":"log","EdgeRateLimitID":12345,"EdgeRequestHost":"api.deals-referral.com","EdgeResponseBytes":1063,"EdgeResponseCompressionRatio":1,"EdgeResponseContentType":"application/json; charset=utf-8","EdgeResponseStatus":200,"EdgeServerIP":"89.160.20.156","EdgeStartTimestamp":"2025-09-17T08:37:18.000Z","EdgeTimeToFirstByteMs":65,"FirewallMatchesActions":"skip","FirewallMatchesRuleIDs":"fddd53f90acd49a89aea84e1c665269c","FirewallMatchesSources":"firewallManaged","OriginDNSResponseTimeMs":2,"OriginIP":"172.16.10.100","OriginResponseBytes":980,"OriginResponseDurationMs":62,"OriginResponseHTTPExpires":"Wed, 17 Sep 2025 08:42:18 GMT","OriginResponseHTTPLastModified":"Wed, 17 Sep 2025 08:30:00 GMT","OriginResponseStatus":200,"OriginResponseTime":62543210,"OriginSSLProtocol":"TLSv1.3","ParentRayID":"980747e7ad8f0001","RayID":"980747e7ad8fbdf3","SecurityLevel":"high","WAFAction":"log","WAFFlags":"0","WAFMatchedVar":"ARGS:pricingCaseId","WAFProfile":"default","WAFRuleID":"942100","WAFRuleMessage":"SQL Injection Attack Detected","WorkerCPUTime":1601,"WorkerStatus":"ok","WorkerSubrequest":false,"WorkerSubrequestCount":1,"WorkerWallTimeUs":36953,"ZoneID":296364127}

packages/cloudflare_logpush/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.40.0"
+  changes:
+    - description: Added support for Azure Blob Storage input for all data streams.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/15440
 - version: "1.39.2"
   changes:
     - description: Do not convert timestamps in spectrum_event datastream that are 0, "0" or "".

packages/cloudflare_logpush/data_stream/access_request/_dev/test/policy/test-abs.expected

@@ -0,0 +1,59 @@
+inputs:
+    - data_stream:
+        namespace: ep
+      meta:
+        package:
+            name: cloudflare_logpush
+      name: test-abs-cloudflare_logpush
+      streams:
+        - account_name: devstoreaccount1
+          auth.connection_string.uri: DefaultEndpointsProtocol=https;AccountName=dummystorageacct;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;EndpointSuffix=core.windows.net
+          auth.oauth2:
+            client_id: ${SECRET_0}
+            client_secret: ${SECRET_1}
+            tenant_id: 12345
+          auth.shared_credentials.account_key: ${SECRET_2}
+          containers:
+            - name: test-container
+          data_stream:
+            dataset: cloudflare_logpush.access_request
+          expand_event_list_from_field: records
+          file_selectors:
+            - regex: access-logs/
+          max_workers: 3
+          poll: true
+          poll_interval: 15s
+          processors:
+            - decode_json_fields:
+                fields:
+                    - message
+                max_depth: 10
+                overwrite_keys: true
+                process_array: true
+                target: ""
+          publisher_pipeline.disable_host: true
+          tags:
+            - preserve_original_event
+            - preserve_duplicate_custom_fields
+            - forwarded
+          timestamp_epoch: 10000000000
+      type: azure-blob-storage
+      use_output: default
+output_permissions:
+    default:
+        _elastic_agent_checks:
+            cluster:
+                - monitor
+        _elastic_agent_monitoring:
+            indices: []
+        uuid-for-permissions-on-related-indices:
+            indices:
+                - names:
+                    - logs-cloudflare_logpush.access_request-ep
+                  privileges:
+                    - auto_configure
+                    - create_doc
+secret_references:
+    - {}
+    - {}
+    - {}

packages/cloudflare_logpush/data_stream/access_request/_dev/test/policy/test-abs.yml

@@ -0,0 +1,31 @@
+input: azure-blob-storage
+vars:
+  account_name: devstoreaccount1
+  oauth2: true
+  client_id: devstoreaccount1
+  tenant_id: 12345
+  client_secret: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
+  service_account_key: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
+  service_account_uri: DefaultEndpointsProtocol=https;AccountName=dummystorageacct;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;EndpointSuffix=core.windows.net
+data_stream:
+  vars:
+    number_of_workers: 3
+    poll: true
+    poll_interval: 15s
+    containers: |
+      - name: test-container
+    file_selectors: |
+      - regex: access-logs/
+    timestamp_epoch: 10000000000
+    expand_event_list_from_field: records
+    preserve_original_event: true
+    preserve_duplicate_custom_fields: true
+    processors: |
+      - decode_json_fields:
+          fields: ["message"]
+          process_array: true
+          max_depth: 10
+          target: ""
+          overwrite_keys: true
+    tags: 
+      - forwarded

packages/cloudflare_logpush/data_stream/access_request/_dev/test/system/test-abs-config.yml

@@ -0,0 +1,18 @@
+deployer: docker
+service: azure-blob-storage-emulator
+input: azure-blob-storage
+vars:
+  account_name: devstoreaccount1
+  service_account_key: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
+  storage_url: "http://{{Hostname}}:{{Port}}/devstoreaccount1/"
+data_stream:
+  vars:
+    number_of_workers: 1
+    poll: true
+    poll_interval: 15s
+    containers: |
+      - name: test-container
+    file_selectors: |
+      - regex: "access_request"
+assert:
+  hit_count: 1

packages/cloudflare_logpush/data_stream/access_request/agent/stream/abs.yml.hbs

@@ -0,0 +1,60 @@
+{{#if account_name}}
+account_name: {{account_name}}
+{{/if}}
+{{#if oauth2}}
+auth.oauth2:
+  client_id: {{client_id}}
+  client_secret: {{client_secret}}
+  tenant_id: {{tenant_id}}
+{{/if}}
+{{#if service_account_key}}
+auth.shared_credentials.account_key: {{service_account_key}}
+{{/if}}
+{{#if service_account_uri}}
+auth.connection_string.uri: {{service_account_uri}}
+{{/if}}
+{{#if storage_url}}
+storage_url: {{storage_url}}
+{{/if}}
+{{#if number_of_workers}}
+max_workers: {{number_of_workers}}
+{{/if}}
+{{#if poll}}
+poll: {{poll}}
+{{/if}}
+{{#if poll_interval}}
+poll_interval: {{poll_interval}}
+{{/if}}
+
+{{#if containers}}
+containers:
+{{containers}}
+{{/if}}
+{{#if file_selectors}}
+file_selectors:
+{{file_selectors}}
+{{/if}}
+{{#if timestamp_epoch}}
+timestamp_epoch: {{timestamp_epoch}}
+{{/if}}
+{{#if expand_event_list_from_field}}
+expand_event_list_from_field: {{expand_event_list_from_field}}
+{{/if}}
+
+tags:
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#if preserve_duplicate_custom_fields}}
+  - preserve_duplicate_custom_fields
+{{/if}}
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+{{#if processors}}
+processors:
+{{processors}}
+{{/if}}