[Netskope] Add multiple system tests for Alerts_v2 and Events_v2 data stream #14887
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| version: "2.3" | ||
| services: | ||
| azure-blob-storage-emulator: | ||
| image: mcr.microsoft.com/azure-storage/azurite | ||
| command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000 | ||
| ports: | ||
| - "10000/tcp" | ||
| uploader: | ||
| image: mcr.microsoft.com/azure-cli | ||
| depends_on: | ||
| - azure-blob-storage-emulator | ||
| volumes: | ||
| - ./sample_logs:/sample_logs | ||
| entrypoint: > | ||
| sh -c " | ||
| sleep 5 && | ||
| gzip -c /sample_logs/test-alerts-v2.csv > /sample_logs/test-alerts-v2.csv.gz && | ||
| export AZURE_STORAGE_CONNECTION_STRING='DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azure-blob-storage-emulator:10000/devstoreaccount1;' && | ||
| az storage container create --name test-container && | ||
| az storage blob upload --container-name test-container --file /sample_logs/test-alerts-v2.csv.gz --name test-alerts-v2.csv.gz | ||
| " | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| _id,access_method,account_name,acked,acting_user,action,activity,act_user,alert,alert_name,severity,alert_source,alert_type,appcategory,appsuite,app,app_session_id,assignee,bcc,browser,browser_session_id,server_bytes,client_bytes,cc,cci,ccl,cloud_provider,breach_id,eeml,breach_score,connection_id,src_country,shared_credential_user,breach_date,policy_name,policy_action,dst_country,dst_geoip_src,dsthost,dstip,dst_location,dstport,dst_region,dst_timezone,dst_zipcode,detection_engine,device,device_classification,device_sn,dlp_file,dlp_fingerprint_classification,dlp_fingerprint_match,dlp_fingerprint_score,dlp_match_info,inline_dlp_match_info,dlp_incident_id,dlp_parent_id,dlp_profile_name,dlp_profile,dlp_rule_count,dlp_rule,dlp_rule_severity,dlp_rule_score,dlp_unique_count,dlp_is_unique_count,dns_profile,domain,driver,conn_duration,encryption_status,conn_endtime,end_time,computer_name,executable_hash,executable_signed,sharedType,file_category,destination_file_directory,file_exposure,file_id,file_md5,destination_file_name,filename,file_origin,file_owner,destination_file_path,file_path,filepath,sha256,file_size,file_type,email_from_user,from_user,app-gdpr-level,usergroup,device_type,hostname,dinsid,incident_id,latest_incident_id,instance_id,instance,instance_name,sanctioned_instance,ip_protocol,dst_latitude,src_latitude,local_md5,local_sha1,local_sha256,loc,location,src_location,dst_longitude,src_longitude,mal_id,malware_id,mal_sev,malware_severity,mal_type,malware_type,managed_app,managementID,vendor_id,md5,message_id,mime_type,tss_mode,product_id,modified_date,src_network,network_session_id,ur_normalized,oauth,object,object_id,owner,object_type,org,organization_unit,os,os_details,os_family,os_user_name,os_version,page,parent_id,owner_pdl,policy_name_enforced,policy,policy_version,pop_id,netskope_pop,port,web_url,connection_type,process_name,process_cert_subject,pid,process_path,publisher_cn,domain_ip,redirect_url,referer,region_name,src_region,region_id,iaas_remediated,iaas_remediation_action,iaas_remediated_by,iaas_remediated_on,req,req_cnt,request_id,resource_category,resource_group,resp,resp_cnt,risk_level_id,sa_profile_name,sa_rule_compliance,sa_rule_name,sa_rule_severity,sender,session_duration,session_number_unique,serverity,severity_level,severity_id,shared_with,shared_domains,tunnel_id,smtp_status,smtp_to,src_geoip_src,srcip,srcport,conn_starttime,start_time,status,subject,tags,telemetry_app,threat_type,timestamp,src_timezone,to_user,numbytes,traffic_type,transaction_id,tss_license,two_factor_auth,type,unc_path,nsdeviceuid,url,user,useragent,user_confidence_index,user_confidence_level,user_id,userip,userkey,violation,site,src_zipcode,account_id,alert_id,appact,audit_type,response_time,email_modified,email_title,subtype,event_uuid,file_cls_encrypted,fllg,file_pdl,local_source_time,server_packets,client_packets,flpp,risk_score,suppression_count,spet,spst,thr,email_user,tur,total_packets,num_users,watchlist_name,custom_attr,record_type | ||
| 2bebaadf4ac868577ea32140,Endpoint,-,false,-,block,File Share Access,-,yes,CDS TEST,-,-,Device,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,CDS TEST,block,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,Win11-50-1-105,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,[email protected],-,-,-,-,-,-,-,Windows,Microsoft Windows 11 Pro 10.0.22621 64-bit,-,-,-,-,-,-,TEST,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,1747209128,-,-,-,-,-,-,-,endpoint,-,-,-,[email protected],-,-,-,-,-,[email protected],-,-,-,-,-,-,-,-,-,-,-,64907a4d-66d6-4a3b-8693-069b206a4479,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,alert | ||
| 772202b2ea0d6057f886f053,Endpoint,-,false,-,block,Insert,-,yes,BlockEndpoint,-,-,Device,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,BlockEndpoint,block,-,-,-,-,-,-,-,-,-,-,-,MacOs check,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,N49J4M9T3C,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,[email protected],-,-,-,-,-,-,-,macOS,Mac OS X Sonoma 14.7.5 arm64,-,-,-,-,-,-,BlockEndpoint,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,1747134127,-,-,-,-,-,-,-,endpoint,-,-,-,[email protected],-,-,-,-,-,[email protected],-,-,-,-,-,-,-,-,-,-,-,5a5574fd-0083-41c3-996a-81c67e6c45d6,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,alert | ||
| eb8fc9903c2fbb6aa05537ff,Client,-,false,-,alert,Edit,-,yes,Web Access Allow,-,-,policy,IT Service/Application Management,Amazon,Amazon Systems Manager,2241753685910532990,-,-,Native,4940241048203471891,-,-,-,92,excellent,-,-,-,-,2631086121425559188,SE,-,-,-,-,SE,-,-,81.2.69.142,Stockholm,443,Stockholm County,Europe/Stockholm,100 04,-,Windows Device,unmanaged,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,ssm.eu-north-1.amazonaws.com,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,Test-IDMHT6TII,-,5254981775376249392,-,202533540828,-,-,-,-,18.0717|59.328699999999998,18.0717|59.328699999999998,-,-,-,-,-,Stockholm,18.0717|59.328699999999998,18.0717|59.328699999999998,-,-,-,-,-,-,no,-,-,-,-,-,-,-,-,-,-,[email protected],-,-,-,-,-,-,-,Windows 11,-,Windows,-,Windows NT 11.0,ssm.eu-north-1.amazonaws.com,-,-,-,Web Access Allow,-,-,SE-STO1,443,-,-,-,-,-,-,-,-,-,-,-,Stockholm County,-,-,-,-,-,-,-,5254981775376249392,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,81.2.69.142,-,-,-,-,-,-,-,-,1747134122,Europe/Stockholm,-,-,CloudApp,5254981775376249392,-,-,nspolicy,-,-,ssm.eu-north-1.amazonaws.com/,[email protected],aws-sdk-go/1.55.5 (go1.23.7; windows; amd64) amazon-ssm-agent/3.3.2299.0,-,-,-,81.2.69.142,[email protected],-,Amazon Systems Manager,100 04,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,alert |
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,26 +1,67 @@ | ||
| variable "TEST_RUN_ID" { | ||
| default = "detached" | ||
| } | ||
|
|
||
| variable "BRANCH" { | ||
| description = "Branch name or pull request for tagging purposes" | ||
| default = "unknown-branch" | ||
| } | ||
|
|
||
| variable "BUILD_ID" { | ||
| description = "Build ID in the CI for tagging purposes" | ||
| default = "unknown-build" | ||
| } | ||
|
|
||
| variable "CREATED_DATE" { | ||
| description = "Creation date in epoch time for tagging purposes" | ||
| default = "unknown-date" | ||
| } | ||
|
|
||
| variable "ENVIRONMENT" { | ||
| default = "unknown-environment" | ||
| } | ||
|
|
||
| variable "REPO" { | ||
| default = "unknown-repo" | ||
| } | ||
|
|
||
| variable "FILE_PATH" { | ||
| description = "The local path to the file to upload" | ||
| type = string | ||
| default = "./files/test-alerts-v2.csv.gz" | ||
| } | ||
|
|
||
| variable "OBJECT_NAME" { | ||
| description = "The name of the object in the bucket" | ||
| type = string | ||
| default = "test-alerts-v2.csv.gz" | ||
| } | ||
|
|
||
| variable "BUCKET_REGION" { | ||
| description = "The region of the bucket" | ||
| type = string | ||
| default = "US" | ||
| } | ||
|
|
||
| // If testing using the elastic-siem account then update the default value for below | ||
| // mentioned variable GOOGLE_CREDENTIALS and service_account_key in test-gcs-config.yml | ||
| // with your actual credentials | ||
| variable "GOOGLE_CREDENTIALS" { | ||
| description = "GCP service account credentials in JSON format" | ||
| type = string | ||
| default = <<EOF | ||
| { | ||
| "type": "{account_type}", | ||
| "project_id": "{project_id}", | ||
| "private_key_id": "{private_key_id}", | ||
| "private_key": "{private_key}", | ||
| "client_email": "{client_email}", | ||
| "client_id": "{client_id}", | ||
| "auth_uri": "{auth_uri}", | ||
| "token_uri": "{token_uri}", | ||
| "auth_provider_x509_cert_url": "{auth_provider_x509_cert_url}", | ||
| "client_x509_cert_url": "{client_x509_cert_url}", | ||
| "universe_domain": "{universe_domain}" | ||
| } | ||
| EOF | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| deployer: docker | ||
| service: azure-blob-storage-emulator | ||
| input: azure-blob-storage | ||
| vars: | ||
| data_stream: | ||
| vars: | ||
| csv_comma: "," | ||
| account_name: devstoreaccount1 | ||
| service_account_key: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" | ||
| storage_url: "http://{{Hostname}}:{{Port}}/devstoreaccount1/" | ||
| containers: | | ||
| - name: test-container | ||
| max_workers: 3 | ||
| poll: true | ||
| poll_interval: 15s | ||
| assert: | ||
| hit_count: 3 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| deployer: tf | ||
| input: gcs | ||
| wait_for_data_timeout: 10m | ||
| data_stream: | ||
| vars: | ||
| csv_comma: "," | ||
| service_account_key: | | ||
| {{GOOGLE_CLOUD_KEYFILE_JSON}} | ||
| project_id: "{{GCLOUD_PROJECT}}" | ||
| buckets: | | ||
| - name: "{{TF_OUTPUT_gcs_netskope_alert_bucket_name}}" | ||
| poll: true | ||
| poll_interval: 15s | ||
| preserve_original_event: true | ||
| preserve_duplicate_custom_fields: true | ||
| assert: | ||
| hit_count: 3 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From what I can see, for Azure we can gzip in docker like this, and for AWS S3 we can do it in Terraform with
content_base64 = base64gzip(file("./files/events.csv")), but for GCS thegoogle_storage_bucket_objectresource doesn't have that option, so*.csv.gzfiles are comitted in the PR.I think that a similar thing can be done for GCS as for S3, but like this:
(doing the base64 decode because the only gzip function I see also base64 encodes).
Then the
*.csv.gzfiles can be removed.If that's impossible for some reason, maybe it's better to switch to only using having the
*.gzfiles committed.