Handle Array Exists/Not Exists in Flat Collections

[suddendust]

Description

For entity-service queries that check for an array to be IS EMPTY or IS NOT EMPTY (this is what ES receives from GQL):

{
 "entityType": "API",
 "filter": {
   "operator": "AND",
   "childFilter": [
     {
       "lhs": {
         "columnIdentifier": {
           "columnName": "API.responseDataTypeIds",
           "alias": "responseDataTypeIds"
         }
       },
       "operator": "NEQ",
       "rhs": {
         "literal": {
           "value": {
             "valueType": "STRING",
             "string": "null"
           }
         }
       }
     }
   ]
 },
 "selection": [
   {
     "columnIdentifier": {
       "columnName": "API.id"
     }
   },
   {
     "columnIdentifier": {
       "columnName": "API.responseDataTypeIds",
       "alias": "responseDataTypeIds"
     }
   }
 ]
}

The currently parsed PG query is:

SELECT "id" AS "API.id", "attributes.isLearnt" AS "isLearnt", "attributes.isAuthenticated" AS "isAuthenticated", "attributes.responseDataTypeIds" AS "responseDataTypeIds" FROM "entities_api" WHERE (("attributes.responseDataTypeIds" IS NOT NULL)) AND ("tenantId" = ('3e761879-c77b-4d8f-a075-62ff28e8fa8a'))

This predicate "attributes.responseDataTypeIds" IS NOT NULL is semantically incorrect as this returns empty arrays, which is not in sync-with Mongo's behaviour (which only returns arrays with length > 0). We need to align this behaviour.

These changes use JsonIdentifierExpression to identify array fields in JSON cols, and based on that information, it uses the cardinality operator to check the length of an array.

Summary

Note: For JSONB fields, the parser assumes a GIN index on them (so for customAttribute->tags, we've a GIN index on customAttribute).

Query	Operator	Field Type	Generated Filter Clause
EXISTS on top-level scalar field	EXISTS	Scalar	"status" IS NOT NULL
EXISTS on top-level array field	EXISTS	Array	(cardinality("tags") > 0)
EXISTS on JSONB scalar field	EXISTS	JSONB Scalar	"customAttribute" ? 'brand'
EXISTS on JSONB array field	EXISTS	JSONB Array	("props" @> '{"colors": []}' AND jsonb_array_length("props"->'colors') > 0)
NOT EXISTS on top-level scalar field	NOT_EXISTS	Scalar	"status" IS NULL
NOT EXISTS on top-level array field	NOT_EXISTS	Array	COALESCE(cardinality("tags"), 0) = 0
NOT EXISTS on JSONB scalar field	NOT_EXISTS	JSONB Scalar	NOT ("customAttribute" ? 'brand')
NOT EXISTS on JSONB array field	NOT_EXISTS	JSONB Array	COALESCE(jsonb_array_length("props"->'colors'), 0) = 0

Checklist:

• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• Any dependent changes have been merged and published in downstream modules

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.02%. Comparing base (c7bb6ad) to head (1b2cc0a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #250      +/-   ##
============================================
+ Coverage     79.70%   80.02%   +0.31%     
- Complexity     1209     1249      +40     
============================================
  Files           226      227       +1     
  Lines          5701     5756      +55     
  Branches        492      500       +8     
============================================
+ Hits           4544     4606      +62     
+ Misses          809      805       -4     
+ Partials        348      345       -3     

Flag	Coverage Δ
integration	80.02% <100.00%> (+0.31%)	⬆️
unit	58.06% <100.00%> (+0.73%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.