Skip to content

Commit 4ae6089

Browse files
ciaranschutteCiaran Schuttejustincorrigible
committed
Feature/federated (#926)
* Network aggregation search - Initial Config (#876) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feat/#873 retrieve remote schemas (#877) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Pr feedback - doc updates (#879) * update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/#874 merge schemas (#878) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Add core functionality for remote connection resolvers (#880) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/agg resolver (#881) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolve stats (#883) * add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolver responses (#884) * add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix network agg resolution (#885) * add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Response data & batching requests (#887) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed Search request timeouts (#889) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <[email protected]> * Reorg connection (#891) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <[email protected]> * Accumulator pipeline (#893) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed total hits (#895) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <[email protected]> * feat/agg acc tests (#897) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed numeric agg resolution (#898) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * use Math methods (#900) Co-authored-by: Ciaran Schutte <[email protected]> * Individual node config query fields (#899) * use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/aggregate not available (#903) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/filters (#904) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Minor comment updates (#905) * improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed search cleanup typings (#906) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix broken build (#907) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <[email protected]> * add agg mode env config (#908) * add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feature/threshold calc (#909) * add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <[email protected]> * multiple node with data thresholding (#912) * fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search cleanup (#920) * add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search networked cleanup (#921) * cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <[email protected]> * Update modules/server/src/app.js Co-authored-by: Anders Richardsson <[email protected]> * make horrible ajax.ts console logging somewhat useful * improve release process * fix paths and imports for new build config * update tests to todos * fix some imports * fix build issues with lodash and missing config flag * fix paths * comment out test files, broken with tsx vs jest * fix path * fix path * improve release process * allow toolbar options customization * fix setsIndex conflict resolutioon misses * default for ENABLE_DOCUMENT_HITS env flag is true * default enable document hits to true * add DATA_MASK_MIN_THRESHOLD env var as pass through var * debug * debug * debug * add default param value * debug * string util important fix * fix for tests * Add comment re data masking * remove debug log * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/masking.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolveHitsFromAggs.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * remove specific NetworkAggError class * fix error response type narrow * missing param * add explanation comment for aggs resolver, gql info lookup --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>
1 parent 0ad9610 commit 4ae6089

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+16617
-177
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
lerna-debug.log
22
.groovylintrc.json
33
.vscode/*.log
4+
.vscode/launch.json
45

56
**/dist
67
**/node_modules

‎integration-tests/server/test/readSearchData.js‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,8 @@ export default ({ api, documentType, gqlPath }) => {
9999
},
100100
},
101101
},
102-
});
102+
},
103+
);
103104

104105
assert.deepEqual(
105106
await api
@@ -141,7 +142,8 @@ export default ({ api, documentType, gqlPath }) => {
141142
},
142143
},
143144
},
144-
});
145+
},
146+
);
145147

146148
assert.deepEqual(
147149
await api
@@ -188,7 +190,8 @@ export default ({ api, documentType, gqlPath }) => {
188190
},
189191
},
190192
},
191-
});
193+
},
194+
);
192195

193196
assert.deepEqual(
194197
await api
@@ -235,7 +238,8 @@ export default ({ api, documentType, gqlPath }) => {
235238
},
236239
},
237240
},
238-
});
241+
},
242+
);
239243
});
240244

241245
test('3.excludes access_denied files', async () => {
@@ -264,7 +268,7 @@ export default ({ api, documentType, gqlPath }) => {
264268

265269
assert.deepEqual(
266270
data?.data?.[documentType]?.hits?.edges?.every((edge) => !edge.node.access_denied),
267-
true
271+
true,
268272
);
269273
});
270274

@@ -306,9 +310,6 @@ export default ({ api, documentType, gqlPath }) => {
306310
console.log('readSearchData error', err);
307311
});
308312

309-
assert.deepEqual(
310-
data?.data?.[documentType]?.hits?.edges?.length,
311-
0
312-
);
313+
assert.deepEqual(data?.data?.[documentType]?.hits?.edges?.length, 0);
313314
});
314315
};

‎modules/server/.env.schema‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
ALLOW_CUSTOM_MAX_DOWNLOAD_ROWS=false
22
CONFIG_PATH=./configs
3+
DATA_MASK_MIN_THRESHOLD=
34
DEBUG=false
45
DOCUMENT_TYPE=''
56
DOWNLOAD_STREAM_BUFFER_SIZE=2000
7+
ENABLE_DOCUMENT_HITS=true
68
ENABLE_LOGS=false
79
ES_HOST=http://localhost:9200
810
ES_INDEX=''
@@ -14,4 +16,4 @@ MAX_LIVE_VERSIONS=3
1416
MAX_RESULTS_WINDOW=10000
1517
PING_MS=2200
1618
PING_PATH=/ping
17-
PORT=5050
19+
PORT=5050

‎modules/server/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,11 @@ Build (required before publish):
1818
```
1919
npm run prepare
2020
```
21+
22+
## Federated Search
23+
24+
[Placeholder]
25+
26+
### Config
27+
28+
[Placeholder]

‎modules/server/configTemplates/configs.json.schema‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,4 +73,13 @@
7373
"maxResultsWindow": 10000,
7474
"rowIdFieldName": "analysis.analysis_id"
7575
}
76+
"network": {
77+
"servers": [
78+
{
79+
"displayName": "Toronto",
80+
"graphqlUrl": "http://<URL>/graphql",
81+
"documentType": "file"
82+
}
83+
]
84+
}
7685
}

‎modules/server/configTemplates/network.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"network": {
3+
"servers": []
4+
}
5+
}

0 commit comments

Comments
 (0)