Bullmq alon rebased (#3401)

* feat: webhooks implementation & queue system

* fix

* wip worker

* wip fixes

* worker is up

* finalized impl

* fix circular

* env.example

* fix rebase conflicts

* fix bug caused by rebase

* trigger CI

* add customer feature flag for opt-in queue system

* fix: env schema

* wip merge conflicts

* fixes

* lockfile

* env example

* Add Docker, Helm, and GitHub workflows for combined deployment

* feat(metrics): add BullMQ metrics integration and dashboards

- Introduce Prometheus metrics for BullMQ job states and integrate with OpenTelemetry
- Set up Grafana dashboards for visualizing BullMQ performance metrics
- Implement alerts and queuing system for better monitoring and job processing

* refactor(workflows-service): clean up shutdown logic and remove unused types

- Remove commented-out code related to graceful shutdown
- Delete unused type declaration file for BullMQ OpenTelemetry integration

* fix(monitoring): handle absence of BullMQ queues in Prometheus metrics export

- Add a check for the existence of BullMQ queue before exporting metrics
- Return a placeholder message if no queues are available
- Remove the unused OpenTelemetry initialization module from the project

* feat(workflows): improve configuration and queue management

- Introduce specific environment configuration for worker module
- Update Redis client configuration with max retries per request
- Enhance job scheduler with job ID configuration to avoid multiple tasks in the cron queue

* feat(monitoring): enhance metrics collection and docker configuration

- Add Prometheus metrics endpoint for BullMQ queues
- Configure Docker Compose for monitoring with Grafana and Prometheus
- Register BullMQ queues in Prometheus service for comprehensive metrics tracking

* refactor(env): change getEnvFilePath to an arrow function

- Convert function declaration to arrow function syntax
- Maintain existing functionality in retrieving environment file path

* fix(queue): initialize Redis connection conditionally

- Check if QUEUE_SYSTEM_ENABLED environment variable is set
- Only initialize Redis connection when enabled

* Revert "fix(queue): initialize Redis connection conditionally"

This reverts commit 67a1997.

* fix(queue): initialize Redis connection conditionally

- Check if QUEUE_SYSTEM_ENABLED environment variable is set
- Initialize Redis connection only when the feature is enabled

* chore(docs): remove outdated BullMQ metrics and monorepo navigation documentation

- Delete deprecated documentation files for BullMQ metrics implementation
- Remove outdated monorepo navigation and command execution guidelines

* fix(prometheus): remove public decorator from getMetrics endpoint

- Eliminate the public decorator for enhanced security
- Ensure metrics are accessible without public exposure

* chore(metrics): remove obsolete Grafana dashboard configuration

- Delete grafana-dashboard.json file as it is no longer needed
- Streamline application code for clarity and maintainability

fix(middleware): update request authorization checks for admin users

- Ensure middleware correctly accesses user type from session
- Improve security by preventing unauthorized access

feat(metrics): add metrics endpoint for Prometheus status

- Implement /prometheus endpoint to check metrics collection status
- Enhance team visibility into system health with descriptive messaging

* feat(middlewares): add metrics authentication middleware

- Implement middleware to validate metrics token from request headers
- Set up entity for metrics access if authentication is successful

* refactor(queue): improve queue initialization logic and metrics handling

- Rename and remove unnecessary metrics middleware from routes
- Update Redis client configuration to allow unlimited retries per request
- Enhance logging and setup checks for the queue system in relevant services

* fix(workflows-service): correct alwaysApply setting in rules

- Update alwaysApply to true for the workflows service rules
- Ensure proper application of rules during service execution

* update pnpm lock file

---------

Co-authored-by: blokh <[email protected]>
Co-authored-by: Sasha Mysak <[email protected]>
Co-authored-by: Alon Peretz <[email protected]>
Co-authored-by: Omer Harel <[email protected]>

Author: 5 people

.cursor/rules/workflows-service.mdc

@@ -1,6 +1,7 @@
 ---
 description: Workflow Service Rules
 globs: ["services/workflows-service/**/*.{ts}"]
+
 ---
 ### Code Organization & Structure
 

.vscode/extensions.json

@@ -8,6 +8,7 @@
     "bradlc.vscode-tailwindcss",
     "github.vscode-github-actions",
     "streetsidesoftware.code-spell-checker",
-    "orta.vscode-jest"
+    "orta.vscode-jest",
+    "alefragnani.bookmarks"
   ]
 }

.vscode/settings.json

@@ -13,5 +13,7 @@
     "**/data-migrations": false
   },
   "search.followSymlinks": true,
-  "search.useIgnoreFiles": false
+  "search.useIgnoreFiles": false,
+  "bookmarks.saveBookmarksInProject": true,
+  "bookmarks.saveBookmarksInProjectFilePath": ".vscode/bookmarks.json"
 }

apps/backoffice-v2/src/pages/MerchantMonitoring/components/MerchantMonitoringReportStatus/fetchers.ts

@@ -1,5 +1,5 @@
 import { z } from 'zod';
-import { UpdateableReportStatus } from '@ballerine/common';
+import type { UpdateableReportStatus } from '@ballerine/common';
 
 import { Method } from '@/common/enums';
 import { apiClient } from '@/common/api-client/api-client';

deploy/.env

@@ -50,4 +50,4 @@ VITE_KYB_DEFINITION_ID="kyb_parent_kyc_session_example"
 
 ##Workflow-Dashboard
 MODE=development
-VITE_IMAGE_LOGO_URL=
+VITE_IMAGE_LOGO_URL=