feat(datastores): add DataStore API and registry infrastructure #5054
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Create public DataStore API for unified access to system state: New files: - api/v1beta1/datastores/types.go: Core data types * ProcessInfo, ContainerInfo, SymbolInfo, DNSResponse * HealthInfo, HealthStatus, DataStoreMetrics * DataStoreMetadata, error types - api/v1beta1/datastores/interfaces.go: Store interfaces * DataStore: Base interface with health & metrics * ProcessStore: Process tree access * ContainerStore: Container information * KernelSymbolStore: Symbol resolution * DNSCacheStore: DNS query cache - api/v1beta1/datastores/registry.go: Registry interface * Unified access to all datastores * Custom store registration support * Metadata and availability queries
- Add store.go files implementing DataStore and specific store interfaces - Process: GetProcess, GetChildProcesses - Container: GetContainer, GetContainerByName - Symbol: ResolveSymbolByAddress (returns all aliases), GetSymbolAddress, ResolveSymbolsBatch - DNS: GetDNSResponse - Track lastAccess timestamp for operational metrics - Rename Manager.GetContainer -> LookupContainer to avoid method collision All datastores implement Name(), GetHealth(), GetMetrics() for monitoring.
Add tests for DataStore interface implementations in process, container, symbol, and dns datastores, validating Name(), GetHealth(), GetMetrics(), store-specific methods, and LastAccess tracking.
Implement Registry providing centralized, thread-safe access to all registered datastores: - Exported Registry struct with RegisterStore() for Tracee engine - Implements datastores.Registry interface for detector access - Type-safe accessor methods for core stores - Generic GetCustom() for extension datastores - Store listing, availability checks, and metadata retrieval - Comprehensive tests covering all functionality Implements single concrete type usable both for registration (Tracee engine) and read-only access (detectors via interface).
Initialize the datastore registry in Tracee.Init() and register core datastores (process, container, symbol, dns) for access by detectors and extensions. Kernel symbol store uses an adapter to support runtime hot-reload without stale references.
Add SystemStore interface and SystemInfo type to provide detectors with read-only access to system metadata collected at startup. SystemInfo includes: - Architecture and kernel version - Hostname and boot time - OS distribution info (name, version, pretty name) - Tracee version - Init process namespaces Design decisions: - Single struct return for simplicity and forward compatibility - Immutable data (collected once, never changes) - Optional datastore (detectors opt-in via requirements)
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a unified DataStore API and registry infrastructure that provides read-only access to system state for detectors and extensions. It establishes a foundation layer for the upcoming detector framework by creating standardized interfaces and implementations for accessing process trees, containers, kernel symbols, DNS cache, system information, and syscall metadata.
Key Changes
- Added API interfaces defining contracts for six datastore types (ProcessStore, ContainerStore, KernelSymbolStore, DNSStore, SystemStore, SyscallStore) with a unified Registry for centralized access
- Implemented store wrappers around existing Tracee components with health monitoring, metrics tracking, and thread-safe access patterns
- Integrated the registry into Tracee's initialization lifecycle with proper store registration and availability checks
Reviewed Changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| api/v1beta1/datastores/interfaces.go | Defines base DataStore interface and six specialized store interfaces with standardized error handling |
| api/v1beta1/datastores/registry.go | Registry interface for unified access to all datastores |
| api/v1beta1/datastores/types.go | Common data types and structures shared across datastores |
| pkg/datastores/registry.go | Thread-safe registry implementation with type-safe accessor methods |
| pkg/datastores/system/collector.go | System information collection at startup |
| pkg/datastores/system/store.go | Immutable system information store implementation |
| pkg/datastores/syscall/store.go | Syscall ID/name mapping store implementation |
| pkg/datastores/symbol/adapter.go | Adapter for hot-reloadable kernel symbol tables |
| pkg/datastores/symbol/store.go | Kernel symbol resolution with DataStore interface |
| pkg/datastores/process/store.go | Process tree access with DataStore interface |
| pkg/datastores/dns/store.go | DNS cache access with DataStore interface |
| pkg/datastores/container/store.go | Container information access with DataStore interface |
| pkg/ebpf/tracee.go | Integration of datastore registry into Tracee initialization |
Implement immutable SystemStore with collector, store, and tests. Collector gathers system info using existing utilities (environment.GetOSInfo, timeutil.GetBootTime, init namespaces from /proc/1/ns). Store is always healthy with zero dynamic metrics (immutable read-only data). Tests verify collection, interface implementation, and immutability.
Register system datastore in registry with System() accessor. Collect system info at Tracee startup after timeutil.Init() and register as optional datastore (non-critical for operation).
Implement a new SyscallStore datastore that provides: - GetSyscallName(id int32): lookup syscall name by ID - GetSyscallID(name string): lookup syscall ID by name The store wraps events.Core and provides architecture-specific syscall information (x86 vs ARM). Uses int32 for syscall IDs to match kernel ABI and protobuf definitions. This enables detectors to resolve syscall names without direct access to internal pkg/events structures. Includes comprehensive unit tests
yanivagman
force-pushed
the
datastores
branch
from
fa86257 to
242a5ec
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5054 +/- ##
==========================================
+ Coverage 29.72% 30.52% +0.79%
==========================================
Files 234 237 +3
Lines 26186 26171 -15
==========================================
+ Hits 7785 7988 +203
+ Misses 17864 17650 -214
+ Partials 537 533 -4
🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a unified DataStore API (
api/v1beta1/datastores/) that providesread-only access to system state for detectors and extensions. This is the
foundation layer for the detector framework.
What's Included
API Interfaces (
api/v1beta1/datastores/):DataStore- Base interface with health & metricsProcessStore- Process tree accessContainerStore- Container informationKernelSymbolStore- Symbol resolutionDNSCacheStore- DNS query cacheSystemStore- Immutable system info (OS, kernel, hostname)SyscallStore- Syscall ID/name mappingRegistry- Unified access to all storesStore Implementations (
pkg/datastores/):Registry (
pkg/datastores/registry.go):Testing
make test-unit PKG=pkg/datastoresImpact