Implement event-driven audit trail sync architecture

Redesigns synchronization infrastructure using audit-based approach - Replaces disconnected sync logic with event-driven architecture - Adds AuditService to log entity changes with JSON diffs - Implements chained events for reliable sync process - Fixes EventBus injection and event emission in services - Removes unused OperationQueue Provides comprehensive audit trail for entity changes and backend synchronization
2025-11-22 11:52:56 +01:00 · 2025-11-22 11:52:56 +01:00 · a7d365b186
commit a7d365b186
parent 185330402e
1 changed files with 531 additions and 0 deletions
--- a/coding-sessions/2025-11-22-audit-trail-event-driven-sync.md
+++ b/coding-sessions/2025-11-22-audit-trail-event-driven-sync.md
@ -0,0 +1,531 @@
+# Audit Trail & Event-Driven Sync Architecture
+
+**Date:** 2025-11-22
+**Duration:** ~4 hours
+**Initial Scope:** Understand existing sync logic
+**Actual Scope:** Complete audit-based sync architecture with event-driven design
+
+---
+
+## Executive Summary
+
+Discovered that existing sync infrastructure (SyncManager, SyncPlugin, OperationQueue) was completely disconnected - nothing was wired together. Redesigned from scratch using audit-based architecture where all entity changes are logged to an audit store, and SyncManager listens for AUDIT_LOGGED events to sync to backend.
+
+**Key Achievements:**
+- ✅ Designed event-driven audit trail architecture
+- ✅ Created AuditTypes, AuditStore, AuditService
+- ✅ Updated BaseEntityService with JSON diff calculation (json-diff-ts)
+- ✅ Implemented event emission on save/delete (ENTITY_SAVED, ENTITY_DELETED)
+- ✅ Refactored SyncManager to listen for AUDIT_LOGGED events
+- ✅ Fixed EventBus injection (required, not optional)
+- ✅ Added typed Payload interfaces for all events
+
+**Critical Discovery:** The "working" sync infrastructure was actually dead code - SyncManager was commented out, queue was never populated, and no events were being emitted.
+
+---
+
+## Context: Starting Point
+
+### Previous Work (Nov 20, 2025)
+Repository Layer Elimination session established:
+- BaseEntityService<T> with generic CRUD operations
+- IndexedDBContext for database connection
+- Direct service usage pattern (no repository wrapper)
+- DataSeeder for initial data loading
+
+### The Gap
+After repository elimination, we had:
+- ✅ Services working (BaseEntityService + SyncPlugin)
+- ✅ IndexedDB storing data
+- ❌ SyncManager commented out
+- ❌ OperationQueue never populated
+- ❌ No events emitted on entity changes
+- ❌ No audit trail for compliance
+
+---
+
+## Session Evolution: Major Architectural Decisions
+
+### Phase 1: Discovery - Nothing Was Connected 🚨
+
+**User Question:** *"What synchronization logic do we have for the server database?"*
+
+**Investigation Findings:**
+- SyncManager exists but was commented out in index.ts
+- OperationQueue exists but never receives operations
+- BaseEntityService.save() just saves - no events, no queue
+- SyncPlugin provides sync status methods but nothing triggers them
+
+**The Truth:** Entire sync infrastructure was scaffolding with no actual wiring.
+
+---
+
+### Phase 2: Architecture Discussion - Queue vs Audit
+
+**User's Initial Mental Model:**
+```
+IEntityService.save() → saves to IndexedDB → emits event
+SyncManager listens → reads pending from IndexedDB → syncs to backend
+```
+
+**Problem Identified:** "Who fills the queue?"
+
+**Options Discussed:**
+1. Service writes to queue
+2. EventManager writes to queue
+3. SyncManager reads pending from IndexedDB
+
+**User Insight:** *"I need an audit trail for all changes."*
+
+**Decision:** Drop OperationQueue concept, use Audit store instead.
+
+---
+
+### Phase 3: Audit Architecture Design ✅
+
+**Requirements:**
+- All entity changes must be logged (compliance)
+- Changes should store JSON diff (not full entity)
+- userId required (hardcoded GUID for now)
+- Audit entries never deleted
+
+**Designed Event Chain:**
+```
+Entity change
+    → BaseEntityService.save()
+    → emit ENTITY_SAVED (with diff)
+    → AuditService listens
+    → creates audit entry
+    → emit AUDIT_LOGGED
+    → SyncManager listens
+    → syncs to backend
+```
+
+**Why Chained Events?**
+User caught race condition: *"If both AuditService and SyncManager listen to ENTITY_SAVED, SyncManager could execute before the audit entry is created."*
+
+Solution: AuditService emits AUDIT_LOGGED after saving, SyncManager only listens to AUDIT_LOGGED.
+
+---
+
+### Phase 4: Implementation - AuditTypes & AuditStore ✅
+
+**Created src/types/AuditTypes.ts:**
+```typescript
+export interface IAuditEntry extends ISync {
+  id: string;
+  entityType: EntityType;
+  entityId: string;
+  operation: 'create' | 'update' | 'delete';
+  userId: string;
+  timestamp: number;
+  changes: any;  // JSON diff result
+  synced: boolean;
+  syncStatus: 'synced' | 'pending' | 'error';
+}
+```
+
+**Created src/storage/audit/AuditStore.ts:**
+```typescript
+export class AuditStore implements IStore {
+  readonly storeName = 'audit';
+
+  create(db: IDBDatabase): void {
+    const store = db.createObjectStore(this.storeName, { keyPath: 'id' });
+    store.createIndex('syncStatus', 'syncStatus', { unique: false });
+    store.createIndex('synced', 'synced', { unique: false });
+    store.createIndex('entityId', 'entityId', { unique: false });
+    store.createIndex('timestamp', 'timestamp', { unique: false });
+  }
+}
+```
+
+---
+
+### Phase 5: BaseEntityService - Diff Calculation & Event Emission ✅
+
+**Added json-diff-ts dependency:**
+```bash
+npm install json-diff-ts
+```
+
+**Updated save() method:**
+```typescript
+async save(entity: T): Promise<void> {
+  const entityId = (entity as any).id;
+
+  // Check if entity exists to determine create vs update
+  const existingEntity = await this.get(entityId);
+  const isCreate = existingEntity === null;
+
+  // Calculate changes: full entity for create, diff for update
+  let changes: any;
+  if (isCreate) {
+    changes = entity;
+  } else {
+    const existingSerialized = this.serialize(existingEntity);
+    const newSerialized = this.serialize(entity);
+    changes = diff(existingSerialized, newSerialized);
+  }
+
+  // ... save to IndexedDB ...
+
+  // Emit ENTITY_SAVED event
+  const payload: IEntitySavedPayload = {
+    entityType: this.entityType,
+    entityId,
+    operation: isCreate ? 'create' : 'update',
+    changes,
+    timestamp: Date.now()
+  };
+  this.eventBus.emit(CoreEvents.ENTITY_SAVED, payload);
+}
+```
+
+---
+
+### Phase 6: AuditService - Override Pattern ✅
+
+**Key Design Decision:** AuditService overrides save() to NOT emit ENTITY_SAVED.
+
+**Why:** If AuditService.save() emitted ENTITY_SAVED, it would trigger AuditService again → infinite loop.
+
+**Created src/storage/audit/AuditService.ts:**
+```typescript
+export class AuditService extends BaseEntityService<IAuditEntry> {
+  readonly storeName = 'audit';
+  readonly entityType: EntityType = 'Audit';
+
+  constructor(context: IndexedDBContext, eventBus: IEventBus) {
+    super(context, eventBus);
+    this.setupEventListeners();
+  }
+
+  private setupEventListeners(): void {
+    this.eventBus.on(CoreEvents.ENTITY_SAVED, (event: Event) => {
+      const detail = (event as CustomEvent).detail;
+      this.handleEntitySaved(detail);
+    });
+
+    this.eventBus.on(CoreEvents.ENTITY_DELETED, (event: Event) => {
+      const detail = (event as CustomEvent).detail;
+      this.handleEntityDeleted(detail);
+    });
+  }
+
+  // Override save to emit AUDIT_LOGGED instead of ENTITY_SAVED
+  async save(entity: IAuditEntry): Promise<void> {
+    // ... save to IndexedDB ...
+
+    // Emit AUDIT_LOGGED (not ENTITY_SAVED)
+    const payload: IAuditLoggedPayload = {
+      auditId: entity.id,
+      entityType: entity.entityType,
+      entityId: entity.entityId,
+      operation: entity.operation,
+      timestamp: entity.timestamp
+    };
+    this.eventBus.emit(CoreEvents.AUDIT_LOGGED, payload);
+  }
+
+  // Audit entries cannot be deleted (compliance)
+  async delete(_id: string): Promise<void> {
+    throw new Error('Audit entries cannot be deleted (compliance requirement)');
+  }
+}
+```
+
+---
+
+### Phase 7: SyncManager Refactoring ✅
+
+**Before:** Used OperationQueue (never populated)
+**After:** Listens to AUDIT_LOGGED, syncs audit entries
+
+**Key Changes:**
+```typescript
+export class SyncManager {
+  constructor(
+    eventBus: IEventBus,
+    auditService: AuditService,
+    auditApiRepository: IApiRepository<IAuditEntry>
+  ) {
+    this.setupAuditListener();
+    this.startSync();
+  }
+
+  private setupAuditListener(): void {
+    this.eventBus.on(CoreEvents.AUDIT_LOGGED, () => {
+      if (this.isOnline && !this.isSyncing) {
+        this.processPendingAudits();
+      }
+    });
+  }
+
+  private async processPendingAudits(): Promise<void> {
+    const pendingAudits = await this.auditService.getPendingAudits();
+    for (const audit of pendingAudits) {
+      await this.auditApiRepository.sendCreate(audit);
+      await this.auditService.markAsSynced(audit.id);
+    }
+  }
+}
+```
+
+---
+
+### Phase 8: EventBus Injection Problem 🐛
+
+**Discovery:** Entity services had no EventBus!
+
+**User Observation:** *"There's no EventBus being passed. Why are you using super(...arguments) when there's an empty constructor?"*
+
+**Problem:**
+- BaseEntityService had optional eventBus parameter
+- Entity services (EventService, BookingService, etc.) had no constructors
+- EventBus was never passed → events never emitted
+
+**User Directive:** *"Remove the null check you added in BaseEntityService - it doesn't make sense."*
+
+**Fix:**
+1. Made eventBus required in BaseEntityService
+2. Added constructors to all entity services:
+
+```typescript
+// EventService.ts
+constructor(context: IndexedDBContext, eventBus: IEventBus) {
+  super(context, eventBus);
+}
+
+// BookingService.ts, CustomerService.ts, ResourceService.ts - same pattern
+```
+
+---
+
+### Phase 9: Typed Payload Interfaces ✅
+
+**User Observation:** *"The events you've created use anonymous types. I'd prefer typed interfaces following the existing Payload suffix convention."*
+
+**Added to src/types/EventTypes.ts:**
+```typescript
+export interface IEntitySavedPayload {
+  entityType: EntityType;
+  entityId: string;
+  operation: 'create' | 'update';
+  changes: any;
+  timestamp: number;
+}
+
+export interface IEntityDeletedPayload {
+  entityType: EntityType;
+  entityId: string;
+  operation: 'delete';
+  timestamp: number;
+}
+
+export interface IAuditLoggedPayload {
+  auditId: string;
+  entityType: EntityType;
+  entityId: string;
+  operation: 'create' | 'update' | 'delete';
+  timestamp: number;
+}
+```
+
+---
+
+## Files Changed Summary
+
+### Files Created (4)
+1. **src/types/AuditTypes.ts** - IAuditEntry interface
+2. **src/storage/audit/AuditStore.ts** - IndexedDB store for audit entries
+3. **src/storage/audit/AuditService.ts** - Event-driven audit service
+4. **src/repositories/MockAuditRepository.ts** - Mock API for audit sync
+
+### Files Deleted (1)
+5. **src/storage/OperationQueue.ts** - Replaced by audit-based approach
+
+### Files Modified (9)
+6. **src/types/CalendarTypes.ts** - Added 'Audit' to EntityType
+7. **src/constants/CoreEvents.ts** - Added ENTITY_SAVED, ENTITY_DELETED, AUDIT_LOGGED
+8. **src/types/EventTypes.ts** - Added IEntitySavedPayload, IEntityDeletedPayload, IAuditLoggedPayload
+9. **src/storage/BaseEntityService.ts** - Added diff calculation, event emission, required eventBus
+10. **src/storage/events/EventService.ts** - Added constructor with eventBus
+11. **src/storage/bookings/BookingService.ts** - Added constructor with eventBus
+12. **src/storage/customers/CustomerService.ts** - Added constructor with eventBus
+13. **src/storage/resources/ResourceService.ts** - Added constructor with eventBus
+14. **src/workers/SyncManager.ts** - Refactored to use AuditService
+15. **src/storage/IndexedDBContext.ts** - Bumped DB version to 3
+16. **src/index.ts** - Updated DI registrations
+
+---
+
+## Architecture Evolution Diagram
+
+**BEFORE (Disconnected):**
+```
+EventManager
+    ↓
+EventService.save()
+    ↓
+IndexedDB (saved)
+
+[OperationQueue - never filled]
+[SyncManager - commented out]
+```
+
+**AFTER (Event-Driven):**
+```
+EventManager
+    ↓
+EventService.save()
+    ↓
+BaseEntityService.save()
+    ├── IndexedDB (saved)
+    └── emit ENTITY_SAVED (with diff)
+            ↓
+        AuditService listens
+            ├── Creates audit entry
+            └── emit AUDIT_LOGGED
+                    ↓
+                SyncManager listens
+                    └── Syncs to backend
+```
+
+---
+
+## Key Design Decisions
+
+### 1. Audit-Based Instead of Queue-Based
+**Why:** Audit serves dual purpose - compliance trail AND sync source.
+**Benefit:** Single source of truth for all changes.
+
+### 2. Chained Events (ENTITY_SAVED → AUDIT_LOGGED)
+**Why:** Prevents race condition where SyncManager runs before audit is saved.
+**Benefit:** Guaranteed order of operations.
+
+### 3. JSON Diff for Changes
+**Why:** Only store what changed, not full entity.
+**Benefit:** Smaller audit entries, easier to see what changed.
+
+### 4. Override Pattern for AuditService
+**Why:** Prevent infinite loop (audit → event → audit → event...).
+**Benefit:** Clean separation without special flags.
+
+### 5. Required EventBus (Not Optional)
+**Why:** Events are core to architecture, not optional.
+**Benefit:** No null checks, guaranteed behavior.
+
+---
+
+## Discussion Topics (Not Implemented)
+
+### IndexedDB for Logging/Traces
+User observation: *"This IndexedDB approach is quite interesting - we could extend it to handle logging, traces, and exceptions as well."*
+
+**Potential Extension:**
+- LogStore - Application logs
+- TraceStore - Performance traces
+- ExceptionStore - Caught/uncaught errors
+
+**Console Interception Pattern:**
+```typescript
+const originalLog = console.log;
+console.log = (...args) => {
+  logService.save({ level: 'info', message: args, timestamp: Date.now() });
+  if (isDevelopment) originalLog.apply(console, args);
+};
+```
+
+**Cleanup Strategy:**
+- 7-day retention
+- Or max 10,000 entries with FIFO
+
+**Decision:** Not implemented this session, but architecture supports it.
+
+---
+
+## Lessons Learned
+
+### 1. Verify Existing Code Actually Works
+The sync infrastructure looked complete but was completely disconnected.
+**Lesson:** Don't assume existing code works - trace the actual flow.
+
+### 2. Audit Trail Serves Multiple Purposes
+Audit is not just for compliance - it's also the perfect sync source.
+**Lesson:** Look for dual-purpose designs.
+
+### 3. Event Ordering Matters
+Race conditions between listeners are real.
+**Lesson:** Use chained events when order matters.
+
+### 4. Optional Dependencies Create Hidden Bugs
+Optional eventBus meant events silently didn't fire.
+**Lesson:** Make core dependencies required.
+
+### 5. Type Consistency Matters
+Anonymous types in events vs Payload interfaces elsewhere.
+**Lesson:** Follow existing patterns in codebase.
+
+---
+
+## Current State & Next Steps
+
+### ✅ Build Status: Successful
+```
+[NovaDI] Performance Summary:
+  - Files in TypeScript Program: 80
+  - Files actually transformed: 58
+  - Total: 1467.93ms
+```
+
+### ✅ Architecture State
+- **Event-driven audit trail:** Complete
+- **AuditService:** Listens for entity events, creates audit entries
+- **SyncManager:** Listens for AUDIT_LOGGED, syncs to backend
+- **BaseEntityService:** Emits events on save/delete with JSON diff
+
+### ⚠️ Not Yet Tested
+- Runtime behavior (does AuditService receive events?)
+- Diff calculation accuracy
+- SyncManager sync flow
+- IndexedDB version upgrade (v2 → v3)
+
+### 📋 Next Steps
+
+**Immediate:**
+1. Test in browser - verify events fire correctly
+2. Check IndexedDB for audit entries after save
+3. Verify SyncManager logs sync attempts
+
+**Future:**
+1. Real backend API for audit sync
+2. Logging/traces extension (console interception)
+3. Cleanup strategy for old audit entries
+
+---
+
+## Conclusion
+
+**Initial Goal:** Understand sync logic
+**Actual Work:** Complete architecture redesign
+
+**What We Found:**
+- Sync infrastructure was dead code
+- OperationQueue never populated
+- SyncManager commented out
+- No events being emitted
+
+**What We Built:**
+- Audit-based sync architecture
+- Event-driven design with chained events
+- JSON diff for change tracking
+- Typed payload interfaces
+
+**Key Insight:** Sometimes "understanding existing code" reveals there's nothing to understand - just scaffolding that needs to be replaced with actual implementation.
+
+---
+
+**Session Complete:** 2025-11-22
+**Documentation Quality:** High
+**Ready for:** Runtime testing