Implement event-driven audit trail sync architecture

Redesigns synchronization infrastructure using audit-based approach - Replaces disconnected sync logic with event-driven architecture - Adds AuditService to log entity changes with JSON diffs - Implements chained events for reliable sync process - Fixes EventBus injection and event emission in services - Removes unused OperationQueue Provides comprehensive audit trail for entity changes and backend synchronization
2025-11-22 11:52:56 +01:00 · 2025-11-22 11:52:56 +01:00 · a7d365b186
commit a7d365b186
parent 185330402e
1 changed files with 531 additions and 0 deletions
--- a/coding-sessions/2025-11-22-audit-trail-event-driven-sync.md
+++ b/coding-sessions/2025-11-22-audit-trail-event-driven-sync.md
@ -0,0 +1,531 @@
 # Audit Trail & Event-Driven Sync Architecture
 **Date:** 2025-11-22
 **Duration:** ~4 hours
 **Initial Scope:** Understand existing sync logic
 **Actual Scope:** Complete audit-based sync architecture with event-driven design
 ---
 ## Executive Summary
 Discovered that existing sync infrastructure (SyncManager, SyncPlugin, OperationQueue) was completely disconnected - nothing was wired together. Redesigned from scratch using audit-based architecture where all entity changes are logged to an audit store, and SyncManager listens for AUDIT_LOGGED events to sync to backend.
 **Key Achievements:**
 - ✅ Designed event-driven audit trail architecture
 - ✅ Created AuditTypes, AuditStore, AuditService
 - ✅ Updated BaseEntityService with JSON diff calculation (json-diff-ts)
 - ✅ Implemented event emission on save/delete (ENTITY_SAVED, ENTITY_DELETED)
 - ✅ Refactored SyncManager to listen for AUDIT_LOGGED events
 - ✅ Fixed EventBus injection (required, not optional)
 - ✅ Added typed Payload interfaces for all events
 **Critical Discovery:** The "working" sync infrastructure was actually dead code - SyncManager was commented out, queue was never populated, and no events were being emitted.
 ---
 ## Context: Starting Point
 ### Previous Work (Nov 20, 2025)
 Repository Layer Elimination session established:
 - BaseEntityService<T> with generic CRUD operations
 - IndexedDBContext for database connection
 - Direct service usage pattern (no repository wrapper)
 - DataSeeder for initial data loading
 ### The Gap
 After repository elimination, we had:
 - ✅ Services working (BaseEntityService + SyncPlugin)
 - ✅ IndexedDB storing data
 - ❌ SyncManager commented out
 - ❌ OperationQueue never populated
 - ❌ No events emitted on entity changes
 - ❌ No audit trail for compliance
 ---
 ## Session Evolution: Major Architectural Decisions
 ### Phase 1: Discovery - Nothing Was Connected 🚨
 **User Question:** *"What synchronization logic do we have for the server database?"*
 **Investigation Findings:**
 - SyncManager exists but was commented out in index.ts
 - OperationQueue exists but never receives operations
 - BaseEntityService.save() just saves - no events, no queue
 - SyncPlugin provides sync status methods but nothing triggers them
 **The Truth:** Entire sync infrastructure was scaffolding with no actual wiring.
 ---
 ### Phase 2: Architecture Discussion - Queue vs Audit
 **User's Initial Mental Model:**
 ```
 IEntityService.save() → saves to IndexedDB → emits event
 SyncManager listens → reads pending from IndexedDB → syncs to backend
 ```
 **Problem Identified:** "Who fills the queue?"
 **Options Discussed:**
 1. Service writes to queue
 2. EventManager writes to queue
 3. SyncManager reads pending from IndexedDB
 **User Insight:** *"I need an audit trail for all changes."*
 **Decision:** Drop OperationQueue concept, use Audit store instead.
 ---
 ### Phase 3: Audit Architecture Design ✅
 **Requirements:**
 - All entity changes must be logged (compliance)
 - Changes should store JSON diff (not full entity)
 - userId required (hardcoded GUID for now)
 - Audit entries never deleted
 **Designed Event Chain:**
 ```
 Entity change
    → BaseEntityService.save()
    → emit ENTITY_SAVED (with diff)
    → AuditService listens
    → creates audit entry
    → emit AUDIT_LOGGED
    → SyncManager listens
    → syncs to backend
 ```
 **Why Chained Events?**
 User caught race condition: *"If both AuditService and SyncManager listen to ENTITY_SAVED, SyncManager could execute before the audit entry is created."*
 Solution: AuditService emits AUDIT_LOGGED after saving, SyncManager only listens to AUDIT_LOGGED.
 ---
 ### Phase 4: Implementation - AuditTypes & AuditStore ✅
 **Created src/types/AuditTypes.ts:**
 ```typescript
 export interface IAuditEntry extends ISync {
  id: string;
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update' | 'delete';
  userId: string;
  timestamp: number;
  changes: any;  // JSON diff result
  synced: boolean;
  syncStatus: 'synced' | 'pending' | 'error';
 }
 ```
 **Created src/storage/audit/AuditStore.ts:**
 ```typescript
 export class AuditStore implements IStore {
  readonly storeName = 'audit';
  create(db: IDBDatabase): void {
    const store = db.createObjectStore(this.storeName, { keyPath: 'id' });
    store.createIndex('syncStatus', 'syncStatus', { unique: false });
    store.createIndex('synced', 'synced', { unique: false });
    store.createIndex('entityId', 'entityId', { unique: false });
    store.createIndex('timestamp', 'timestamp', { unique: false });
  }
 }
 ```
 ---
 ### Phase 5: BaseEntityService - Diff Calculation & Event Emission ✅
 **Added json-diff-ts dependency:**
 ```bash
 npm install json-diff-ts
 ```
 **Updated save() method:**
 ```typescript
 async save(entity: T): Promise<void> {
  const entityId = (entity as any).id;
  // Check if entity exists to determine create vs update
  const existingEntity = await this.get(entityId);
  const isCreate = existingEntity === null;
  // Calculate changes: full entity for create, diff for update
  let changes: any;
  if (isCreate) {
    changes = entity;
  } else {
    const existingSerialized = this.serialize(existingEntity);
    const newSerialized = this.serialize(entity);
    changes = diff(existingSerialized, newSerialized);
  }
  // ... save to IndexedDB ...
  // Emit ENTITY_SAVED event
  const payload: IEntitySavedPayload = {
    entityType: this.entityType,
    entityId,
    operation: isCreate ? 'create' : 'update',
    changes,
    timestamp: Date.now()
  };
  this.eventBus.emit(CoreEvents.ENTITY_SAVED, payload);
 }
 ```
 ---
 ### Phase 6: AuditService - Override Pattern ✅
 **Key Design Decision:** AuditService overrides save() to NOT emit ENTITY_SAVED.
 **Why:** If AuditService.save() emitted ENTITY_SAVED, it would trigger AuditService again → infinite loop.
 **Created src/storage/audit/AuditService.ts:**
 ```typescript
 export class AuditService extends BaseEntityService<IAuditEntry> {
  readonly storeName = 'audit';
  readonly entityType: EntityType = 'Audit';
  constructor(context: IndexedDBContext, eventBus: IEventBus) {
    super(context, eventBus);
    this.setupEventListeners();
  }
  private setupEventListeners(): void {
    this.eventBus.on(CoreEvents.ENTITY_SAVED, (event: Event) => {
      const detail = (event as CustomEvent).detail;
      this.handleEntitySaved(detail);
    });
    this.eventBus.on(CoreEvents.ENTITY_DELETED, (event: Event) => {
      const detail = (event as CustomEvent).detail;
      this.handleEntityDeleted(detail);
    });
  }
  // Override save to emit AUDIT_LOGGED instead of ENTITY_SAVED
  async save(entity: IAuditEntry): Promise<void> {
    // ... save to IndexedDB ...
    // Emit AUDIT_LOGGED (not ENTITY_SAVED)
    const payload: IAuditLoggedPayload = {
      auditId: entity.id,
      entityType: entity.entityType,
      entityId: entity.entityId,
      operation: entity.operation,
      timestamp: entity.timestamp
    };
    this.eventBus.emit(CoreEvents.AUDIT_LOGGED, payload);
  }
  // Audit entries cannot be deleted (compliance)
  async delete(_id: string): Promise<void> {
    throw new Error('Audit entries cannot be deleted (compliance requirement)');
  }
 }
 ```
 ---
 ### Phase 7: SyncManager Refactoring ✅
 **Before:** Used OperationQueue (never populated)
 **After:** Listens to AUDIT_LOGGED, syncs audit entries
 **Key Changes:**
 ```typescript
 export class SyncManager {
  constructor(
    eventBus: IEventBus,
    auditService: AuditService,
    auditApiRepository: IApiRepository<IAuditEntry>
  ) {
    this.setupAuditListener();
    this.startSync();
  }
  private setupAuditListener(): void {
    this.eventBus.on(CoreEvents.AUDIT_LOGGED, () => {
      if (this.isOnline && !this.isSyncing) {
        this.processPendingAudits();
      }
    });
  }
  private async processPendingAudits(): Promise<void> {
    const pendingAudits = await this.auditService.getPendingAudits();
    for (const audit of pendingAudits) {
      await this.auditApiRepository.sendCreate(audit);
      await this.auditService.markAsSynced(audit.id);
    }
  }
 }
 ```
 ---
 ### Phase 8: EventBus Injection Problem 🐛
 **Discovery:** Entity services had no EventBus!
 **User Observation:** *"There's no EventBus being passed. Why are you using super(...arguments) when there's an empty constructor?"*
 **Problem:**
 - BaseEntityService had optional eventBus parameter
 - Entity services (EventService, BookingService, etc.) had no constructors
 - EventBus was never passed → events never emitted
 **User Directive:** *"Remove the null check you added in BaseEntityService - it doesn't make sense."*
 **Fix:**
 1. Made eventBus required in BaseEntityService
 2. Added constructors to all entity services:
 ```typescript
 // EventService.ts
 constructor(context: IndexedDBContext, eventBus: IEventBus) {
  super(context, eventBus);
 }
 // BookingService.ts, CustomerService.ts, ResourceService.ts - same pattern
 ```
 ---
 ### Phase 9: Typed Payload Interfaces ✅
 **User Observation:** *"The events you've created use anonymous types. I'd prefer typed interfaces following the existing Payload suffix convention."*
 **Added to src/types/EventTypes.ts:**
 ```typescript
 export interface IEntitySavedPayload {
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update';
  changes: any;
  timestamp: number;
 }
 export interface IEntityDeletedPayload {
  entityType: EntityType;
  entityId: string;
  operation: 'delete';
  timestamp: number;
 }
 export interface IAuditLoggedPayload {
  auditId: string;
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update' | 'delete';
  timestamp: number;
 }
 ```
 ---
 ## Files Changed Summary
 ### Files Created (4)
 1. **src/types/AuditTypes.ts** - IAuditEntry interface
 2. **src/storage/audit/AuditStore.ts** - IndexedDB store for audit entries
 3. **src/storage/audit/AuditService.ts** - Event-driven audit service
 4. **src/repositories/MockAuditRepository.ts** - Mock API for audit sync
 ### Files Deleted (1)
 5. **src/storage/OperationQueue.ts** - Replaced by audit-based approach
 ### Files Modified (9)
 6. **src/types/CalendarTypes.ts** - Added 'Audit' to EntityType
 7. **src/constants/CoreEvents.ts** - Added ENTITY_SAVED, ENTITY_DELETED, AUDIT_LOGGED
 8. **src/types/EventTypes.ts** - Added IEntitySavedPayload, IEntityDeletedPayload, IAuditLoggedPayload
 9. **src/storage/BaseEntityService.ts** - Added diff calculation, event emission, required eventBus
 10. **src/storage/events/EventService.ts** - Added constructor with eventBus
 11. **src/storage/bookings/BookingService.ts** - Added constructor with eventBus
 12. **src/storage/customers/CustomerService.ts** - Added constructor with eventBus
 13. **src/storage/resources/ResourceService.ts** - Added constructor with eventBus
 14. **src/workers/SyncManager.ts** - Refactored to use AuditService
 15. **src/storage/IndexedDBContext.ts** - Bumped DB version to 3
 16. **src/index.ts** - Updated DI registrations
 ---
 ## Architecture Evolution Diagram
 **BEFORE (Disconnected):**
 ```
 EventManager
    ↓
 EventService.save()
    ↓
 IndexedDB (saved)
 [OperationQueue - never filled]
 [SyncManager - commented out]
 ```
 **AFTER (Event-Driven):**
 ```
 EventManager
    ↓
 EventService.save()
    ↓
 BaseEntityService.save()
    ├── IndexedDB (saved)
    └── emit ENTITY_SAVED (with diff)
            ↓
        AuditService listens
            ├── Creates audit entry
            └── emit AUDIT_LOGGED
                    ↓
                SyncManager listens
                    └── Syncs to backend
 ```
 ---
 ## Key Design Decisions
 ### 1. Audit-Based Instead of Queue-Based
 **Why:** Audit serves dual purpose - compliance trail AND sync source.
 **Benefit:** Single source of truth for all changes.
 ### 2. Chained Events (ENTITY_SAVED → AUDIT_LOGGED)
 **Why:** Prevents race condition where SyncManager runs before audit is saved.
 **Benefit:** Guaranteed order of operations.
 ### 3. JSON Diff for Changes
 **Why:** Only store what changed, not full entity.
 **Benefit:** Smaller audit entries, easier to see what changed.
 ### 4. Override Pattern for AuditService
 **Why:** Prevent infinite loop (audit → event → audit → event...).
 **Benefit:** Clean separation without special flags.
 ### 5. Required EventBus (Not Optional)
 **Why:** Events are core to architecture, not optional.
 **Benefit:** No null checks, guaranteed behavior.
 ---
 ## Discussion Topics (Not Implemented)
 ### IndexedDB for Logging/Traces
 User observation: *"This IndexedDB approach is quite interesting - we could extend it to handle logging, traces, and exceptions as well."*
 **Potential Extension:**
 - LogStore - Application logs
 - TraceStore - Performance traces
 - ExceptionStore - Caught/uncaught errors
 **Console Interception Pattern:**
 ```typescript
 const originalLog = console.log;
 console.log = (...args) => {
  logService.save({ level: 'info', message: args, timestamp: Date.now() });
  if (isDevelopment) originalLog.apply(console, args);
 };
 ```
 **Cleanup Strategy:**
 - 7-day retention
 - Or max 10,000 entries with FIFO
 **Decision:** Not implemented this session, but architecture supports it.
 ---
 ## Lessons Learned
 ### 1. Verify Existing Code Actually Works
 The sync infrastructure looked complete but was completely disconnected.
 **Lesson:** Don't assume existing code works - trace the actual flow.
 ### 2. Audit Trail Serves Multiple Purposes
 Audit is not just for compliance - it's also the perfect sync source.
 **Lesson:** Look for dual-purpose designs.
 ### 3. Event Ordering Matters
 Race conditions between listeners are real.
 **Lesson:** Use chained events when order matters.
 ### 4. Optional Dependencies Create Hidden Bugs
 Optional eventBus meant events silently didn't fire.
 **Lesson:** Make core dependencies required.
 ### 5. Type Consistency Matters
 Anonymous types in events vs Payload interfaces elsewhere.
 **Lesson:** Follow existing patterns in codebase.
 ---
 ## Current State & Next Steps
 ### ✅ Build Status: Successful
 ```
 [NovaDI] Performance Summary:
  - Files in TypeScript Program: 80
  - Files actually transformed: 58
  - Total: 1467.93ms
 ```
 ### ✅ Architecture State
 - **Event-driven audit trail:** Complete
 - **AuditService:** Listens for entity events, creates audit entries
 - **SyncManager:** Listens for AUDIT_LOGGED, syncs to backend
 - **BaseEntityService:** Emits events on save/delete with JSON diff
 ### ⚠️ Not Yet Tested
 - Runtime behavior (does AuditService receive events?)
 - Diff calculation accuracy
 - SyncManager sync flow
 - IndexedDB version upgrade (v2 → v3)
 ### 📋 Next Steps
 **Immediate:**
 1. Test in browser - verify events fire correctly
 2. Check IndexedDB for audit entries after save
 3. Verify SyncManager logs sync attempts
 **Future:**
 1. Real backend API for audit sync
 2. Logging/traces extension (console interception)
 3. Cleanup strategy for old audit entries
 ---
 ## Conclusion
 **Initial Goal:** Understand sync logic
 **Actual Work:** Complete architecture redesign
 **What We Found:**
 - Sync infrastructure was dead code
 - OperationQueue never populated
 - SyncManager commented out
 - No events being emitted
 **What We Built:**
 - Audit-based sync architecture
 - Event-driven design with chained events
 - JSON diff for change tracking
 - Typed payload interfaces
 **Key Insight:** Sometimes "understanding existing code" reveals there's nothing to understand - just scaffolding that needs to be replaced with actual implementation.
 ---
 **Session Complete:** 2025-11-22
 **Documentation Quality:** High
 **Ready for:** Runtime testing