Calendar/coding-sessions/2025-11-22-audit-trail-event-driven-sync.md

Audit Trail & Event-Driven Sync Architecture

Date: 2025-11-22 Duration: ~4 hours Initial Scope: Understand existing sync logic Actual Scope: Complete audit-based sync architecture with event-driven design

---

Executive Summary

Discovered that existing sync infrastructure (SyncManager, SyncPlugin, OperationQueue) was completely disconnected - nothing was wired together. Redesigned from scratch using audit-based architecture where all entity changes are logged to an audit store, and SyncManager listens for AUDIT_LOGGED events to sync to backend.

Key Achievements:

• ✅ Designed event-driven audit trail architecture
• ✅ Created AuditTypes, AuditStore, AuditService
• ✅ Updated BaseEntityService with JSON diff calculation (json-diff-ts)
• ✅ Implemented event emission on save/delete (ENTITY_SAVED, ENTITY_DELETED)
• ✅ Refactored SyncManager to listen for AUDIT_LOGGED events
• ✅ Fixed EventBus injection (required, not optional)
• ✅ Added typed Payload interfaces for all events

Critical Discovery: The "working" sync infrastructure was actually dead code - SyncManager was commented out, queue was never populated, and no events were being emitted.

---

Context: Starting Point

Previous Work (Nov 20, 2025)

Repository Layer Elimination session established:

• BaseEntityService with generic CRUD operations
• IndexedDBContext for database connection
• Direct service usage pattern (no repository wrapper)
• DataSeeder for initial data loading

The Gap

After repository elimination, we had:

• ✅ Services working (BaseEntityService + SyncPlugin)
• ✅ IndexedDB storing data
• ❌ SyncManager commented out
• ❌ OperationQueue never populated
• ❌ No events emitted on entity changes
• ❌ No audit trail for compliance

---

Session Evolution: Major Architectural Decisions

Phase 1: Discovery - Nothing Was Connected 🚨

User Question: "What synchronization logic do we have for the server database?"

Investigation Findings:

• SyncManager exists but was commented out in index.ts
• OperationQueue exists but never receives operations
• BaseEntityService.save() just saves - no events, no queue
• SyncPlugin provides sync status methods but nothing triggers them

The Truth: Entire sync infrastructure was scaffolding with no actual wiring.

---

Phase 2: Architecture Discussion - Queue vs Audit

User's Initial Mental Model:

IEntityService.save() → saves to IndexedDB → emits event
SyncManager listens → reads pending from IndexedDB → syncs to backend

Problem Identified: "Who fills the queue?"

Options Discussed:

1. Service writes to queue
2. EventManager writes to queue
3. SyncManager reads pending from IndexedDB

User Insight: "I need an audit trail for all changes."

Decision: Drop OperationQueue concept, use Audit store instead.

---

Phase 3: Audit Architecture Design ✅

Requirements:

• All entity changes must be logged (compliance)
• Changes should store JSON diff (not full entity)
• userId required (hardcoded GUID for now)
• Audit entries never deleted

Designed Event Chain:

Entity change
    → BaseEntityService.save()
    → emit ENTITY_SAVED (with diff)
    → AuditService listens
    → creates audit entry
    → emit AUDIT_LOGGED
    → SyncManager listens
    → syncs to backend

Why Chained Events? User caught race condition: "If both AuditService and SyncManager listen to ENTITY_SAVED, SyncManager could execute before the audit entry is created."

Solution: AuditService emits AUDIT_LOGGED after saving, SyncManager only listens to AUDIT_LOGGED.

---

Phase 4: Implementation - AuditTypes & AuditStore ✅

Created src/types/AuditTypes.ts:

export interface IAuditEntry extends ISync {
  id: string;
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update' | 'delete';
  userId: string;
  timestamp: number;
  changes: any;  // JSON diff result
  synced: boolean;
  syncStatus: 'synced' | 'pending' | 'error';
}

Created src/storage/audit/AuditStore.ts:

export class AuditStore implements IStore {
  readonly storeName = 'audit';

  create(db: IDBDatabase): void {
    const store = db.createObjectStore(this.storeName, { keyPath: 'id' });
    store.createIndex('syncStatus', 'syncStatus', { unique: false });
    store.createIndex('synced', 'synced', { unique: false });
    store.createIndex('entityId', 'entityId', { unique: false });
    store.createIndex('timestamp', 'timestamp', { unique: false });
  }
}

---

Phase 5: BaseEntityService - Diff Calculation & Event Emission ✅

Added json-diff-ts dependency:

npm install json-diff-ts

Updated save() method:

async save(entity: T): Promise<void> {
  const entityId = (entity as any).id;

  // Check if entity exists to determine create vs update
  const existingEntity = await this.get(entityId);
  const isCreate = existingEntity === null;

  // Calculate changes: full entity for create, diff for update
  let changes: any;
  if (isCreate) {
    changes = entity;
  } else {
    const existingSerialized = this.serialize(existingEntity);
    const newSerialized = this.serialize(entity);
    changes = diff(existingSerialized, newSerialized);
  }

  // ... save to IndexedDB ...

  // Emit ENTITY_SAVED event
  const payload: IEntitySavedPayload = {
    entityType: this.entityType,
    entityId,
    operation: isCreate ? 'create' : 'update',
    changes,
    timestamp: Date.now()
  };
  this.eventBus.emit(CoreEvents.ENTITY_SAVED, payload);
}

---

Phase 6: AuditService - Override Pattern ✅

Key Design Decision: AuditService overrides save() to NOT emit ENTITY_SAVED.

Why: If AuditService.save() emitted ENTITY_SAVED, it would trigger AuditService again → infinite loop.

Created src/storage/audit/AuditService.ts:

export class AuditService extends BaseEntityService<IAuditEntry> {
  readonly storeName = 'audit';
  readonly entityType: EntityType = 'Audit';

  constructor(context: IndexedDBContext, eventBus: IEventBus) {
    super(context, eventBus);
    this.setupEventListeners();
  }

  private setupEventListeners(): void {
    this.eventBus.on(CoreEvents.ENTITY_SAVED, (event: Event) => {
      const detail = (event as CustomEvent).detail;
      this.handleEntitySaved(detail);
    });

    this.eventBus.on(CoreEvents.ENTITY_DELETED, (event: Event) => {
      const detail = (event as CustomEvent).detail;
      this.handleEntityDeleted(detail);
    });
  }

  // Override save to emit AUDIT_LOGGED instead of ENTITY_SAVED
  async save(entity: IAuditEntry): Promise<void> {
    // ... save to IndexedDB ...

    // Emit AUDIT_LOGGED (not ENTITY_SAVED)
    const payload: IAuditLoggedPayload = {
      auditId: entity.id,
      entityType: entity.entityType,
      entityId: entity.entityId,
      operation: entity.operation,
      timestamp: entity.timestamp
    };
    this.eventBus.emit(CoreEvents.AUDIT_LOGGED, payload);
  }

  // Audit entries cannot be deleted (compliance)
  async delete(_id: string): Promise<void> {
    throw new Error('Audit entries cannot be deleted (compliance requirement)');
  }
}

---

Phase 7: SyncManager Refactoring ✅

Before: Used OperationQueue (never populated) After: Listens to AUDIT_LOGGED, syncs audit entries

Key Changes:

export class SyncManager {
  constructor(
    eventBus: IEventBus,
    auditService: AuditService,
    auditApiRepository: IApiRepository<IAuditEntry>
  ) {
    this.setupAuditListener();
    this.startSync();
  }

  private setupAuditListener(): void {
    this.eventBus.on(CoreEvents.AUDIT_LOGGED, () => {
      if (this.isOnline && !this.isSyncing) {
        this.processPendingAudits();
      }
    });
  }

  private async processPendingAudits(): Promise<void> {
    const pendingAudits = await this.auditService.getPendingAudits();
    for (const audit of pendingAudits) {
      await this.auditApiRepository.sendCreate(audit);
      await this.auditService.markAsSynced(audit.id);
    }
  }
}

---

Phase 8: EventBus Injection Problem 🐛

Discovery: Entity services had no EventBus!

User Observation: "There's no EventBus being passed. Why are you using super(...arguments) when there's an empty constructor?"

Problem:

• BaseEntityService had optional eventBus parameter
• Entity services (EventService, BookingService, etc.) had no constructors
• EventBus was never passed → events never emitted

User Directive: "Remove the null check you added in BaseEntityService - it doesn't make sense."

Fix:

1. Made eventBus required in BaseEntityService
2. Added constructors to all entity services:

// EventService.ts
constructor(context: IndexedDBContext, eventBus: IEventBus) {
  super(context, eventBus);
}

// BookingService.ts, CustomerService.ts, ResourceService.ts - same pattern

---

Phase 9: Typed Payload Interfaces ✅

User Observation: "The events you've created use anonymous types. I'd prefer typed interfaces following the existing Payload suffix convention."

Added to src/types/EventTypes.ts:

export interface IEntitySavedPayload {
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update';
  changes: any;
  timestamp: number;
}

export interface IEntityDeletedPayload {
  entityType: EntityType;
  entityId: string;
  operation: 'delete';
  timestamp: number;
}

export interface IAuditLoggedPayload {
  auditId: string;
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update' | 'delete';
  timestamp: number;
}

---

Files Changed Summary

Files Created (4)

1. src/types/AuditTypes.ts - IAuditEntry interface
2. src/storage/audit/AuditStore.ts - IndexedDB store for audit entries
3. src/storage/audit/AuditService.ts - Event-driven audit service
4. src/repositories/MockAuditRepository.ts - Mock API for audit sync

Files Deleted (1)

5. src/storage/OperationQueue.ts - Replaced by audit-based approach

Files Modified (9)

6. src/types/CalendarTypes.ts - Added 'Audit' to EntityType
7. src/constants/CoreEvents.ts - Added ENTITY_SAVED, ENTITY_DELETED, AUDIT_LOGGED
8. src/types/EventTypes.ts - Added IEntitySavedPayload, IEntityDeletedPayload, IAuditLoggedPayload
9. src/storage/BaseEntityService.ts - Added diff calculation, event emission, required eventBus
10. src/storage/events/EventService.ts - Added constructor with eventBus
11. src/storage/bookings/BookingService.ts - Added constructor with eventBus
12. src/storage/customers/CustomerService.ts - Added constructor with eventBus
13. src/storage/resources/ResourceService.ts - Added constructor with eventBus
14. src/workers/SyncManager.ts - Refactored to use AuditService
15. src/storage/IndexedDBContext.ts - Bumped DB version to 3
16. src/index.ts - Updated DI registrations

---

Architecture Evolution Diagram

BEFORE (Disconnected):

EventManager
    ↓
EventService.save()
    ↓
IndexedDB (saved)

[OperationQueue - never filled]
[SyncManager - commented out]

AFTER (Event-Driven):

EventManager
    ↓
EventService.save()
    ↓
BaseEntityService.save()
    ├── IndexedDB (saved)
    └── emit ENTITY_SAVED (with diff)
            ↓
        AuditService listens
            ├── Creates audit entry
            └── emit AUDIT_LOGGED
                    ↓
                SyncManager listens
                    └── Syncs to backend

---

Key Design Decisions

1. Audit-Based Instead of Queue-Based

Why: Audit serves dual purpose - compliance trail AND sync source. Benefit: Single source of truth for all changes.

2. Chained Events (ENTITY_SAVED → AUDIT_LOGGED)

Why: Prevents race condition where SyncManager runs before audit is saved. Benefit: Guaranteed order of operations.

3. JSON Diff for Changes

Why: Only store what changed, not full entity. Benefit: Smaller audit entries, easier to see what changed.

4. Override Pattern for AuditService

Why: Prevent infinite loop (audit → event → audit → event...). Benefit: Clean separation without special flags.

5. Required EventBus (Not Optional)

Why: Events are core to architecture, not optional. Benefit: No null checks, guaranteed behavior.

---

Discussion Topics (Not Implemented)

IndexedDB for Logging/Traces

User observation: "This IndexedDB approach is quite interesting - we could extend it to handle logging, traces, and exceptions as well."

Potential Extension:

• LogStore - Application logs
• TraceStore - Performance traces
• ExceptionStore - Caught/uncaught errors

Console Interception Pattern:

const originalLog = console.log;
console.log = (...args) => {
  logService.save({ level: 'info', message: args, timestamp: Date.now() });
  if (isDevelopment) originalLog.apply(console, args);
};

Cleanup Strategy:

• 7-day retention
• Or max 10,000 entries with FIFO

Decision: Not implemented this session, but architecture supports it.

---

Lessons Learned

1. Verify Existing Code Actually Works

The sync infrastructure looked complete but was completely disconnected. Lesson: Don't assume existing code works - trace the actual flow.

2. Audit Trail Serves Multiple Purposes

Audit is not just for compliance - it's also the perfect sync source. Lesson: Look for dual-purpose designs.

3. Event Ordering Matters

Race conditions between listeners are real. Lesson: Use chained events when order matters.

4. Optional Dependencies Create Hidden Bugs

Optional eventBus meant events silently didn't fire. Lesson: Make core dependencies required.

5. Type Consistency Matters

Anonymous types in events vs Payload interfaces elsewhere. Lesson: Follow existing patterns in codebase.

---

Current State & Next Steps

✅ Build Status: Successful

[NovaDI] Performance Summary:
  - Files in TypeScript Program: 80
  - Files actually transformed: 58
  - Total: 1467.93ms

✅ Architecture State

• Event-driven audit trail: Complete
• AuditService: Listens for entity events, creates audit entries
• SyncManager: Listens for AUDIT_LOGGED, syncs to backend
• BaseEntityService: Emits events on save/delete with JSON diff

⚠️ Not Yet Tested

• Runtime behavior (does AuditService receive events?)
• Diff calculation accuracy
• SyncManager sync flow
• IndexedDB version upgrade (v2 → v3)

📋 Next Steps

Immediate:

1. Test in browser - verify events fire correctly
2. Check IndexedDB for audit entries after save
3. Verify SyncManager logs sync attempts

Future:

1. Real backend API for audit sync
2. Logging/traces extension (console interception)
3. Cleanup strategy for old audit entries

---

Conclusion

Initial Goal: Understand sync logic Actual Work: Complete architecture redesign

What We Found:

• Sync infrastructure was dead code
• OperationQueue never populated
• SyncManager commented out
• No events being emitted

What We Built:

• Audit-based sync architecture
• Event-driven design with chained events
• JSON diff for change tracking
• Typed payload interfaces

Key Insight: Sometimes "understanding existing code" reveals there's nothing to understand - just scaffolding that needs to be replaced with actual implementation.

---

Session Complete: 2025-11-22 Documentation Quality: High Ready for: Runtime testing