Calendar/coding-sessions/2025-11-22-audit-trail-event-driven-sync.md

# Audit Trail & Event-Driven Sync Architecture

**Date:** 2025-11-22
**Duration:** ~4 hours
**Initial Scope:** Understand existing sync logic
**Actual Scope:** Complete audit-based sync architecture with event-driven design

---

## Executive Summary

Discovered that existing sync infrastructure (SyncManager, SyncPlugin, OperationQueue) was completely disconnected - nothing was wired together. Redesigned from scratch using audit-based architecture where all entity changes are logged to an audit store, and SyncManager listens for AUDIT_LOGGED events to sync to backend.

**Key Achievements:**
- ✅ Designed event-driven audit trail architecture
- ✅ Created AuditTypes, AuditStore, AuditService
- ✅ Updated BaseEntityService with JSON diff calculation (json-diff-ts)
- ✅ Implemented event emission on save/delete (ENTITY_SAVED, ENTITY_DELETED)
- ✅ Refactored SyncManager to listen for AUDIT_LOGGED events
- ✅ Fixed EventBus injection (required, not optional)
- ✅ Added typed Payload interfaces for all events

**Critical Discovery:** The "working" sync infrastructure was actually dead code - SyncManager was commented out, queue was never populated, and no events were being emitted.

---

## Context: Starting Point

### Previous Work (Nov 20, 2025)
Repository Layer Elimination session established:
- BaseEntityService<T> with generic CRUD operations
- IndexedDBContext for database connection
- Direct service usage pattern (no repository wrapper)
- DataSeeder for initial data loading

### The Gap
After repository elimination, we had:
- ✅ Services working (BaseEntityService + SyncPlugin)
- ✅ IndexedDB storing data
- ❌ SyncManager commented out
- ❌ OperationQueue never populated
- ❌ No events emitted on entity changes
- ❌ No audit trail for compliance

---

## Session Evolution: Major Architectural Decisions

### Phase 1: Discovery - Nothing Was Connected 🚨

**User Question:** *"What synchronization logic do we have for the server database?"*

**Investigation Findings:**
- SyncManager exists but was commented out in index.ts
- OperationQueue exists but never receives operations
- BaseEntityService.save() just saves - no events, no queue
- SyncPlugin provides sync status methods but nothing triggers them

**The Truth:** Entire sync infrastructure was scaffolding with no actual wiring.

---

### Phase 2: Architecture Discussion - Queue vs Audit

**User's Initial Mental Model:**
```
IEntityService.save() → saves to IndexedDB → emits event
SyncManager listens → reads pending from IndexedDB → syncs to backend
```

**Problem Identified:** "Who fills the queue?"

**Options Discussed:**
1. Service writes to queue
2. EventManager writes to queue
3. SyncManager reads pending from IndexedDB

**User Insight:** *"I need an audit trail for all changes."*

**Decision:** Drop OperationQueue concept, use Audit store instead.

---

### Phase 3: Audit Architecture Design ✅

**Requirements:**
- All entity changes must be logged (compliance)
- Changes should store JSON diff (not full entity)
- userId required (hardcoded GUID for now)
- Audit entries never deleted

**Designed Event Chain:**
```
Entity change
    → BaseEntityService.save()
    → emit ENTITY_SAVED (with diff)
    → AuditService listens
    → creates audit entry
    → emit AUDIT_LOGGED
    → SyncManager listens
    → syncs to backend
```

**Why Chained Events?**
User caught race condition: *"If both AuditService and SyncManager listen to ENTITY_SAVED, SyncManager could execute before the audit entry is created."*

Solution: AuditService emits AUDIT_LOGGED after saving, SyncManager only listens to AUDIT_LOGGED.

---

### Phase 4: Implementation - AuditTypes & AuditStore ✅

**Created src/types/AuditTypes.ts:**
```typescript
export interface IAuditEntry extends ISync {
  id: string;
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update' | 'delete';
  userId: string;
  timestamp: number;
  changes: any;  // JSON diff result
  synced: boolean;
  syncStatus: 'synced' | 'pending' | 'error';
}
```

**Created src/storage/audit/AuditStore.ts:**
```typescript
export class AuditStore implements IStore {
  readonly storeName = 'audit';

  create(db: IDBDatabase): void {
    const store = db.createObjectStore(this.storeName, { keyPath: 'id' });
    store.createIndex('syncStatus', 'syncStatus', { unique: false });
    store.createIndex('synced', 'synced', { unique: false });
    store.createIndex('entityId', 'entityId', { unique: false });
    store.createIndex('timestamp', 'timestamp', { unique: false });
  }
}
```

---

### Phase 5: BaseEntityService - Diff Calculation & Event Emission ✅

**Added json-diff-ts dependency:**
```bash
npm install json-diff-ts
```

**Updated save() method:**
```typescript
async save(entity: T): Promise<void> {
  const entityId = (entity as any).id;

  // Check if entity exists to determine create vs update
  const existingEntity = await this.get(entityId);
  const isCreate = existingEntity === null;

  // Calculate changes: full entity for create, diff for update
  let changes: any;
  if (isCreate) {
    changes = entity;
  } else {
    const existingSerialized = this.serialize(existingEntity);
    const newSerialized = this.serialize(entity);
    changes = diff(existingSerialized, newSerialized);
  }

  // ... save to IndexedDB ...

  // Emit ENTITY_SAVED event
  const payload: IEntitySavedPayload = {
    entityType: this.entityType,
    entityId,
    operation: isCreate ? 'create' : 'update',
    changes,
    timestamp: Date.now()
  };
  this.eventBus.emit(CoreEvents.ENTITY_SAVED, payload);
}
```

---

### Phase 6: AuditService - Override Pattern ✅

**Key Design Decision:** AuditService overrides save() to NOT emit ENTITY_SAVED.

**Why:** If AuditService.save() emitted ENTITY_SAVED, it would trigger AuditService again → infinite loop.

**Created src/storage/audit/AuditService.ts:**
```typescript
export class AuditService extends BaseEntityService<IAuditEntry> {
  readonly storeName = 'audit';
  readonly entityType: EntityType = 'Audit';

  constructor(context: IndexedDBContext, eventBus: IEventBus) {
    super(context, eventBus);
    this.setupEventListeners();
  }

  private setupEventListeners(): void {
    this.eventBus.on(CoreEvents.ENTITY_SAVED, (event: Event) => {
      const detail = (event as CustomEvent).detail;
      this.handleEntitySaved(detail);
    });

    this.eventBus.on(CoreEvents.ENTITY_DELETED, (event: Event) => {
      const detail = (event as CustomEvent).detail;
      this.handleEntityDeleted(detail);
    });
  }

  // Override save to emit AUDIT_LOGGED instead of ENTITY_SAVED
  async save(entity: IAuditEntry): Promise<void> {
    // ... save to IndexedDB ...

    // Emit AUDIT_LOGGED (not ENTITY_SAVED)
    const payload: IAuditLoggedPayload = {
      auditId: entity.id,
      entityType: entity.entityType,
      entityId: entity.entityId,
      operation: entity.operation,
      timestamp: entity.timestamp
    };
    this.eventBus.emit(CoreEvents.AUDIT_LOGGED, payload);
  }

  // Audit entries cannot be deleted (compliance)
  async delete(_id: string): Promise<void> {
    throw new Error('Audit entries cannot be deleted (compliance requirement)');
  }
}
```

---

### Phase 7: SyncManager Refactoring ✅

**Before:** Used OperationQueue (never populated)
**After:** Listens to AUDIT_LOGGED, syncs audit entries

**Key Changes:**
```typescript
export class SyncManager {
  constructor(
    eventBus: IEventBus,
    auditService: AuditService,
    auditApiRepository: IApiRepository<IAuditEntry>
  ) {
    this.setupAuditListener();
    this.startSync();
  }

  private setupAuditListener(): void {
    this.eventBus.on(CoreEvents.AUDIT_LOGGED, () => {
      if (this.isOnline && !this.isSyncing) {
        this.processPendingAudits();
      }
    });
  }

  private async processPendingAudits(): Promise<void> {
    const pendingAudits = await this.auditService.getPendingAudits();
    for (const audit of pendingAudits) {
      await this.auditApiRepository.sendCreate(audit);
      await this.auditService.markAsSynced(audit.id);
    }
  }
}
```

---

### Phase 8: EventBus Injection Problem 🐛

**Discovery:** Entity services had no EventBus!

**User Observation:** *"There's no EventBus being passed. Why are you using super(...arguments) when there's an empty constructor?"*

**Problem:**
- BaseEntityService had optional eventBus parameter
- Entity services (EventService, BookingService, etc.) had no constructors
- EventBus was never passed → events never emitted

**User Directive:** *"Remove the null check you added in BaseEntityService - it doesn't make sense."*

**Fix:**
1. Made eventBus required in BaseEntityService
2. Added constructors to all entity services:

```typescript
// EventService.ts
constructor(context: IndexedDBContext, eventBus: IEventBus) {
  super(context, eventBus);
}

// BookingService.ts, CustomerService.ts, ResourceService.ts - same pattern
```

---

### Phase 9: Typed Payload Interfaces ✅

**User Observation:** *"The events you've created use anonymous types. I'd prefer typed interfaces following the existing Payload suffix convention."*

**Added to src/types/EventTypes.ts:**
```typescript
export interface IEntitySavedPayload {
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update';
  changes: any;
  timestamp: number;
}

export interface IEntityDeletedPayload {
  entityType: EntityType;
  entityId: string;
  operation: 'delete';
  timestamp: number;
}

export interface IAuditLoggedPayload {
  auditId: string;
  entityType: EntityType;
  entityId: string;
  operation: 'create' | 'update' | 'delete';
  timestamp: number;
}
```

---

## Files Changed Summary

### Files Created (4)
1. **src/types/AuditTypes.ts** - IAuditEntry interface
2. **src/storage/audit/AuditStore.ts** - IndexedDB store for audit entries
3. **src/storage/audit/AuditService.ts** - Event-driven audit service
4. **src/repositories/MockAuditRepository.ts** - Mock API for audit sync

### Files Deleted (1)
5. **src/storage/OperationQueue.ts** - Replaced by audit-based approach

### Files Modified (9)
6. **src/types/CalendarTypes.ts** - Added 'Audit' to EntityType
7. **src/constants/CoreEvents.ts** - Added ENTITY_SAVED, ENTITY_DELETED, AUDIT_LOGGED
8. **src/types/EventTypes.ts** - Added IEntitySavedPayload, IEntityDeletedPayload, IAuditLoggedPayload
9. **src/storage/BaseEntityService.ts** - Added diff calculation, event emission, required eventBus
10. **src/storage/events/EventService.ts** - Added constructor with eventBus
11. **src/storage/bookings/BookingService.ts** - Added constructor with eventBus
12. **src/storage/customers/CustomerService.ts** - Added constructor with eventBus
13. **src/storage/resources/ResourceService.ts** - Added constructor with eventBus
14. **src/workers/SyncManager.ts** - Refactored to use AuditService
15. **src/storage/IndexedDBContext.ts** - Bumped DB version to 3
16. **src/index.ts** - Updated DI registrations

---

## Architecture Evolution Diagram

**BEFORE (Disconnected):**
```
EventManager
    ↓
EventService.save()
    ↓
IndexedDB (saved)

[OperationQueue - never filled]
[SyncManager - commented out]
```

**AFTER (Event-Driven):**
```
EventManager
    ↓
EventService.save()
    ↓
BaseEntityService.save()
    ├── IndexedDB (saved)
    └── emit ENTITY_SAVED (with diff)
            ↓
        AuditService listens
            ├── Creates audit entry
            └── emit AUDIT_LOGGED
                    ↓
                SyncManager listens
                    └── Syncs to backend
```

---

## Key Design Decisions

### 1. Audit-Based Instead of Queue-Based
**Why:** Audit serves dual purpose - compliance trail AND sync source.
**Benefit:** Single source of truth for all changes.

### 2. Chained Events (ENTITY_SAVED → AUDIT_LOGGED)
**Why:** Prevents race condition where SyncManager runs before audit is saved.
**Benefit:** Guaranteed order of operations.

### 3. JSON Diff for Changes
**Why:** Only store what changed, not full entity.
**Benefit:** Smaller audit entries, easier to see what changed.

### 4. Override Pattern for AuditService
**Why:** Prevent infinite loop (audit → event → audit → event...).
**Benefit:** Clean separation without special flags.

### 5. Required EventBus (Not Optional)
**Why:** Events are core to architecture, not optional.
**Benefit:** No null checks, guaranteed behavior.

---

## Discussion Topics (Not Implemented)

### IndexedDB for Logging/Traces
User observation: *"This IndexedDB approach is quite interesting - we could extend it to handle logging, traces, and exceptions as well."*

**Potential Extension:**
- LogStore - Application logs
- TraceStore - Performance traces
- ExceptionStore - Caught/uncaught errors

**Console Interception Pattern:**
```typescript
const originalLog = console.log;
console.log = (...args) => {
  logService.save({ level: 'info', message: args, timestamp: Date.now() });
  if (isDevelopment) originalLog.apply(console, args);
};
```

**Cleanup Strategy:**
- 7-day retention
- Or max 10,000 entries with FIFO

**Decision:** Not implemented this session, but architecture supports it.

---

## Lessons Learned

### 1. Verify Existing Code Actually Works
The sync infrastructure looked complete but was completely disconnected.
**Lesson:** Don't assume existing code works - trace the actual flow.

### 2. Audit Trail Serves Multiple Purposes
Audit is not just for compliance - it's also the perfect sync source.
**Lesson:** Look for dual-purpose designs.

### 3. Event Ordering Matters
Race conditions between listeners are real.
**Lesson:** Use chained events when order matters.

### 4. Optional Dependencies Create Hidden Bugs
Optional eventBus meant events silently didn't fire.
**Lesson:** Make core dependencies required.

### 5. Type Consistency Matters
Anonymous types in events vs Payload interfaces elsewhere.
**Lesson:** Follow existing patterns in codebase.

---

## Current State & Next Steps

### ✅ Build Status: Successful
```
[NovaDI] Performance Summary:
  - Files in TypeScript Program: 80
  - Files actually transformed: 58
  - Total: 1467.93ms
```

### ✅ Architecture State
- **Event-driven audit trail:** Complete
- **AuditService:** Listens for entity events, creates audit entries
- **SyncManager:** Listens for AUDIT_LOGGED, syncs to backend
- **BaseEntityService:** Emits events on save/delete with JSON diff

### ⚠️ Not Yet Tested
- Runtime behavior (does AuditService receive events?)
- Diff calculation accuracy
- SyncManager sync flow
- IndexedDB version upgrade (v2 → v3)

### 📋 Next Steps

**Immediate:**
1. Test in browser - verify events fire correctly
2. Check IndexedDB for audit entries after save
3. Verify SyncManager logs sync attempts

**Future:**
1. Real backend API for audit sync
2. Logging/traces extension (console interception)
3. Cleanup strategy for old audit entries

---

## Conclusion

**Initial Goal:** Understand sync logic
**Actual Work:** Complete architecture redesign

**What We Found:**
- Sync infrastructure was dead code
- OperationQueue never populated
- SyncManager commented out
- No events being emitted

**What We Built:**
- Audit-based sync architecture
- Event-driven design with chained events
- JSON diff for change tracking
- Typed payload interfaces

**Key Insight:** Sometimes "understanding existing code" reveals there's nothing to understand - just scaffolding that needs to be replaced with actual implementation.

---

**Session Complete:** 2025-11-22
**Documentation Quality:** High
**Ready for:** Runtime testing