Event Pipeline Architecture

The Event Pipeline ensures reliable event delivery through a persistent, file-based queue with per-sink delivery tracking. Events are written to an NDJSON file and processed in batches. Each sink maintains independent retry state with exponential backoff, and permanently failed events are moved to a dead-letter queue for manual resolution.

The ProfileSafeEvent Envelope

Every event dispatched through the pipeline is wrapped in a canonical ProfileSafeEvent envelope:

Field	Description
EventId	Deterministic GUID derived from the audit sequence number (used for deduplication)
TimestampUtc	Event timestamp in UTC
SchemaVersion	Always 1.0.0
Severity	Info, Warning, Error, or Critical (see Audit Logging — Severity Mapping)
Source	GUI, CLI, or Core
OperationId	GUID correlating all events from a single export/import/simulation run
Category	Export, Import, Simulation, Security, Privacy, or System
Action	The AuditAction name (e.g. ExportCompleted)
Context	MachineName, UserName, Domain, Tenant
Payload	Action-specific data (target, success, duration, file count, etc.)
ErrorCode	APS error code on failure (e.g. APS-2103), null on success
Extensions	Open key-value map for custom metadata

For the full JSON schema, see Event Schema Reference.

Persistent Queue

Events are persisted to rotating NDJSON files in the EventPipeline folder:

File	Purpose
queue.jsonl	Active events (pending, partially delivered)
deadletter.jsonl	Permanently failed events awaiting manual resolution

The queue is cross-process safe via a Named Mutex and uses atomic write operations (temp file + move). Each queue entry stores the event payload (immutable) alongside per-sink delivery state (mutable).

Per-Sink Delivery Tracking

Each queue entry maintains an independent SinkDeliveryState for every active sink:

Field	Description
SinkId	Identifies the target sink (e.g. siem-http, webhook-primary)
Status	Pending, Delivered, FailedPermanent, or Skipped
IsCritical	Whether this sink affects the overall event delivery state
AttemptCount	Number of delivery attempts so far
NextAttemptUtc	Earliest time for the next retry (backoff-controlled)
LastError	Truncated error message from the last failed attempt (max 500 characters)
LastHttpStatus	HTTP status code from the last failed attempt (if applicable)

The overall event state is computed from critical sinks only: Delivered (all critical sinks succeeded or were skipped), PartiallyDelivered (at least one critical sink delivered), Pending (retries outstanding), or DeadLettered (all critical sinks permanently failed).

Failure Classification

When a sink delivery fails, the error is classified to determine whether the event should be retried:

Class	Behavior	Examples
Transient	Retry with exponential backoff	HTTP 429 (rate limit), HTTP 5xx (server error), timeout, DNS failure, connection refused
Permanent	Dead-letter immediately	HTTP 400/401/403/404/405/409/422, certificate errors, authentication failures

Unknown HTTP status codes default to transient (retry). Events also move to dead-letter when MaxRetries is reached (default: 10).

Exponential Backoff

Transient failures are retried with exponential backoff plus random jitter (±25%):

delay = BaseSeconds × 2^attemptCount  (capped at MaxSeconds)
jitter = delay × 0.25 × random(-1..+1)
nextAttempt = now + max(BaseSeconds, delay + jitter)

The backoff policy is configured in eventpipeline.xml:

<Backoff>
  <BaseSeconds>5</BaseSeconds>
  <MaxSeconds>600</MaxSeconds>
</Backoff>

With the defaults, retries start at ~5 seconds and cap at 10 minutes. After 10 attempts over approximately 20 minutes, the event is dead-lettered.

eventpipeline.xml Reference

<EventPipeline xmlns="urn:appprofilesafe:eventpipeline:v1" SchemaVersion="1.0.0">
  <QueueFolder>%ProgramData%\...\EventPipeline</QueueFolder>
  <MaxRetries>10</MaxRetries>
  <SendTimeoutSeconds>3</SendTimeoutSeconds>
  <Backoff>
    <BaseSeconds>5</BaseSeconds>
    <MaxSeconds>600</MaxSeconds>
  </Backoff>
  <Sinks>
    <Sink id="siem-http"       critical="true" />
    <Sink id="siem-syslog"     critical="false" />
    <Sink id="webhook-primary" critical="true" />
    <Sink id="eventlog"        critical="false" enabled="true"
          sourceName="AppProfileSafe" logName="Application" />
  </Sinks>
</EventPipeline>

Whether a sink is active is controlled by its own configuration file (<Active> in siem.xml, <Enabled> in webhook.xml). The <Sinks> section in eventpipeline.xml only declares routing metadata such as critical.

Queue Flushing

Queue processing is triggered in two ways:

• GUI: DispatchAndFlushAsync is called asynchronously (fire-and-forget) each time an audit entry is written.
• CLI: Events are dispatched synchronously to the persistent queue during operations. FlushQueue is called at the start and end of each CLI run to deliver all pending events to sinks. There is no background thread — all deliveries complete before the process exits.

Each flush processes up to FlushBatchSize (default: 100) due entries — only events where at least one sink has a pending delivery with NextAttemptUtc ≤ now.