Development journey and operational experience

This account follows what the project planned, implemented and observed—including corrections and limits recorded along the way. The operational results below are historical retained reports for named exercises, not fresh execution, current service telemetry or a claim of complete production acceptance. For the mechanisms themselves, see Evidence Lifecycle and Engineering decisions and tradeoffs.

Evaluation summary

Each exercise below asked one bounded question. For each one: the design, what would have counted against it, what was recorded and what the result was allowed to change. The narrative sections that follow give the detail and the limits.

Early synthetic acceptance

• Question: Does local evidence survive interruption, retry and contention?
• Design: 10 scripted scenarios, including SIGKILL barriers and a corruption stop
• Would have counted against it: Any scenario ending in a false success or a lost record
• Recorded result: 10 of 10 PASS
• Consequence: Scoped to synthetic media; not physical-camera or production durability

SPEC-010 evidence package

• Question: Is the acceptance evidence itself clean?
• Design: Inventory review of the retained package
• Would have counted against it: Host artefacts or unaccounted files in the inventory
• Recorded result: Rejected (.DS_Store present), then accepted after a layout guard
• Consequence: Passing tests and a publishable receipt treated as separate gates

OPBD-1, two real sources

• Question: Can two physical sources keep separate capture occasions while processing shares the spool?
• Design: 35 paired intervals; iPad and Surface/MediaMTX; warm QNAP replica
• Would have counted against it: Missing or lingering PREPARED captures; unverified replicas; corruption
• Recorded result: All captures COMPLETE; replica fully verified; processing exited on lock contention, then caught up
• Consequence: Evidence for a distinct processing lifecycle

OPBD-2, 7,200 s

• Question: Does accounting stay truthful over a longer two-source run?
• Design: Append-only ledger and terminal receipt; deep checkpoints
• Would have counted against it: Hidden source loss; a breached attempt relabelled as a pass
• Recorded result: First attempt: harness BREACH (240 s timeout vs ~270 s measured). Corrected run: OPBD2_2H_LIMITATION, 75 iPad source gaps kept
• Consequence: Harness budget 600 s; no eight-hour tier authorised

OPBD-3 Drill A

• Question: Can capture continue and Replication catch up when its worker is absent?
• Design: Worker withheld past the lag threshold; one catch-up run
• Would have counted against it: Lost local authority; catch-up needing a new queue; unverified objects
• Recorded result: Lag BREACH above 60 s, then one catch-up, 12 of 12 verified, full readback
• Consequence: Supports Replication as a separate worker

OPBD-4

• Question: Which worker deserves its own lifecycle?
• Design: Decision record over OPBD-1 to OPBD-3 and the notebook
• Would have counted against it: — (a decision, not a test)
• Recorded result: Extract Processing; keep Replication; defer supervisor and operator API
• Consequence: Processor extracted; broader orchestration not built

Real GCS, Phase B

• Question: Does Replication refuse to overwrite on a real provider?
• Design: One approved attempt against a frozen manifest on a disposable bucket
• Would have counted against it: Overwriting an existing object; inventory changing outside the authorised record
• Recorded result: Exact copy found, adopted and read back; no retry; inventory unchanged
• Consequence: Adoption path shown on a real provider; create path still untested there

Public Showcase CI

• Question: Can a reviewer reproduce the cross-component handoff from public packages?
• Design: Clean CI from published Processor and Replication packages
• Would have counted against it: CI passing only by changing worker code
• Recorded result: Several packaging and setup failures, each fixed without touching the workers
• Consequence: Reviewer journey tested; not a running pipeline

How to read the result labels

• PASS means the exercise’s stated question was answered positively within its scope.
• LIMITATION means the system behaved and accounted correctly, but the evidence shows a property of the inputs or environment, such as source gaps, that the result must carry forward.
• BREACH means a stated rule or threshold was exceeded. A run-level breach, like OPBD-2’s first attempt, ends that attempt: it is never resumed or relabelled, and a corrected attempt starts on fresh authority. A threshold breach can also be the condition a drill sets out to create, like OPBD-3’s replication lag above 60 seconds.
• Rejected / accepted applies to an evidence package, not to the software it describes.

Threats to validity

These limits apply to every exercise above. They are why the results are stated as narrowly as they are.

• One run each. Every exercise is a single accepted run or drill. None gives a rate, a variance or a failure probability. The ~270 s reconciliation time is one benchmark on a scratch copy.
• No comparison condition. There is no baseline system or alternative design run under the same conditions. “The lock conflict justified extraction” is a design argument from an observation, not a measured improvement.
• Lab conditions. Physical exercises used two source classes on one network with one replica target. They do not cover other camera profiles, network contention attributable per source, or provider outages.
• Mostly self-assessed. The project owner ran exercises such as OPBD-2. Independent review is recorded for some closeouts, such as OPBD-3 Wave 1A and the SPEC-025 integrated demonstration, but not for all of them.
• Retained, not fresh. These are historical records. None of them is a statement about current uptime or a current deployment.
• Synthetic where stated. Early acceptance, the Showcase and both hosted demonstrations use synthetic media or records.

What the table does support is the direction of each decision: where an exercise could have failed, the record shows what happened, and the project changed only what that observation justified.

The problem came before the separate platform

Author recollection (owner-supplied; retrospective, not a contemporaneous engineering record). My recollection is that Edge Evidence grew out of the drone project. I wanted a reliable system for transferring and handling its video—the preservation, processing, recovery, and verification responsibilities that became Edge Evidence. The reliability foundation developed into a separate engineering project, while drone detection remained a distinct workload.

The early project charter and accepted Stage 0 decisions support the planned separation: build a reliability foundation for media from independently failing sources; keep drone detection, calibration and fusion in their own research/workload project; integrate later through defined processor or field-adapter interfaces. The documents do not independently establish the owner’s personal origin story or its precise sequence. Nor do they establish that a completed drone detector ran end-to-end through the eventual platform.

The Stage 0 decision was notably modest. Instead of starting with a collection of services, the platform would begin as a modular monolith with explicit seams: deterministic source, one edge-agent process, SQLite, local spool and administrative checks. Additional workers, NAS replication, broker delivery and cloud runtime belonged to later gates. The charter framed this as avoiding distributed-systems cost before a failure, state, lifecycle or scaling boundary justified it. Those are recorded project decisions. It would be wrong to retroactively claim that every later worker was planned as a separately deployed production service from day one.

First prove that a recording can survive an interruption

The initial specifications progressed in deliberately narrow slices: repository/contracts without captured media; a deterministic FFmpeg source; media finalization; content and capture-occurrence registration; then crash and restart reconciliation. This sequence made the first test surface stable before adding network cameras, multiple producers or storage targets.

One retained early synthetic acceptance report lists 10 of 10 scenarios PASS, including normal completion, exact retry, reconciliation, gap recovery, SIGKILL barriers, contention and corruption stop. That is an earlier synthetic execution record, not evidence of current production durability or physical-camera compatibility. A separate RTSP acceptance record adds a controlled loopback MediaMTX publisher and provenance; it explicitly retains the same synthetic-only limitations.

The surrounding build log is more revealing than a green badge by itself. On August 5, the SPEC-010 implementation and scenarios were considered technically acceptable, but the retained acceptance evidence was rejected because host-generated .DS_Store had entered the inventory. The subsequent correction added a portable evidence-layout guard; the August 6 record then accepted the sanitized evidence and promoted the next multi-source scope. This is a documented observation → narrow correction → later acceptance chain. It was an evidence-package hygiene failure, not proof that the camera/recovery mechanism failed. It also shows why the project did not treat a passing test suite and a publishable receipt as equivalent.

The question that then became operational was less abstract: when two sources run at once, can their distinct capture occasions remain truthful while another part of the system inspects or processes the same spool?

Real-input work exposed an actual boundary

The OPBD-1 real-input baseline used two bounded physical source classes, an iPad feed and a Surface/MediaMTX path, with sanitized diagnostics and retained authority. The accepted report records 35 paired intervals, 70 intended capture opportunities, all terminal as COMPLETE with zero missing and zero remaining PREPARED. Local verification passed. The local outbox remained deliberately undistributed: 70 committed/pending events, zero publication attempts. A warm QNAP target reached 214 of 214 eligible objects VERIFIED, with full destination readback at named checkpoints and no pending remainder.

Those numbers are a single accepted lab baseline, not a supported long-running fleet rate. The record explicitly prevents inventing an earlier timestamped run ledger—OPBD-1 predated that format. It also does not attribute aggregate shared-Wi-Fi traffic to individual sources, certify every camera profile, or treat the replica as original evidence authority.

The interesting event was an expected lock conflict. Concurrent capture held shared spool access; processing still required exclusive access. The exercise physically observed the processing operation exit with its documented lock-contention code while the captures completed. Running the processing operation again after capture yielded a safe result, and later exact reruns remained idempotent. This was not a corruption incident. It was a repeatable signal that processing and capture wanted different resource lifecycles.

The August 10 notebook normalized this retained baseline for a walkthrough and later comparison. It distinguished measured observations, reconstruction from accepted counts, and architectural interpretation. Its later isolated execution record reports that the notebook ran on sanitized OPBD-1 and OPBD-2 datasets without notebook errors. That made evidence easier to read, not more authoritative: the notebook was explicitly forbidden from acting as a second database, a scheduler or a live monitoring service.

A longer run preserved a limitation rather than hiding it

OPBD-2 attempted a controlled 7,200-second two-source scheduling tier with append-only diagnostic ledger and terminal receipt. Its historical trail includes a real run-harness problem, then a bounded correction—not merely a cleaned-up final chart.

The v1.3 attempt stopped before a 90-minute checkpoint could finish. The canonical reconcile command reached the harness’s 240-second control timeout, after which timeout cleanup tried to use an unimported signal name and raised NameError. An undrained subprocess-output pipe was also confirmed as a helper defect, but the record does not prove that it caused the interruption. Review classified the attempt as a run/harness-level BREACH, while the partially retained SQLite, spool, ledger and QNAP evidence still passed their specified integrity checks. It was not a terminal two-hour PASS and was never silently resumed.

A scratch-copy benchmark then observed canonical reconciliation finishing in about 270 seconds, beyond the existing timeout. The documented v1.4 correction raised that deep-checkpoint budget to 600 seconds, imported signal, replaced the vulnerable monitored pipes with local temporary files, and added bounded cleanup/output regressions. The corrected attempt started on fresh authority rather than converting the breached partial run into a success.

The final owner-run OPBD-2 result was OPBD2_2H_LIMITATION with evidence integrity preserved, not OPBD2_2H_PASS. The retained adjudication reports all 222 intended occurrences terminal, no missing or lingering PREPARED, QNAP 445/445 VERIFIED and full readback, together with 75 honest iPad source_gap_observed records and an aggregate truthful_gaps_present finding. There were zero integrity breach findings. The source gaps matter: reliable accounting preserved loss information; it did not make the underlying stream lossless. The limitation did not authorize an eight-hour endurance tier.

This is a particularly useful operational lesson because it contains two different problems and two different responses. The harness’s observed failure prompted a recorded harness correction. The final source gaps were retained as limitations, not “repaired” out of the evidence by changing the accounting rules. None of those events, by itself, proves that the platform caused or could prevent the source gaps.

A missing worker became a test of independence

OPBD-3 deliberately exercised existing boundaries, not a newly invented control plane. One drill left Replication absent while capture continued, then required catch-up and independent verification. Another interrupted a chosen capture after durable intent and asked whether its sibling could finish and reconciliation could classify the interrupted occurrence truthfully.

The first wave did not close cleanly: the Replication drill lacked satisfactory sanitized QNAP-lag breach/readback evidence. The correction was scoped to Drill A only; the capture-interruption Drill B was not repeated because its intended behavior had already been established. The independently reviewed Wave 1A closeout records a QNAP lag BREACH above 60 seconds, then one catch-up invocation and 12/12 VERIFIED, zero pending/missing, a passing final lag check and full readback. The privacy review mattered too: the raw diagnostic command included private target identity, so the revised retained evidence stored only sanitized status, value and counts. Here the improvement was in observation and redaction, not an invented replication-repair algorithm.

The lesson was bounded: the absence of a worker did not erase local capture authority, and it was possible to catch up without a new queue. It did not prove continuous unattended replication, arbitrary provider outages or every future network-failure mode.

Let operation decide which process deserves its own lifecycle

OPBD-4 explicitly answered the extraction question using OPBD-1 through OPBD-3 and the verified notebook. The recorded decision was to extract Processing as an independent bounded processor-worker, retain Replication as an already separate worker, keep source capture as bounded invocations, leave event distribution optional, and defer both a generic source supervisor and an operator API. The reason for Processor extraction is unusually well supported: repeated physical capture/processing lock contention plus separately durable processing jobs and successful delayed catch-up demonstrated a distinct failure/resource lifecycle. The decision does not claim that horizontal scale, a cloud worker fleet or a broker had been measured and required.

Subsequent work assembled a SPEC-025 integrated foundation demonstration and operational-truth snapshot using the existing platform CLI and authority, with backlog/catch-up and readback. Its closure records scoped local, physical NFS/systemd and external GCS operational legs with independently reviewed retained archives. That is a platform-specific integrated proof, not the later public Showcase or an always-on deployed pipeline. It also does not turn the OPBD-4 decision to defer a product operator API into a contradiction: a bounded read-only operational-truth snapshot is not a standing service authority.

Wave C then set a different objective: export Processor and Replication as individually installable, deterministic, public-facing products, generated from the canonical private implementation, with separately provable tests and scopes. The acceptance contract made their independent demonstrations mandatory; optional read-only model adapters could not hold the two workers’ release hostage. Public projection and green CI established those generated candidate versions and tests, not new camera/source acceptance or private-provider operation.

A related operational proof used a disposable real GCS destination under separate approvals. Phase A took read-only inventory and froze one exact candidate, without a worker or provider mutation. Phase B was authorized against that exact manifest: it found the target object already matched 522 bytes and the expected digest, adopted rather than overwrote it, independently read it back, and changed only the authorized SQLite replica state. The retained records report one approved attempt, no retry, and unchanged destination inventory. This is stronger than a simulated object-store test but much narrower than production provider coverage. It does not demonstrate the create path on this real provider, arbitrary bucket policy or continuous replication.

The public Showcase taught a different kind of lesson

The public Showcase assembled a synthetic Processor-report → Replication handoff from separately packaged components. Its purpose was to give reviewers a reproducible cross-component example while preserving its independence from the original capture database.

The first public CI effort exposed practical packaging failures despite working component logic. The correction record names missing FFmpeg and optional test dependency setup; a bind-mounted output directory incompatible with atomic replacement; the wrong shell/path context; --network none preventing even local synthetic TCP transport; a virtual-environment symlink resolving to the wrong interpreter; and an obsolete .json URI where the worker required a digest-addressed .source URI. These findings prompted specific, bounded CI/setup/handoff corrections. The record expressly prohibited changing Processor or Replication worker implementations as a shortcut.

Then another test failure turned out not to be nondeterministic evidence at all. Two generated fingerprint JSON values matched, but GNU Make had echoed different output-root commands into the captured stdout. The corrective change used make --silent fingerprint; the retained public CI reconciliation reports both deterministic demos and the container lane passing, with one transient external Docker-image download error confined to a later retry. A further container smoke correction moved readiness/HTTP probing inside the isolated container after host-to-internal-network port publishing proved unreachable. These are clear observed problem → recorded correction → scoped CI result chains. They prove the value of testing the packaging and proof reader’s journey, not a continuously running capture-to-replica service.

The Visual Gateway remains a different demonstration. A dated October 13 public PR explicitly describes its single-process, same-origin Web UI + synthetic Evidence API + deterministic Edge Agent arrangement as an easier short introduction for first-time reviewers, with less service isolation than the three separately packaged services. Its inspector classifies supplied synthetic records; it does not start the Processor or Replication workers, replay a live capture or independently read a storage provider. The default-off ADK hook is not a completed Ollama evaluation. Nothing in either demonstration, or in a bounded accepted cloud proof, certifies current uptime without a present-tense observation.

What the experience justifies—and what it does not

The strongest supported developmental progression is not “we knew the entire architecture on day one.” It is a series of deliberately bounded questions:

Can local evidence survive an interruption? Early synthetic tests gave scoped answers. Can two real inputs preserve their separate occasions while a worker contends for access? The lab baseline exposed a measurable lock boundary. Can we observe longer operation without silently losing truth? The two-hour run ended with a limitation and a repaired harness, not an erased source gap. Can a missing worker catch up? The follow-up drill supported a separate Replication lifecycle. Which worker actually needs extraction? OPBD-4 explicitly named Processing, while deferring broader orchestration. Can people inspect the system without mistaking a projection for authority? The standalone public products, Showcase and Visual Gateway answer only their own narrower questions.

That development account should not be stretched into an uninterrupted triumph. The early tests were conditional or synthetic; physical exercises were lab-bound; the provider proof had one approved adoption candidate; public CI failures were real; accepted corrected CI applied to exact versions; and historical cloud or hosted-projection receipts are not current availability checks. The private operational notebook is a read-only analysis of accepted sanitized datasets, not a source of live authority.

The Evidence Lifecycle chapter describes component responsibilities in more detail; Engineering decisions and tradeoffs explains their benefits and costs. This account contributes the sequence of evidence and observations. Some originally personal priorities, alternate proposals and experiential turning points are not independently documented. They remain unclaimed; answering further recollection prompts is optional, not a prerequisite for this factual account.

Finally, the frozen Project 03 research is a different historical analytical estate. Its W0–W8 evidence packages, verification/acceptance distributions, provenance findings, projection runtime classes and author retrospective cannot be used to prove this platform’s operational success, quantify AI causation or establish why its architecture was chosen. The later reorganization of those notes explains presentation, not this project’s beginning. The shared lesson is methodological: a result deserves the narrowest truthful claim that its evidence can actually carry. That is a present-day connection, not a newly discovered causal history.