Command Queue

Ingest, refresh, retag, and ingest-url all load the embedding model, scan Qdrant, and — for retag and background tagging — run the LLM tagger. Running two in parallel on a workstation is the fastest way to OOM the box. Imprint serializes them with a shared single-slot queue. When nothing waits, the slot runs background work (LLM tagging, then Organize when organize.background is on), which steps aside for every other job.

Semantics

• One job at a time, system-wide. A single advisory flock on data/queue.lock gates every heavy command, regardless of which side started it.
• UI queues; the CLI hands over or waits. POST /api/commands/{cmd} from the dashboard writes a row to data/queue.sqlite3 and returns a job_id immediately; the FastAPI dispatcher picks it up in FIFO order. Direct CLI invocations (imprint ingest, imprint refresh, imprint learn, imprint retag, imprint scrub, imprint ingest-url, imprint refresh-urls) try the lock non-blocking. If it is held they print the current holder, then enqueue themselves through the dashboard’s API when the dashboard runs, or wait for the lock otherwise. IMPRINT_QUEUE_NO_WAIT=1 makes them exit nonzero instead. Why the queue exists: ADR 0064.
• The Stop hook’s learner waits its turn. After each turn the Stop hook starts a detached learner (python -m imprint.session_learn <transcript>) and returns. The learner takes the conversation’s waiting slot in data/learn_slots/, or exits when another learner holds it, since that one has not read the conversation yet. It lets a burst of turns settle for 60 s, then waits for the lock without handing over to the dashboard (stopping background tagging, as any taker does), gives up its slot, and holds the lock as learn while it learns the exchanges, extracts decisions and, when enabled, summarizes. So a conversation has at most one learner waiting. Why: ADR 0077.
• Unapproved roots are refused before queueing. For ingest and refresh, POST /api/commands/{cmd} checks the body’s dir against the workspace’s roots with workspace_policy.unapproved_root, which mirrors ensure_ingest_paths including the first-root binding. Instead of queueing a job that would fail when it runs, it returns 409 {error, code: "root_not_approved", workspace, root, overlaps}; the dashboard asks to approve that root and resubmits (ADR 0015, ADR 0039).
• Identical waiting jobs merge. Submitting a job identical to one still waiting (same command, workspace and arguments) returns that job with merged: true instead of queueing another; the waiting job counts the requests it answers (_merged). One identical to the running job queues a single follow-up, since the running job may have read its inputs before the new request. Arguments are compared after normalization: dashboard fields and CLI arguments name the same job, option order does not matter, paths are compared without trailing separators or ./.. segments, and relative paths are compared as given. Browser captures never merge. The CLI makes its paths absolute before handing a job to the dashboard. Why: ADR 0068.
• Background tagging runs last and steps aside. With tagger.llm on, learn, ingest, refresh, ingest-url, the Stop hook and inline content store memories without running the LLM and mark them tag_pending. When no job runs or waits and no other process holds the lock, the dispatcher starts the background tagging job (tag-backlog, python -m imprint.tag_backlog <job_id>). It tags every workspace’s marked memories, newest first (in random order while a paired device also tags, ADR 0167), tagger.llm_concurrency at a time and one memory per write, and exits between groups when a job is queued (exit code 75, recorded as yielded). The dispatcher looks again when a job ends, when a terminal job releases the lock, when a tagger.* setting changes, on POST /api/queue/background/resume and every 30 s. After a cancel it waits until more memories are marked; after a failure (for example no usable tagger), 30 minutes. It keeps the latest 20 runs. Why: ADR 0067; tagging details in tagging.
• Or a paired device tags them. With tagger.llm off and tagger.peer on, a device paired with one that runs LLM tagging marks its new memories too, but starts no tagging job: sync takes what waits to that device, which tags it in its own background job, and the tags come back. The Queue page’s background card then says the memories wait for a paired device (ADR 0138).
• Then Organize, when it runs in the background. With organize.background on, the same job organizes each workspace that changed since its last pass, after tagging: the dispatcher also starts it for that alone. A workspace has changed when a signature read from payload indexes differs: catch-all sizes, the memories without a project, the known projects, the preference memories, the profiles due and organize.auto_move. The pass writes due profiles, analyses new or changed catch-all sources (moving strong ones, ADR 0133), reads new preference memories, and exits between them as yielded when a job is queued; a yielded pass is not recorded and the next one continues. An organize.* setting change wakes the dispatcher. Why: ADR 0132.
• Synced moves wait for the lock. A placement rule received from a paired device moves this device’s copies holding the lock; when a job holds it, an apply-placements job (python -m imprint.placement_sync) is queued for after it (ADR 0134).
• Any other lock taker stops background tagging. The background job holds the lock as a preemptible holder. A Go or Python process that wants the lock and finds a live preemptible holder stops it (SIGTERM, SIGKILL 3 s later; TerminateProcess with exit code 15 on Windows) and takes the lock within a few seconds, instead of refusing (update, dashboard and MCP purge, move), waiting (CLI purge, move, sync import) or failing (IMPRINT_QUEUE_NO_WAIT=1). The stopped run loses at most the memory it was classifying and is recorded as yielded.
• Cancel = kill the process group. Subprocesses are spawned with start_new_session=True, so killpg(pgid, SIGTERM) takes down the Python process, its httpx worker threads (in-flight LLM tagger calls die with it), any llama-cpp inference thread, and any descendant helpers. If the group is still alive 3 s later, the dispatcher escalates to SIGKILL.
• Persistent history. The SQLite queue survives API restarts. recover_on_startup() marks rows stuck in running whose PID is dead as failed (error='api_restart') and clears stale lock files. A background tagging run left by the previous server is stopped and recorded as yielded.

Storage

Path	Purpose
data/queue.sqlite3	FIFO queue + job history (see schema below)
data/queue.lock	fcntl advisory flock + JSON body describing the current holder
data/queue_logs/{job_id}.log	Full subprocess stdout+stderr per job (tailed by /stream); jobs run their Python unbuffered, so lines arrive as they are printed (ADR 0064)
data/progress.json	The lock holder’s progress: command, phase, processed of total and the unit it counts (files, conversations, chunks, memories, pages, steps). Python jobs, the Stop hook’s learner and Go setup write it; /api/queue joins it into the active job, the top bar shows it on every page, the desktop app reads it while the dashboard server is down, and a waiting command prints it (ADR 0089)
data/learn_slots/{hash}.lock	Stop hook learners’ waiting slots, one per conversation and workspace; removed when the waiting learner takes the queue lock
data/session_learn.log	Stop hook learners’ output; starts over past 1 MiB

Schema

associate-projects is a foreground maintenance job invoked from Projects or Commands. The dispatcher launches python -m imprint.project_associations, which takes the shared queue lock. It scans existing canonical snapshots for known project identifiers; optional model: true adds local section analysis with an installed helper. It does not move sources. Review suggestions after the job finishes; a cancelled scan retains completed source decisions and can be rerun. A source that exhausts the helper’s 120-second budget logs a warning, saves identifier matches and continues with the next source. It is not cached as a completed model analysis; rerunning retries it (ADR 0105). The running scan reports source counts through the shared progress record. See project association review.

CREATE TABLE jobs (
    id          TEXT PRIMARY KEY,   -- uuid4 hex
    command     TEXT NOT NULL,      -- ingest | refresh | retag | ingest-url | ...
    body_json   TEXT NOT NULL,      -- request body the UI/CLI submitted
    status      TEXT NOT NULL,      -- queued | running | done | failed | cancelled | yielded (tag-backlog only)
    pid         INTEGER,
    pgid        INTEGER,
    exit_code   INTEGER,
    error       TEXT,
    created_at  REAL NOT NULL,
    started_at  REAL,
    ended_at    REAL
);
-- Indexes: (status, created_at), (created_at DESC), and (ended_at DESC, id DESC) for paging history.
CREATE INDEX idx_jobs_status  ON jobs(status, created_at);
CREATE INDEX idx_jobs_created ON jobs(created_at DESC);

Lock file body

{"pid": 12345, "job_id": "<uuid>", "command": "ingest", "started_at": 1.713e9}

The background tagging job adds "preemptible": true. A holder clears the body before it unlocks, so a process that keeps running after a release (the dashboard after a purge) is not named as the holder.

Both sides parse the same shape: Python via imprint/queue_lock.py, Go via internal/queuelock/lock.go. On Windows, where locks are mandatory for reads, Python locks one byte at offset 2^30, past the body, so other processes can read who holds the lock; Go locks the whole file, so the two still exclude each other.

HTTP API

All endpoints live on the Imprint dashboard server (default http://127.0.0.1:8420).

Method	Path	Body	Response
POST	/api/commands/{command}	{...flags}	{job_id, position, merged} — position is 0 if it started immediately, otherwise its 1-based slot; merged is true when an identical job was already waiting and job_id is that job
GET	/api/queue?recent_limit=20	—	{active, queued[], recent[], background}; active is enriched with live progress fields (phase, processed, total, percent, eta_seconds). Background runs appear only in background: {enabled, tagging, organize: {enabled, due}, pending, state, job, resume_at}, where state is off, running, yielded, waiting, paused (after a cancel), failed (retried at resume_at) or idle, pending counts marked memories across workspaces, organize.due the workspaces changed since background Organize’s last pass, and job is the running or latest run. Every job also carries label, params and workspace (below)
POST	/api/queue/background/resume	—	The background object; starts background work again after a cancel or failure as soon as the queue is idle
GET	/api/queue/history?limit=25&before=&status=&bounds=	—	One page of finished jobs (done, failed, cancelled; not background runs), newest first: {jobs[], next, total, counts}. next is the cursor ("<ended_at>:<id>") to pass as before for the following page, null on the last; limit is at most 100. status keeps one outcome, total and counts included. bounds, comma-separated period starts in epoch seconds, newest first, adds counts: the jobs that ended in each period and, last, before them all; a start later than the one before it is an empty period. Each job is named like /api/queue’s and carries twin: {id, status} of an identical job (same dedupe_key) that waits or runs now, else null
GET	/api/jobs	—	Back-compat — {jobs: [active]} (empty array when nothing is running)
GET	/api/jobs/{id}	—	Full row with label, params and workspace, and progress when running
GET	/api/jobs/{id}/stream	—	SSE — replays the stored log then tails new lines; emits a final {type:"done", exit_code, status}
POST	/api/jobs/{id}/cancel	—	{ok, was_running}. Queued → marked cancelled; running → SIGTERM → SIGKILL (3 s)
POST	/api/jobs/{id}/retry	—	Queues a finished job again: the same command and body (without _merged) in the workspace it ran in, through submit, so an identical waiting job absorbs it (ADR 0068). {job_id, position, merged, retry_of} like POST /api/commands/{command}. 404 for an unknown job; 409 for one still waiting or running, for background work (which resumes by itself), or with code: root_not_approved when its folder is no longer an approved root; 400 when its workspace no longer exists
GET	/api/commands	—	{commands, clients}: the Commands page catalog and the CLI’s clients (see below); 502 when imprint clients fails

Allowed commands

imprint/commands.py holds the catalog the Commands page renders: each command’s label, description, group, fields (path, text, switch, project, client) and, for a command that removes something, the confirmation the page asks first. POST /api/commands/{command} accepts the catalog’s commands plus wipe, migrate and workspace, which the workspace dialogs queue, and wsl (setup or remove of one distribution’s client profile), which Settings → WSL profiles queues. queue.build_command_args maps each field to the CLI’s arguments; a test checks that every catalog field reaches them. A client field offers the clients imprint clients --json reports, from the CLI’s client table, with whether each is connected.

The same catalog names queued jobs: commands.describe gives each job in /api/queue and /api/jobs/{id} its command’s label, its arguments as params in the words of the command’s fields (a switch by its label alone, a value as {label, value}, a CLI-enqueued job’s verbatim args as one entry) and the workspace it was queued for. Jobs outside the catalog (tag-backlog, the workspace and WSL dialogs’ jobs) have fixed names there. The Queue page and the top bar show these names instead of command names and raw bodies.

Group	Commands
Index	learn (desktop), ingest, refresh, ingest-url, refresh-urls
Maintain	retag (project, all, dry_run, pending: only the memories waiting for background tagging), scrub (apply; a dry run without it, secret redaction), sync (export)
Clients	connect (Connect clients), disconnect (Disconnect clients, confirmed first), both with target: a client or all
Inspect	status, doctor (probe_mcp), config

Safe to run from the dashboard. A job runs under the dashboard server, from the runtime it serves, so the catalog leaves out commands that would change either one (ADR 0053):

• connect registers the installed runtime with clients and installs and starts nothing.
• disconnect only unregisters clients, and the dashboard and Qdrant keep running.
• setup, enable, bootstrap and profile install dependencies into the venv the server has loaded. Not offered. No setting needs them: model.device and model.gpu_device choose where llama.cpp runs and install nothing (ADR 0086), so the dashboard never restarts for a device change.

Setup and the queue. Setup takes the queue lock while it changes the venv, so it waits for a running job and stops background tagging, and holds data/runtime.lock, which the dashboard server’s launchers wait for before they start it. Before it replaces or removes installed packages, it restarts the dashboard server and, on Windows, stops every other process running from the runtime; MCP clients reconnect afterwards (ADR 0084).

• disable, ui, server and uninstall stop or remove the dashboard itself. Not offered.
• wsl setup and remove run from Settings → WSL profiles, where a distribution is chosen. They register the installed Windows runtime in a WSL profile and install nothing.
• Workspaces, purge, move, sync pairing and updates have their own dashboard flows.

Scheduled jobs. A reminder entry of kind job (ADR 0158) submits ingest, refresh, learn, organize, refresh-urls or retag through submit when it is due, in the workspace it was set in, so it merges with an identical waiting job and is refused outside the workspace’s approved roots like any other.

Non-interactive. Jobs have no terminal: the dispatcher gives each one the null device as standard input, so a prompt reads end-of-file instead of waiting. Commands that would ask (wipe) are sent with their confirmation flag by the dialog that asked the user.

UI

• /queue — Running: the running job, or background tagging while it runs, as one card (RunningJob): its name and arguments, phase, progress of total with percent, time so far and time left, Cancel, the newest output line and the whole output on request. Up next: waiting jobs in order (removing one is the row’s X; a merged job shows how many requests it answers). Background work: its state, the memories waiting for tagging, whether LLM tagging and Organize are on and how many workspaces changed, and Resume now after a cancel or failure. When nothing runs or waits, one line says the queue is idle. History: finished jobs, 25 a page (Newer / Older, by cursor), grouped by when they ended in the viewer’s time zone (Today, Yesterday, This week, Last week, This month, Last month, Older; lib/history-periods.ts) with each group’s count from the server, so a group split across pages still counts all its jobs. All / Failed filters it. Each row shows the result, the reason a failed job gave, run time and when it ended (a time today and yesterday, a weekday this and last week, then a date; the full date on hover). A failed row has Retry; a row whose identical job already waits or runs says Queued again or Running again instead, since a retry would only join it. History reloads whenever a job starts, ends or is queued. Any job opens a side sheet with a Retry (failed) or Run again (cancelled or done) action that then follows the new job, a Cancel or Remove action while it can run, the error, its arguments, workspace, times, process, and its output with Copy. Cancelling a running job asks first (Keep running / Cancel job), on every page that shows it; removing a waiting job does not. Job toasts name jobs by their label and a failure’s toast offers Retry and Open queue. Background runs raise no job toasts.
• Dashboard — IngestionProgress shows the same running-job card and how many jobs wait, linking to the queue. Add to memory queues a folder, a document, a web page or conversations; refreshing, retagging and the rest are on the Commands page.
• Top bar — ActivityIndicator shows the running job, or background tagging while it runs, on every page and links to the queue. It and the other queue views share one poll (lib/queue.ts); the offline banner shows the record the desktop app reads while the server is down (ADR 0089).
• QuickIngest / CommandsPage — both enqueue server-owned jobs. The Commands page follows the stored output; leaving it detaches the stream and keeps the job running in Queue. Its Cancel button explicitly cancels the job, including when submission is still pending. The page reloads the command catalog when a job finishes, so client connection states stay current. Legacy streamCommand() callers retain cancel-on-abort behavior unless they opt into detaching.

Verification

1. Queue gate (UI): start ingest via UI → start refresh while the first runs. The second enters the queued list and begins after the first finishes.
2. Queue gate (CLI): while a UI job is running, run imprint ingest <path> from a terminal. Expect it to name the running job, print Queued as <id> and exit 0, with the job in /queue. With IMPRINT_QUEUE_NO_WAIT=1, expect exit code 1 and an error printing the holder’s command, PID and start time and “Cancel it from the UI (/queue) or: kill ”.
3. Cancel running (cascade kill): start ingest with IMPRINT_LLM_TAGS=1 + anthropic provider; hit cancel in UI mid-LLM-call. Within ~3 s ps -ef should show no python/httpx processes from that job. The row ends as cancelled, the lock file is released, the next queued job starts.
4. Cancel queued: enqueue A, B, C. Cancel B while A runs. B’s status flips to cancelled immediately without ever starting; C runs after A.
5. Restart recovery: start a job; pkill -f imprint.api; restart. /api/queue shows the old running row as failed with error='api_restart', the lock file is gone, new jobs enqueue normally.
6. Process-group cascade: during retag with the local llama-cpp tagger, cancel. ps --forest should show the subtree (llama.cpp threads, httpx workers, extractor helpers) gone in one shot, not staggered.
7. Merge: queue refresh of a folder twice while another job runs. The second request returns the first job’s ID with merged: true; the queued row shows “2 requests”; the folder is refreshed once.
8. Background tagging steps aside: with tagger.llm on, run imprint learn; it finishes after embedding and prints how many memories wait. The Queue page shows background tagging running. Queue a URL ingest: tagging yields within one classification, the ingest runs, tagging resumes. Purge a file from the dashboard (or run imprint update) while tagging runs: it takes the lock within seconds, and tagging resumes after it.

Workspace and lock integrity

New submissions store their workspace in the job body as _workspace. The dispatcher pins IMPRINT_WORKSPACE in the child; switching the dashboard’s active workspace does not redirect an already queued job. Old jobs missing this field keep the previous active-at-execution behavior. Go and Python honor the same IMPRINT_DATA_DIR.

Stale metadata cleanup acquires the original lock inode before truncation and never unlinks it. Qdrant first startup has a separate shared lock with a health recheck, preventing competing CLI/API/MCP processes from starting two daemons. Collection creation has its own shared lock (data/qdrant-collections.lock), held by threads as well as processes: Qdrant lists a new collection, and answers 409 to a second create, before the first create finishes, and fails reads of it until then. On Windows this failed the dashboard’s first overview with a 500. PID reuse, crash stress and complete Windows descendant cancellation still need dedicated tests; liveness alone is not process identity.

Queued ingest and refresh accept force: true and batch_size: N; equivalent CLI flags are preserved when enqueueing. Forced runs re-embed unchanged content within the same model profile.

Browser capture prototype

The WIP Chrome extension submits bounded inline text to POST /api/browser/captures. Its internal browser-capture jobs use the same SQLite queue and ingestion lock, with a Python worker launched by job ID instead of the Go CLI. This command is not added to the general HTTP command allowlist. The paired workspace is pinned on submission and pairing is checked again before indexing. Capture text is retained in the job body/history; the browser status endpoint exposes only state and timestamps. The dashboard keeps normal local queue visibility and cancellation. Revocation is not cancellation of a job already indexing; multi-message captures can partly complete before a failure.