# Private-share risk model

This research project trains a small model to assign review priority to a private-share listing or bid. It is not a production compliance system.

The input contains the order, account facts, recent events, company rules, and computed values. The output is JSON with a risk level from 1 to 5 and reasons from a fixed list.

## Data

The dataset has 2,469 accepted synthetic cases: 1,229 suspicious, 625 ordinary, and 615 unusual but benign. GPT-6 Luna reviewed the cases through Droid. Schema, evidence, audit, and policy checks controlled acceptance. Another 31 cases failed these checks and were excluded.

Some fields use SAML-D and BAF source data. SEC cases informed synthetic scenarios. Source labels do not establish private-share fraud. SAML-D and BAF have CC BY-NC-SA 4.0 licenses. Use this work for noncommercial research.

## Training

Supervised fine-tuning (SFT) is complete. No reinforcement learning was used.

| Setting | Value |
|---|---|
| Model | Qwen/Qwen3.5-4B |
| Hardware | One Modal L40S |
| Method | BF16 LoRA; base weights frozen |
| LoRA | Rank 16; alpha 32; dropout 0.05 |
| Trainable parameters | 32.46 million; 0.71% of loaded parameters |
| Train / validation / test cases | 2,199 / 146 / 124 |
| Epochs / optimizer steps | 2 / 276 |
| Effective batch size | 16 |
| Learning rate | 0.0001; 5% warmup; cosine decay |
| Loss | Answer tokens only; thinking disabled |
| Selected checkpoint | Step 150; lowest validation loss |

![Training and validation loss](docs/training-loss.png)

## Results

Validation loss fell from 0.46461 to 0.00319. On the 124 test cases, all outputs matched the JSON schema, risk level, and expected reason set. There were no false alarms on the 64 benign test cases.

The test cases were not used for training or checkpoint selection. No exact test prompt appeared in training. However, all 14 test scenario families also appeared in training. These results measure agreement with a synthetic policy, not real-world fraud detection.

See the [run report](docs/training-report.json), [loss measurements](docs/training-metrics.jsonl), and [development log](devlogs.md).

## Triage decision model

A second model does fast triage, in the style of Cloudflare's Clef decision models. It does not write text. It reads one case and returns a probability for each of two answers:

| Answer | Risk levels | Meaning |
|---|---|---|
| `ok` | 1–2 | Routine, or unusual with specific benign evidence |
| `review` | 3–5 | Needs a deeper human look |

The two answers are single tokens in the Qwen tokenizer. One forward pass gives their logits at the answer position, and a softmax over those two logits gives P(ok) and P(review). Training uses cross-entropy over the two answers only. The base model, LoRA settings, data, and splits are the same as the SFT run.

![Decision model loss, test probabilities, and latency](docs/decision-model.png)

| Measure | Value |
|---|---|
| Selected checkpoint | Step 25; first step with validation loss 0.0 |
| Test accuracy | 124/124; 60 review and 64 ok cases |
| False alarms / missed reviews | 0 / 0 |
| Test AUROC | 1.0 |
| Probability on the two answers | 99.97% of the full vocabulary |
| Base model, before training | AUROC 0.941; flags 41 of 64 ok cases at 0.5 |
| Review threshold | 0.5; lowered only if a validation review case falls below it |
| Median latency, decision model | 194 ms per case on one L40S; 128 ms with `flash-linear-attention` |
| Median latency, generative SFT | 1,535 ms per case; about 19 generated tokens |

The run planned 276 steps, but stopped at step 145 when the Modal account ran out of credits. The weights reached zero validation loss at step 25, about 14 minutes into the run. The evaluated adapter is the saved step-25 checkpoint, not the final weights. Future runs stop after two validation checks without improvement.

The probabilities are almost always 0 or 1. The task is easy for this model because test scenario families also appear in training. These results do not show calibrated risk on new kinds of cases.

See the [decision report](docs/decision-report.json), [loss measurements](docs/decision-metrics.jsonl), and [kernel benchmark](docs/decision-benchmark.json).

## Elixir runtime hook

`elixir/` is an OTP application with Ecto, PostgreSQL, Oban, and Phoenix PubSub. It accepts a case at runtime. The caller builds the same input snapshot used for training, including computed summaries. The hook checks the schema and event order. It does not recompute summaries.

```text
Case snapshot → durable decision queue → auto-clear
                                     → durable SFT queue → human review / escalation
```

```elixir
{:ok, receipt} = HiiveRisk.enqueue_review(snapshot)
HiiveRisk.subscribe(receipt.case_id)
HiiveRisk.get_assessment(receipt.case_id)
HiiveRisk.list_reviews(priority: 4)
HiiveRisk.list_reviews(disposition: "escalate")
```

The receipt confirms intake, not a model verdict. The snapshot's `case_id` identifies the source event. Repeating it under the same `policy_version` returns the existing case. Different evidence or routing options under that key return `:idempotency_conflict`. Use a new event ID for new evidence.

To include intake in a marketplace transaction, use the same repository and database for the order, review tables, and Oban. The caller supplies `MyMarketplace.review_snapshot/1`.

```elixir
Ecto.Multi.new()
|> Ecto.Multi.insert(:order, order_changeset)
|> HiiveRisk.enqueue_review(
  fn %{order: order} -> MyMarketplace.review_snapshot(order) end,
  policy_version: "research-v1"
)
|> HiiveRisk.Repo.transaction()
```

If the transaction fails, neither the review case nor its job remains. Model calls run after commit, outside database transactions. Separate queues allow four decision calls and one SFT call at a time. Oban retries failures up to five attempts. Exhausted cases have status `failed`; failures never auto-clear. After repair, an operator can retry a discarded job with `Oban.retry_job/1`.

The research policy auto-clears P(review) below 0.02 only when answer-token mass is at least 0.95. A random 2% of eligible cases go to SFT for audit. The options `unusual_input: true` and `conflicting_evidence: true` prevent auto-clear. Tests can force or disable sampling with `audit_sample: true` or `false`. The default policy version is `research-v1`.

Other cases go to SFT. Conflicting evidence or disagreement at the decision model's 0.5 and SFT's level-3 boundaries produces an escalation record. No frontier model is called. The human worklist sorts by descending risk level, then oldest first. It is stored in PostgreSQL, not in Oban. Subscribe before reading an assessment to receive `{:risk_updated, assessment}` messages. PubSub messages are advisory; re-read the database after reconnecting. Review completion does not approve an order or move money.

### Run the hook

Use Elixir 1.18 or later and PostgreSQL 15 or later. Set `DATABASE_URL` and a separate `TEST_DATABASE_URL`. The defaults use local `hiive_risk_dev` and `hiive_risk_test` databases with the development-only user and password `postgres`.

From `elixir/`, run:

```sh
mix deps.get
mix ecto.setup
mix test
iex -S mix
```

Set trusted model URLs before starting the app. Both endpoints must accept a POST with the bare case snapshot and return JSON:

| Environment variable | Required response |
|---|---|
| `DECISION_MODEL_URL` | `{"p_review": 0.9, "answer_mass": 0.9997}`; both values from 0 to 1 |
| `SFT_MODEL_URL` | `{"risk_level": 4, "reasons": ["SHARED_BANK"]}`; existing verdict schema |
| `MODEL_API_TOKEN` | Optional bearer token sent to both endpoints |

`answer_mass` is the full-vocabulary probability mass on the two answer tokens. It is not the sum of their renormalized binary probabilities. The HTTP adapter has a 30-second receive timeout. Oban owns retries; redirects are not followed. Missing endpoints and invalid responses fail the job.

### Start both models

From the repository root, with Modal credentials configured:

```sh
.venv/bin/python start_models.py
```

The script loads the saved decision and SFT adapters on one Modal L40S. Both use the same frozen Qwen base. Requests run one at a time so adapter selection cannot race. It uses each run's saved system prompt, disables thinking, and rejects inputs over 8,192 prompt tokens without truncating evidence.

After both adapters load, the script opens a loopback-only gateway on port 8767 and prints the `DECISION_MODEL_URL` and `SFT_MODEL_URL` export commands. Set these in the Elixir process's environment before starting it. POST `/decision` returns `p_review` and full-vocabulary `answer_mass`; POST `/sft` returns the validated verdict. GET `/health` reports gateway readiness. Use `--port` to change the port. If `MODEL_API_TOKEN` is set, the gateway requires that bearer token.

The gateway and Elixir must run on the same machine. One GPU stays warm, and incurs Modal charges, while the script runs. Ctrl-C stops the gateway and exits the temporary Modal app. In an orb, run it as a supervised service with `amp orb service start risk-models --command '.venv/bin/python start_models.py'`; stop it with `amp orb service stop risk-models`.

This script does not retrain or permanently deploy either model. The static dashboard in `web/dashboard/` still replays saved predictions. The routing thresholds remain research rules, not calibrated production controls.

## Code

`dataset.py` defines the schema. `prepare_sources.py`, `quality_batch.py`, `review_batch.py`, and `expand_data.py` prepare and review cases. `sft_modal.py` exports inputs, trains adapters, and evaluates outputs. `decision_modal.py` trains, evaluates, benchmarks, and serves the triage decision model.

```sh
.venv/bin/python decision_modal.py prepare           # local tokenization and checks
.venv/bin/python decision_modal.py smoke             # two-step GPU test
.venv/bin/python decision_modal.py score cases.json  # P(ok) and P(review) for new cases
```

```sh
uv venv
uv pip install -r requirements.txt
.venv/bin/python -m unittest -q
```

Raw datasets, API responses, credentials, and model weights are not included. Training requires the local accepted datasets and Modal credentials. The trained adapters remain in the private Modal volume `hiive-research-sft`, under `qwen35-4b-sft-002/adapter` and `qwen35-4b-decision-001/adapter`.