About

How it was built.

Two LoRA adapters on one frozen Qwen3.5-4B base, trained on synthetic private-share cases, served behind durable Oban queues in Elixir.

Base model: Qwen3.5-4B
Training: LoRA on 1× L40S
Cases: 2,469
Runtime: Elixir · Oban

Evaluation

Both models ran on the same 124 held-out cases, with identical prompts and decoding settings on one Modal L40S.

+59.7 pts full-verdict accuracy · 40.3% → 100%

+28.2 pts risk-level accuracy · 71.8% → 100%

100% valid output contract, up from 87.9%

Untuned base vs SFT · score (%) Untuned Qwen3.5-4B SFT adapter

Valid output contract

87.9% 100%

Exact risk (JSON only)

71.8% 100%

Exact risk + valid contract

62.1% 100%

Exact full verdict

40.3% 100%

Reason-code F1

65.4% 100%

Review recall

66.7% 100%

Identical prompts · greedy decoding · 160-token cap · thinking off

The gain goes beyond JSON and schema compliance: risk-level accuracy alone improved by 28.2 points.

Example

An applicant's stated finances contradicted their accreditation claim.

Untuned base: level 1 · NO_CONCERN
SFT: level 3 · ACCREDITATION_INCONSISTENT

Data

No public dataset covers private-share fraud, so the cases are synthetic and written to our own schema.

Step 1

Build the cases

The generator combined selected SAML-D transaction fields, BAF application telemetry, SEC-inspired scenarios, and fully generated controls. Code created the hypothetical share listing or bid, account, event sequence, documents, and computed summaries. It also assigned an expected risk level and reason codes under our explicit labeling policy.

For example, a missing ownership document plus an oversized listing creates a concern. A large listing backed by sufficient verified ownership is a benign control.

Source laundering or bank-fraud labels did not become private-share fraud labels. IBM AMLworld, RBA, NFT, and phishing datasets were proposed, but were not ingested into this corpus.

Step 2

Generate and audit the verdict

The final accepted corpus used GPT-6 Luna through Droid:

One request proposed a risk level, fixed reason codes, and exact evidence citations.
A separate request audited that proposal for unsupported reasons, missed evidence, contradictions, and arithmetic errors.
Code checked the schema, citation paths and quotations, auditor agreement, and agreement with the saved policy label.
Failed cases received up to two correction rounds. Unresolved cases were quarantined.

The saved answer was not shown to the reviewer. However, acceptance required matching it, so this was a policy-consistency filter, not independent discovery of fraud truth. The separate audit calls also used the same model, not human reviewers.

JSON validation checks

Schema: strict Pydantic models; unknown fields and wrong types fail
Timeline: events oldest first, before review, never before the account existed
Arithmetic: summaries are recomputed from the facts and must match exactly
Order: positive whole shares, positive price, matching company
Verdict: risk 1–5; unique codes from the fixed list
Evidence: every citation must appear word for word in the case
Runtime: the same models export JSON Schema for the Elixir app

Step 3

Expand and retain passing cases

We reviewed 500 candidates, then another 2,000.

2,469 accepted

31 excluded

2,199 train · 146 validation · 124 test

Suspicious 1,229 Ordinary 625 Unusual but benign 615

Earlier Dots3 and DeepSeek experiments tested the workflow; Dots3 also wrote neutral notes for the initial batch.

The result is synthetic, template-based supervision, not confirmed fraud outcomes. Scenario families appear across training and test splits, which helps explain the very high test scores.

SFT model

Supervised fine-tuning teaches the model to answer with a risk level from 1 to 5 and reason codes from a fixed list, as JSON. Loss is computed on the answer tokens only; the prompt is masked and thinking is off.

Loss, log scale Train Validation

Method: BF16 LoRA, base frozen
LoRA: rank 16 · alpha 32 · dropout 0.05
Trainable: 32.46M params (0.71%)
Schedule: 2 epochs · 276 steps · batch 16
Learning rate: 1e-4, 5% warmup, cosine
Checkpoint: step 150, lowest validation loss

Schema-valid: 124 / 124
Exact risk level: 124 / 124
Exact reason set: 124 / 124
False alarms: 0 of 64 benign
Median latency: 1,535 ms · ~19 tokens
Training time: 97 min · 19 GB peak

Decision model

Generating JSON takes about 1.5 s, which is too slow to run on every listing. The decision model, in the style of Cloudflare's Clef, writes nothing. One forward pass reads the logits of two single tokens, ok and review, and a softmax over them gives P(review). Risk levels 1–2 map to ok, 3–5 to review.

Base model, untrained

Test AUROC: 0.941
False alarms at 0.5: 41 of 64
Mass on the two answers: 62%

After training

Test AUROC: 1.0
False alarms at 0.5: 0 of 64
Mass on the two answers: 99.97%

Loss Train Validation

Loss: cross-entropy over the two answer logits
Checkpoint: step 25, ~14 min in
Median latency: 194 → 128 ms with flash-linear-attention
Threshold: min(0.5, lowest validation review score)

What happened in the run

Validation loss reached 0.0 at step 25 and stayed there. The run was planned for 276 steps but stopped at step 145 when the Modal account ran out of credits. The saved step-25 checkpoint is the evaluated adapter. Later runs stop after two validation checks without improvement.

Runtime

An OTP application with Ecto, PostgreSQL, Oban, and Phoenix PubSub. Each model sits behind its own durable Oban queue, so intake survives crashes and slow SFT calls never block fast decisions.

Intake Ecto.Multi Case row and decision job commit together
Oban queue risk_decision 4 workers · 128 ms per call
Policy research-v1 Clear, or send to SFT with a 2% audit
Oban queue risk_sft 1 worker · 1,535 ms per call
PostgreSQL Worklist Highest risk first, then oldest

Ecto.Multi.new()
|> Ecto.Multi.insert(:order, order_changeset)
|> HiiveRisk.enqueue_review(&MyMarketplace.review_snapshot(&1.order))
|> Repo.transaction()

Idempotency: one case and one job per event ID and policy version
Retries: 5 attempts, then a visible failed case; never auto-clears
Safety: model calls outside transactions; row locks on results
Updates: PubSub after commit; the database is the source of truth
Tests: 17 against real PostgreSQL; 20 concurrent submits → 1 job

Serving

start_models.py loads both adapters on one Modal L40S over the same frozen base and opens a loopback gateway for the Elixir app. Requests run one at a time, so the two adapters never race.

POST /decision: returns p_review and answer_mass
POST /sft: returns risk_level and reasons
Input limit: 8,192 prompt tokens; longer cases are rejected, not truncated