Project-Then-Heal: Weight-Space Programming and Subspace-Locked LoRA

Part II (Trapped Inside <think>) showed that a behavioral reflex, such as looping inside <think> or falsely refusing a defensive C++ audit, can be measured as a direction in activation space and removed directly from the matrices that write into the residual stream, with no training at all.

Suppose that after applying that weight surgery, you want to fine-tune the model. Perhaps it needs to learn a new domain format, or you want a brief LoRA healing pass to smooth out the perplexity of a large edit.

A standard LoRA adapter does not encode which subspace the edit removed. Nothing stops it from writing back along those directions: any task gradient with a component along them moves the adapter there. Projecting only the gradient is not enough either, because AdamW rescales each coordinate separately and can rotate a projected step back into the removed directions. Whether that mechanism explains behavioral drift in other fine-tuned models was not tested here.

The projection construction has prior art. OPLoRA, by Yifeng Xiong and Xiaohui Xie (AAAI 2026), projects LoRA updates on both sides against dominant weight-SVD subspaces. SaLoRA, by Li et al. (ICLR 2025), preserves safety alignment with a fixed safety module and constrained adapter features.

GOAL, by Chen et al. (October 2026), Appendix M, defines residual refusal-subspace leakage after AdamW preconditioning, L_Adam, and reports 3.6% to 4.7%. That gradient-step metric differs from our adapter weight-update leakage below. CORD, by Yu et al. (October 2026), cautions that momentum, coordinate scaling and weight decay can change the final step; it does not establish GOAL’s exact metric. PoLoRA, by Ghosh et al. (July 2026), develops a preconditioned orthogonalized LoRA optimizer.

Project-Then-Heal handles this structurally. First, we apply closed-form weight surgery along an orthonormal basis of behavioral directions. We store the basis as the rows of a matrix D (shape k × d_out, D · D^T = I), the same convention as the Secant code. Then, when we attach a LoRA adapter (ΔW = s · B · A), we parameterize B inside the orthogonal complement of those rows on every forward pass:

B = P_perp · (P_perp · B_tilde)     where     P_perp = I - D^T · D

Because D · P_perp = D - (D · D^T) · D = 0, the adapter’s weight update satisfies D · ΔW = 0 in exact arithmetic, regardless of learning rate, momentum, weight decay, or AdamW preconditioning. In floating point it holds to rounding level: after 60 steps of live training on Qwen3.5-0.8B the largest entry of D · ΔW was 9.87e-18 in float64 and 2.49e-10 in the merged F32 delta.

Secant evaluates a post-surgery workflow using a protected behavioral basis and twice-applied float64 projection. This constrains selected writers locally; later layers and storage rounding can change behavior, so the numerical lock is not a general behavioral guarantee.

Prior work and the scope of these tests

The tests build on earlier work in four areas.

1. In-place checkpoint surgery. Prior work: Kiwimaker showed Q8_0 editing with untouched-byte verification, and SC117 showed tensor transplantation and Q2_0 re-encoding.

Tested here: SafeTensors byte-slice editing, touched 34-byte Q8_0 block surgery, and the measured Pi_128 versus Q4_K alignment boundary.

2. Project-Then-Heal. Prior work: OPLoRA forward projection; SaLoRA safety-subspace preservation; GOAL’s exact AdamW leakage metric; CORD’s optimizer-transformation caveat.

Tested here: Locking a post-surgery adapter out of an excised behavioral subspace, twice-applied float64 projection, and direct Arm B versus Arm C measurements.

3. Target-Shift Calibration. Prior work: ROME rank-one associative editing, MEMIT multi-edit work, and later multi-direction projection methods.

Tested here: Reference-activation sizing, RMS-relative dosing, and upper-triangular R^-1 Gram-Schmidt compensation for simultaneous non-orthogonal edits.

4. Boundary-3 auditing. Prior work: Fornaro’s NaN router advisory; HIP-0113 Section 4.5, warning of cast divergence in int8 consensus.

Tested here: x86-64 versus ARM64 synthetic C++ kernel behavior, the spectral Trojan crossover, and role-aware block-scale and sink scanning.

These experiments follow the questions about repeatable editing and static auditing raised by Takehi et al. in Sparks of Weight-Space Programming.

This third post in the series covers the engineering behind Secant (documentation):

1. How to read vocabulary concepts straight out of frozen .safetensors and .gguf weights from the terminal without running inference.
2. Two container-layout traps between HuggingFace .safetensors and llama.cpp .gguf files, plus how Q8_0 and Q4_K block sizes interact with the edited panels.
3. What we measured for Target-Shift Calibration and ProjectThenHealLinear across 17 checkpoints (0.8B to 72.7B), and how forward-pass projection keeps edited subspaces locked during LoRA fine-tuning.

What reading and writing weights looks like in the terminal

Behavioral tests require running the model. Static weight inspection can offer a narrower view of the directions a layer could write, before inference.

With secant inspect and secant read, you point the tool at a .gguf or .safetensors file on disk. Without running a single forward pass, it memory-maps the tensor bytes from disk, extracts the strongest output directions of that matrix (u_1, u_2, u_3) via deterministic float64 power iteration, and multiplies them against the model’s vocabulary table to list which token IDs that layer pushes into the residual stream and which it suppresses (decode the IDs with the model’s tokenizer to read them as words):

$ secant inspect --model qwen35-4b-base-bf16.gguf
qwen35-4b-base-bf16.gguf: gguf, 441 tensors, 8665620384 bytes, payload at +10968992
architecture qwen35, 33 blocks, hidden 2560
  BF16 257 tensor(s)
  F32  184 tensor(s)
qwen3.5 GDN ssm_out: 24 layer(s), logical [2560 x 4096] BF16
  GGUF blk.N.ssm_out.weight stores linear_attn.out_proj.weight values with the LOGICAL INPUT axis
  (4096 columns = 32 value-head blocks x head_v_dim 128) permuted:
  HF block h = k*r + j lands at GGUF block j*num_k_heads + k; edit plans map input columns accordingly.

$ secant read \
    --model qwen35-4b-base-bf16.gguf \
    --tensor "blk.13.ffn_down.weight" \
    --vocab-tensor "token_embd.weight" \
    --top-k-singular 3 --top-cols 8 --detect-anomalies
read qwen35-4b-base-bf16.gguf (gguf): 1/441 tensors selected, 1 analyzed;
  blk.13.ffn_down.weight BF16 [9216 2560]: ok fro=41.1292 s1=2.65075 s2=2.33822 s3=2.01852
      promoted tokens:   151353(0.139) 211741(0.137) 180201(0.132) 200228(0.128) ...
      suppressed tokens: 6843(-0.0415) 3648(-0.0404) 220(-0.0378) 64(-0.0377) ...

Two notes on that output. 33 blocks is the GGUF block_count field as stored in the file: Qwen3.5-4B has 32 decoder blocks (blk.0..31) plus blk.32, which is the Multi-Token Prediction (MTP / NextN, blk.32.nextn.*) head block exported by llama.cpp’s GGUF conversion. And [9216 2560] is the GGUF ne[] order (contiguous input axis first), so the logical shape of blk.13.ffn_down.weight is [2560 x 9216] (d_out x d_in). That ordering difference is exactly Trap 1 below.

And when you compile and apply a multi-subspace surgical plan (B1 + B2 + B3), secant compose, secant patch, and secant verify-patch edit only the 4 target tensors at their exact byte offsets inside the 8.67 GB .gguf file and verify by SHA-256 that the other 437 tensors did not change by a single bit:

$ secant compose \
    --profile docs/profiles/qwen35-4b-cal738.profile.json \
    --vectors ./vectors \
    --out ./plans/cal738.plan.json --json

$ secant patch \
    --base ./qwen35-4b-base-bf16.gguf \
    --plan ./plans/cal738.plan.json \
    --out ./qwen35-4b-cal738-bf16.gguf --json

$ secant verify-patch \
    --base ./qwen35-4b-base-bf16.gguf \
    --patched ./qwen35-4b-cal738-bf16.gguf \
    --expect-changed blk.8.ffn_down.weight,blk.13.ffn_down.weight,blk.16.ssm_out.weight,blk.16.ffn_down.weight --json

For the Qwen3.5-4B cal738 checkpoint, the four target tensors occupy 162.5 MB of an 8.67 GB file (1.88%), and the modified bytes come to just 17.6 MB (0.20%). The recorded run, including copying the file and verifying pre/post SHA-256 digests, took 22 seconds.

Two container-layout traps and one quantization boundary

Direct GGUF editing has prior art, listed under “Prior work” above. The tests here cover SafeTensors byte-slice editing, touched-block Q8_0 edits, and the panel-alignment boundary below.

One common workflow loads weights into PyTorch, edits them in memory, saves a new .safetensors directory, and converts and quantizes it again. To patch .safetensors and .gguf files directly on disk, Secant has to solve two container traps and enforce a strict quantization boundary.

Trap 1: GGUF ne[] metadata vs. logical [d_out, d_in] orientation

A linear writer matrix maps an input vector of length d_in to an output vector of length d_out (y = W x).

• In SafeTensors (HuggingFace nn.Linear), the shape array is [d_out, d_in], stored row-major so d_in is the contiguous inner dimension.
• In GGUF (ggml), the dimension array ne[] lists the contiguous dimension first: ne[0] = d_in and ne[1] = d_out.

Because ne[0] is contiguous in GGUF just as shape[1] is contiguous in SafeTensors, the underlying row-major memory layout on disk is identical in both formats: element (out_row, in_col) sits at offset out_row * d_in + in_col. However, if a GGUF tool naively reads rows = ne[0] and cols = ne[1], it transposes non-square matrices like ffn_down and applies residual-stream projections along the wrong axis. secant-core enforces logical (d_out, d_in) indexing at the type boundary so both formats share identical math.

Trap 2: Inverting Qwen3.5’s 32 x 128 Gated DeltaNet ssm_out permutation

Hybrid Qwen3.5 models interleave standard attention layers with Gated DeltaNet (GDN) linear-attention layers (24 of the 32 layers in Qwen3.5-4B). When llama.cpp converts Qwen3.5 from HuggingFace (linear_attn.out_proj.weight) to GGUF (blk.N.ssm_out.weight), it reorders the 4,096 input columns in 128-column value-head panels from key-head-major to value-head-major.

Specifically, with 16 key heads and 32 value heads (r = 2 value heads per key head, head_v_dim = 128), HuggingFace 128-column panel h = 2k + j lands in GGUF at panel 16j + k:

HuggingFace 128-col panel h:   0   1   2   3   4   5   6   7  ...
GGUF 128-col panel position:   0  16   1  17   2  18   3  19  ...

Without inverting that permutation, an edit computed in HuggingFace order corrupts the wrong value heads on all 24 linear-attention layers (0 / 24 match). With Secant’s exact bijection (Π_128), all 24 Gated DeltaNet layers match HuggingFace bit-for-bit (24 / 24 PASS).

The alignment limit for direct block edits

What happens when the .gguf file on disk is already quantized?

Panels are 128 ssm_out columns. Remaining is the residual refusal component, starting at 0.517; Q8_0 removes 98.5%. Q4_K has 8 straddling blocks per 2,048-column row, or 16 per 4,096-column row.

Because 32 divides 128 evenly (128 = 4 × 32), every 128-column ssm_out panel consists of exactly four complete Q8_0 blocks. Permuting or editing a panel never crosses a block boundary. On Qwen3.5-0.8B.Q8_0.gguf, secant quant patch modifies only the targeted 34-byte blocks in place, leaves all 240,640 untouched blocks 100% bit-identical, and matches BF16 edit-then-quantize within 0.179% to 0.923% relative Frobenius error per edited tensor.

By contrast, a Q4_K superblock spans 256 elements, straddling two 128-element value-head panels that share a single top-level FP16 scale d and minimum dmin. Re-encoding one panel can change the shared scale and affect its neighbor. In this test, 67% of the measured refusal component remained. For that reason, secant quant patch accepts only Q8_0 tensors and refuses Q4_K (and every other dtype) with the error tensor <name> is <dtype>, block-aligned surgery needs Q8_0.

Interactive Explorer: 17 checkpoints in the Project-Then-Heal study, three test regimes

Two different groups of seventeen appear in this series, and they are not the same set. The 17 models behind the 59,440 benchmark rows in Part II run from 0.8B to 32.8B. The 17 checkpoints in this Project-Then-Heal study (2 in Regime A, 1 in Regime B, 14 in Regime C) run from 0.8B to 72.7B, with the two largest tested on a single shard each. We ran them on CPU, AMD Radeon 8060S (ROCm), and NVIDIA H100 80GB hardware. They were not all tested the same way, and the numbers from one regime do not transfer to another:

• Regime A, live fine-tuning after real surgery: Qwen3.5-0.8B (60 AdamW steps) and SmolLM2-1.7B (20 steps). Both LoRA arms train on real completions, and both are scored with the same metric, the largest entry of D · ΔW.
• Regime B, generative benchmark: Qwen3.5-4B at several calibrated doses (the Part II benchmark). No healing run.
• Regime C, H100 stress test: 12 full models (7.2B to 32.8B) plus one down_proj tensor from a single shard of each of two 70B-class models. The edit is a fixed-ratio rank-1 update (γ = 0.045 · ‖W‖_F), not Target-Shift Calibration, compared against an equal-norm random direction. Healing trains for 60 AdamW steps on the unedited down_proj toward a synthetic target Y + 1.5 · v_B2 that lies inside the protected subspace.

Regime C is a stress test of the lock, not a model of real fine-tuning. Because the target lies inside span(D), the locked adapter cannot fit it at all: its loss sits at the floor 2.25 / d_model on every checkpoint, so regime C says nothing about learning speed. The unconstrained percentage is the share of the adapter’s B norm inside span(D) under a task that demands it, which is a different metric from the locked arm’s absolute leakage, so we never divide one by the other. And the edit-versus-random ratio is a geometric sanity check: a random unit direction projects onto the target direction with expected size about 1/sqrt(d_model), so ratios of 10² to 10³ are what geometry predicts.

Click any model in the explorer for its regime and numbers:

Regime B, Qwen3.5-4B dose-response (n = 30, one greedy run per arm):

Counts out of 30 at a 1,536-token cap; closed = emitted </think>, class/frame = strict final-answer scores, root cause = both correct.

Arm key: base = unedited; rand1/rand2/rand3 = the three random-direction controls (cal738rand1..3, seeds 20261105..20261107) described in Part II; g030 = uncalibrated γ = 0.30; b1only = calibrated loop edit; cal738 = B1 + B2 + B3 at 1.0x.

The a05, a06, a07, a08, a09, and a15 arms scale only B1 to 0.50x, 0.60x, 0.70x, 0.80x, 0.90x, and 1.50x; B2 and B3 remain at 1.0x. Shift is the summed B1 target shift; N/A means no calibrated B1 target applies.

Exact two-sided McNemar tests against base: a08 closure p = 4.88e-4 (12 vs. 0 discordant), a09 closure p = 1.22e-4 (14 vs. 0 discordant), cal738 closure p = 9.54e-7 (21 vs. 0 discordant), a15 closure p = 1.86e-9 (30 vs. 0). a15 vs. cal738: closure p = 0.0039, frame p = 0.0078, bug class p = 0.125 (not significant). In the a05..a15 arms only the B1 edits (B1a, B1b) were scaled; B2 and B3 were held at their cal738 (1.0x) values in every arm.

Regime A, live fine-tuning (same metric for both arms):

Leakage is max abs(D · ΔW); locked = ProjectThenHealLinear, free = unconstrained LoRA, F32 = merged delta.

Held-out perplexity at the indicated step; lower is better.

Qwen/Qwen3.5-0.8B ran on CPU after a composed B1 + B2 + B3 GGUF edit (verify-patch exit 0); both arms overfit after step 20. HuggingFaceTB/SmolLM2-1.7B-Instruct ran on ROCm: its L14 o_proj B2 shift was -2.436 versus random -0.0359 (67.9x), and its calibrated L15 down_proj B3 shift was +3.952 versus +0.179 (22.1x).

Regime C, H100 stress test (Part 1, edit shifts vs. random controls) (shift values at dose 1.0x; edit layer = layer of peak SNR_proj):

Model labels omit Instruct/it suffixes; Mistral is v0.3, OLMo is 2-1124, and R1 denotes DeepSeek-R1-Distill. Width is d_model; note markers are defined below.

B2 shifts at 1.0x; ratio compares magnitudes. Single-shard rows [x] use down_proj.

B3 shifts at 1.0x (- indicates single-shard runs where B3 was not evaluated):

Regime C, H100 stress test (Part 2, subspace lock leakage at step 60) (same 14 checkpoints):

Free is the unconstrained adapter’s fraction of B inside the protected subspace, ‖DB‖_F / ‖B‖_F; locked columns report max abs(D · B_locked), so the metrics are not directly comparable.

Notes:

• [d] Dose response is not linear for every edit. For 20 of the 24 full-model edits, raising the dose from 1.0x to 1.5x multiplied the shift by 1.49 to 1.50. The exceptions are phi-4 B2 (1.04), OLMo-2-13B B3 (1.18), gemma-2-9b B3 (1.04), and gemma-2-27b B3 (1.07).
• [s] phi-4 B2 moved the wrong way. The intended shift is positive; the measured shift is -3.399 at 1.0x and -3.519 at 1.5x. The 326.9x ratio is by magnitude only.
• [p] The phi-4 capture looks degenerate. Its peak B2 SNR_proj is 27,362.3 with SNR_trace 60.4 at L32 (and up to 909.6 at L16), against 83.0 to 199.4 and 0.32 to 4.88 on the other full models. This pattern suggests an outlier-dimension artefact rather than a sharper behavioural direction. phi-4 was run twice; the table uses run 1 (the rerun gives 8.70% and 9.03e-10 / 1.74e-08, so the LoRA runs are not bit-reproducible).
• [x] Single-shard rows. Each used one .safetensors shard and one down_proj tensor: no activation capture and no B3 edit. The receipt does not record how the edit direction was chosen, and the producing script is not banked. The banked run log shows the first 72.7B attempt failing and contains no 70.6B run, so these two rows rest on the receipt alone.

Three takeaways, each scoped to the regime that supports it:

1. The two live fine-tuning arms had similar loss (regime A). On Qwen3.5-0.8B both arms reached similar training loss (2.8242 to 0.3120 locked, 0.3096 unconstrained) and similar held-out perplexity minima at step 20 (5.435 vs. 5.423). Both then overfit the 24-example batch (12.106 vs. 12.357 at step 60), so report the step-20 numbers, not step 60. Regime C cannot test this claim.
2. The lock holds under direct pressure (regime C). With a gradient that points straight into span(D), ProjectThenHealLinear leakage stayed at 4.82e-10 to 1.08e-9 (float64) at step 60 on all 14 checkpoints. Regime C has no gradient-projection-only arm, so it does not test the AdamW rotation argument; the separate three-arm test below does.
3. One storage precision rule (F32 vs. BF16 merge). Merging W_base + ΔW into a BF16 tensor rounds to 7 stored fraction bits (8-bit significand precision, ulp ≈ 7.8e-3). In our live runs that left 1.3e-4 to 9.5e-4 of leakage along D in the locked arm, comparable to the unconstrained arm on one SmolLM2 module. So secant heal stores B_locked in F32/F64, and secant verify-subspace fails closed at its default 1e-5 tolerance if you merge into BF16 without keeping the modified writer in F32.

Testing the AdamW argument: a three-arm rotation test

The claim at the top of this post, that projecting only the gradient is not enough under AdamW, needs an arm that does exactly that. This test has one. It is a synthetic regression, not language-model fine-tuning: the writer is the real layer-16 mlp.down_proj of Qwen3.5-0.8B, Qwen3.5-4B and Qwen2.5-7B, surgically edited so that D · W_e = 0, with k = 3 seeded random protected directions, 256 Gaussian inputs, LoRA rank 16, AdamW, 60 full-batch steps, and 3 seeds. Three arms start from the same A_0 and B_0 = 0:

• A: unconstrained LoRA.
• B: gradient-projected LoRA (grad_B ← P_perp · grad_B in float64 right before each AdamW step).
• C: ProjectThenHealLinear.

All three are scored with one metric, lk_W = ‖D · ΔW‖_F / ‖ΔW‖_F. The orthogonal target lies entirely outside span(D). The mixed target adds a target component R_D of relative norm 0.5 inside span(D) (||R_D||_F / ||R_perp||_F = 0.5), whose normalized mean-squared loss floor is 0.5^2 = 0.25 for any arm that cannot write into span(D).

The learning rate here (3e-3) is post-hoc: it was added after the preregistered 1e-2 run failed its loss-parity prediction for arm C on Qwen2.5-7B. The preregistered run gives the same ordering (arm B 1.24% to 2.58%, arm C 6.40e-10 to 1.12e-9 on the orthogonal target).

The plot below draws all 60 steps of every arm from the same receipts as the table. Switch the target to see what the lock is for, and the weight decay to compare its measured effect:

The same three arms on 15 models (H100 Stage-1)

The panel at the bottom of the explorer is a second run of the three-arm test on the H100: 15 models from 0.8B to 32.8B, each starting from its surgically edited writer, trained for 60 AdamW steps on an orthogonal synthetic target plus real-token cross-entropy, one seed per model.

All three arms use one metric, ‖D·ΔW‖_F / ‖ΔW‖_F, at step 60. Arm A (unconstrained) leaks 0.82% to 8.51%; Arm B (gradient-projected, P⊥·∇B applied after backward and immediately before AdamW.step) leaks 0.33% to 1.67%; Arm C (ProjectThenHealLinear) stays at 1.78e-10 to 8.67e-10. Arm B is nonzero on all 15 models, as the AdamW argument earlier in this post predicts for gradient projection, and sits 6.5e6 to 3.8e7 times above Arm C on the same model. Unlike the three-model synthetic rotation test, Arm B sits below Arm A here on 15 of 15 models; the task differs (orthogonal synthetic MSE plus real-token cross-entropy), so the A-versus-B ordering is task-dependent and only the B-versus-C gap is common to both runs.

Limits. One seed (20261105) per model and one edited layer per model. Stage-1 evaluates adapter subspace locking (‖D·ΔW‖_F / ‖ΔW‖_F); downstream benchmark answers are evaluated in Stage-2 below.

H100 Stage-2: public benchmarks on 14 models

Stage-2 evaluated the calibrated B2 + B3 edits at 1.0x and 1.5x, plus norm-matched random controls, on 14 of the Stage-1 models.

Pooled over 14 models; change is calibrated 1.0x minus base in percentage points (pp), and p is the unadjusted exact paired test.

No calibrated per-model cell survived Holm correction across 70 model-by-benchmark comparisons per variant. The ASan pooled gain is not significant. Random 1.5x also lowered pooled MMLU-Pro by 2.0 pp, so edit magnitude explains at least part of that cost. Across 20 pooled tests, only the calibrated 1.5x MMLU-Pro cost survived Holm correction (p = 0.031 adjusted). There is no measured general benchmark uplift.

These are first-N samples: 100 items per general benchmark, 64 HumanEval and 30 ASan cases, one seed, greedy decoding, and a Qwen-heavy model set. ASan root cause uses bug-class and culprit keywords, distinct from Part II’s strict post-</think> scorer.

RMS-relative dosing across 6 models (4B to 14B, 24,000 evaluations)

The separate 6-model RMS-relative study measured sigma_writer = sqrt(mean_i ||W h_i||_2^2) over 24 reference activations and set Delta_target = alpha · sigma_writer. Across Qwen2.5-Math-7B-Instruct (L19), Qwen2.5-Coder-7B-Instruct (L19), DeepSeek-R1-Distill-Llama-8B (L21), Qwen3-4B (L24), Qwen3-8B (L24), and Qwen3-14B (L26), sigma_writer spans 5.847 to 78.148 (13.4x), so an unscaled unit shift of 1.0 ranges from 1.28% (Qwen3-8B) and 1.34% (Qwen2.5-Math-7B-Instruct) to 17.10% (DeepSeek-R1-Distill-Llama-8B) of the writer’s output norm.

All edited arms below use a 1.0x RMS dose; healed means composed plus ProjectThenHealLinear. MATH = MATH-500 and MMLU = MMLU-Pro; scores are correct items out of 100.

Qwen2.5-Math-7B (L19, 74.61). Parentheses give the layer and sigma_writer; healed relative leakage = 5.07e-17 (base_lora_1p0x unconstrained domain leakage = 4.75%; random_healed_rms_1p0x locked D_rand leakage = 5.02e-17).

Qwen2.5-Coder-7B (L19, 40.35). Parentheses give the layer and sigma_writer; healed relative leakage = 3.34e-17 (base_lora_1p0x unconstrained domain leakage = 3.69%; random_healed_rms_1p0x locked D_rand leakage = 5.42e-17).

R1-Llama-8B (L21, 5.85). Parentheses give the layer and sigma_writer; healed relative leakage = 3.43e-17.

Qwen3-4B (L24, 37.81). Parentheses give the layer and sigma_writer; healed relative leakage = 3.18e-17.

Qwen3-8B (L24, 78.15). Parentheses give the layer and sigma_writer; healed relative leakage = 3.75e-17.

Qwen3-14B (L26, 68.76). Parentheses give the layer and sigma_writer; healed relative leakage = 4.80e-17.

On Qwen2.5-Math-7B-Instruct, the 14-arm dose sweep across 0.25x / 0.5x / 1.0x / 2.0x gave GSM8K 88 / 89 / 87 / 89 for composed against 87 / 87 / 88 / 87 for random, and MATH-500 42 / 43 / 43 / 42 against 42 / 43 / 44 / 43. Across all 6 models, unhealed composed arms match their norm-matched random controls. Adding ProjectThenHealLinear (composed_healed_rms_1p0x, 60 steps in float64, keeping relative subspace leakage at 3.18e-17 to 5.07e-17) moved GSM8K from 87/100 to 93/100 on Qwen2.5-Math-7B-Instruct (6 gains, 0 losses, exact McNemar p = 0.031 unadjusted), HumanEval from 36/100 to 42/100 on Qwen2.5-Coder-7B-Instruct (12 gains, 6 losses vs. base, p = 0.238; 11 gains, 5 losses vs. composed, p = 0.210), and HumanEval from 5/100 to 12/100 on DeepSeek-R1-Distill-Llama-8B (p = 0.092 vs. base; Holm-adjusted p = 1.0 across the full test family on all models). Running the two direct 60-step LoRA control arms (base_lora_1p0x, unconstrained LoRA on the unedited base checkpoint, and random_healed_rms_1p0x, ProjectThenHealLinear locking the random basis D_rand on random_rms_1p0x) isolates the LoRA pass from the subspace lock: on Qwen2.5-Math-7B-Instruct, both base_lora_1p0x (93/100 GSM8K, 46/100 MATH-500) and random_healed_rms_1p0x (93/100 GSM8K, 46/100 MATH-500) also reach 93/100 on GSM8K (b = 3, c = 3 discordant pairs vs. composed_healed_rms_1p0x, exact McNemar p = 1.000), showing that the 87 -> 93 GSM8K gain comes from the 60-step LoRA adaptation pass on the calibration texts while ProjectThenHealLinear cuts domain-subspace leakage from 4.75% (base_lora_1p0x) to 5.08e-17. On Qwen2.5-Coder-7B-Instruct, neither base_lora_1p0x (37/100 HumanEval) nor random_healed_rms_1p0x (35/100 HumanEval) reproduces composed_healed_rms_1p0x (42/100 HumanEval, +5 and +7 net items, exact McNemar p = 0.332 and 0.189, within single-benchmark sample noise at n = 100), while ProjectThenHealLinear reduces domain-subspace leakage from 3.69% to 3.34e-17.

Calibration hit the reference targets, but benchmark-prompt mean prefill shifts along the edited directions were only 0.5% to 14.8% of the base output norm. Exact sizing at the reference does not guarantee the same dose on other inputs.

Three-model synthetic rotation results

The following tables return to the three-model synthetic rotation test described before Stage-1. Their three-seed results are separate from the one-seed Stage-1, Stage-2 and RMS studies.

Arm key: A = unconstrained; B = gradient-projected; C = ProjectThenHealLinear. Ortho. loss is orthogonal-target loss; loss ranges span three seeds. Leakage is lk_W at step 60, and wd is weight decay. Under the mixed target (which adds a target component R_D of norm 0.5 * ||R_perp||_F inside span(D)), the normalized mean-squared loss floor for a subspace-locked adapter is 0.5^2 = 0.25.

Qwen3.5-0.8B

Qwen3.5-4B

Qwen2.5-7B

What it shows:

1. Gradient projection alone leaks. Arm B puts 1.56% to 2.98% of its update into span(D) on the orthogonal target, and the number does not move when weight decay goes from 0.01 to 0.0. Weight decay is not the cause.
2. The leak comes from AdamW’s per-coordinate scaling. At step 1 the gradient already lies outside span(D) (‖D·g‖/‖g‖ about 1e-8), so arm B’s projection does nothing. AdamW’s first update is g / (|g| + ε), a sign-like map, and its leakage (1.76% to 3.48% across the 9 model-seed pairs) matches arm B’s recorded step-1 leakage to within 2e-7 relative.
3. The locked arm matched the unconstrained loss on this target. Arm C stays at 5.42e-10 to 1.05e-9 and its final loss matches arm A’s to the precision shown. Arm B ends with a higher loss than both.
4. On the mixed target the lock stops at the floor. Arm C cannot fit the part of the target inside span(D) and ends at 0.25 plus its orthogonal-task excess; arm A gets under the floor by putting about 45% of its update into span(D).

This is one layer, random protected directions and a synthetic task. It isolates the optimizer mechanism; it does not measure fine-tuning quality on real edits, which is what regime A is for.

Code and documentation are linked from the Secant repository and documentation site.

References

1. Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. MIT CSAIL, November 2, 2026. https://rikiyat.github.io/blog/weight-space-programming/
2. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024 (arXiv:2406.11717).
3. Junfeng Fang et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models. ICLR 2025 (arXiv:2410.02355).
4. Tang et al. LoRA-Null: Low-Rank Adaptation via Null Space for Large Language Models. 2025 (arXiv:2503.02659).
5. Yifeng Xiong and Xiaohui Xie. OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning. AAAI 2026.
6. Georgi Gerganov, Iwan Kawrakow (ikawrakow), and the ggml / llama.cpp contributors. GGUF v3 Specification and Q8_0 / Q4_K Super-Block Quantization Reference Kernels. https://github.com/ggml-org/llama.cpp
7. Mingjie Li et al. SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation. ICLR 2025.
8. Chao Chen et al. Revisiting Over-refusal in Safety-aligned Large Language Models: A Representation and Optimization Perspective (GOAL). October 2026, Appendix M.
9. Wenhao Yu et al. CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents. October 2026.
10. Nikhil Ghosh, Tetiana Parshakova, and Robert M. Gower. PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer. July 2026.
11. Kiwimaker. In-place Q8_0 GGUF editing and untouched-byte SHA-256 verification. October 2026.
12. SC117. GGUF tensor transplantation and scale-preserving Q2_0 re-encoding. October 2026.
13. Kevin Meng et al. Locating and Editing Factual Associations in GPT (ROME). NeurIPS 2022; Mass-Editing Memory in a Transformer (MEMIT). ICLR 2023.
14. Vincenzo Fornaro. GHSA-5xpg-vw35-2687: NaN router logits and negative-index heap OOB. August 5, 2026.
15. Hanzo AI. HIP-0113: Cognitive Sidecar & Hanzo Engine Provider Runtime for Thinking Chains. Section 4.5.