ProductsLamBootDocs › Proxmox operations

Proxmox operations

Host-side monitoring, OVMF_VARS, and the fw_cfg fleet flow. Layer 2.

LamBoot is designed as a VM-first bootloader for Proxmox, QEMU, and KVM environments. This guide covers operating LamBoot-booted guests on a Proxmox VE node: installing LamBoot inside guests, deploying Secure Boot trust via firmware db from the host, host-side boot-health monitoring, and the guest-integration layer that injects per-VM fleet metadata. It is current as of LamBoot 0.15.2 (June 2026).

Its key advantages over GRUB and systemd-boot in VM fleets:

  • GUI system info display: VM ID, hypervisor, and Secure Boot state visible at boot.
  • Host-side boot health monitoring via NVRAM variables, requiring no guest agent.
  • Automatic VMID injection via hookscript, requiring no manual config per VM.
  • Crash loop detection with automatic fallback and snapshot-aware reset.
  • Compact binary (about 650 KB on x86_64) for fast VM cold boot.
  • Pre-boot diagnostics accessible without booting the guest OS.
  • Offline repair from the Proxmox host using lamboot-repair.

LamBoot reads ext4, btrfs, and FAT natively, including on LVM, read in place, and loads the kernel itself via its native PE loader rather than relying on firmware LoadImage. LamBoot targets x86_64 and aarch64 UEFI and is licensed MIT OR Apache-2.0.


Installation in a Proxmox VM

Prerequisites

  • Proxmox VE 7.x or 8.x
  • VM configured with OVMF (UEFI) firmware (bios: ovmf in VM config)
  • EFI disk (efidisk0) present

Step 1: Install LamBoot in the Guest

# Inside the guest VM:
sudo mount /dev/vda1 /boot/efi  # or wherever your ESP is

# Copy LamBoot files
sudo mkdir -p /boot/efi/EFI/LamBoot/{drivers,modules,reports}
sudo cp lambootx64.efi /boot/efi/EFI/LamBoot/
sudo cp policy.toml /boot/efi/EFI/LamBoot/

# Create UEFI boot entry
sudo efibootmgr -c -d /dev/vda -p 1 \
    -l '\EFI\LamBoot\lambootx64.efi' -L 'LamBoot'

# (Optional) Set LamBoot as first boot option
sudo efibootmgr -o XXXX  # where XXXX is the LamBoot entry number

Step 2: Install Filesystem Drivers (Optional)

If /boot is on an ext4 or btrfs partition (common on Fedora, Arch, and similar):

sudo cp ext4_x64.efi /boot/efi/EFI/LamBoot/drivers/
sudo cp btrfs_x64.efi /boot/efi/EFI/LamBoot/drivers/  # if needed

Step 3: Install kernel-install Plugin (Optional)

For automatic BLS entry creation when kernels are installed:

sudo cp 90-lamboot.install /usr/lib/kernel/install.d/
sudo chmod +x /usr/lib/kernel/install.d/90-lamboot.install

# Create the entries directory on the ESP
sudo mkdir -p /boot/efi/loader/entries

Step 4: Configure Policy

sudo cat > /boot/efi/EFI/LamBoot/policy.toml << 'EOF'
version = 1
default_timeout_ms = 3000

[security]
crash_threshold = 3
fallback_order = ["fallback"]

[watchdog]
enabled = true
grace_seconds = 10
EOF

Deployment Options

Option A: Guest-Side Install (Recommended)

Use lamboot-install inside the guest VM:

# SSH into the VM
ssh root@vm-hostname

# Run the installer
sudo lamboot-install --with-modules

This is the standard approach: the installer handles ESP detection, driver selection, BLS entry generation, and systemd integration automatically.

Option B: Host-Side Deploy via qemu-nbd

For VMs that cannot boot or for batch deployment:

# On the Proxmox host:
qm stop VMID
lvchange -ay pve/vm-VMID-disk-N
qemu-nbd --connect=/dev/nbd0 -f raw /dev/pve/vm-VMID-disk-N
mount /dev/nbd0p1 /tmp/esp

# Copy LamBoot files
cp -r dist/EFI/LamBoot /tmp/esp/EFI/

umount /tmp/esp
qemu-nbd -d /dev/nbd0
qm start VMID

This method bypasses the installer (no BLS entry generation, no systemd integration). It is suitable for deploying updated binaries to existing installations.

Option C: Offline Repair

If a VM will not boot, use lamboot-repair from the host:

sudo lamboot-repair --offline /dev/pve/vm-VMID-disk-N

See LamBoot Tools for details.

VM Identity and Fleet Management

Displaying VMID on the Boot Screen

LamBoot can display the Proxmox VMID and fleet tags on the boot screen and in boot reports. This is configured via SMBIOS OEM string injection in the VM config.

Step 1: Add SMBIOS OEM strings to the VM config

On the Proxmox host, edit /etc/pve/qemu-server/<vmid>.conf:

args: -smbios type=11,value=lamboot.vmid=201

Or for fleet management with multiple tags:

args: -smbios type=11,value=lamboot.vmid=201,value=lamboot.fleet-id=prod-cluster-01,value=lamboot.role=webserver

Step 2: Verify in LamBoot

After rebooting the VM, LamBoot reads these OEM strings automatically:

  • The VMID appears in the boot report (/EFI/LamBoot/reports/boot.json)
  • The fleet-id is logged at boot and included in the report
  • Serial console shows: SMBIOS OEM strings: N

What the boot report includes:

{
  "lamboot_version": "0.15.2",
  "vmid": "201",
  "fleet_id": "prod-cluster-01",
  "hypervisor": "KVM",
  "system_manufacturer": "QEMU",
  "system_product": "Standard PC (Q35 + ICH9, 2009)",
  "iommu": "Intel VT-d",
  "iommu_units": 2,
  "boot_timing_ms": {"health":5,"drivers":120,"discovery":45,"total":210}
}

Available OEM string keys:

KeyPurposeExample
lamboot.vmidProxmox VM ID201
lamboot.fleet-idFleet/cluster identifierprod-cluster-01
lamboot.roleVM role tagwebserver
lamboot.monitorMonitoring endpoint URLhttp://10.0.0.1:9090

For VM templates: Add the args: line to the template config. Override per-clone by editing the cloned VM's config with the correct VMID.

Hypervisor Detection

LamBoot automatically detects the hypervisor via CPUID:

  • KVM (Proxmox/QEMU): detected as "KVM"
  • Hyper-V, VMware, Xen, VirtualBox: all detected
  • Bare metal: reported as "not detected", handled gracefully

This appears in the boot report hypervisor field and serial console output.

IOMMU Group Detection

LamBoot parses ACPI DMAR (Intel VT-d) or IVRS (AMD-Vi) tables to detect IOMMU hardware units. This is valuable for PCI passthrough troubleshooting: you can see IOMMU group assignments before Linux boots.

The boot report includes iommu (type) and iommu_units (count of DRHD/IVHD blocks).

VM Template Workflow

Creating a LamBoot-Ready Template

  1. Create a new VM in Proxmox with OVMF firmware
  2. Install your base OS (Fedora, Ubuntu, Debian, and similar)
  3. Install LamBoot (steps above)
  4. Install the kernel-install plugin
  5. Test: reboot and verify LamBoot shows the boot menu
  6. Convert to template in Proxmox

Cloning from Template

VMs cloned from the template automatically get LamBoot. Each clone:

  • Gets a new OVMF_VARS.fd (fresh NVRAM, state=Fresh)
  • Has LamBoot on the ESP
  • Auto-detects the installed OS via BLS entries

Host-Side Monitoring

Setup on Proxmox Host

# Copy the monitor script to the Proxmox host
scp tools/lamboot-monitor.py root@proxmox:/usr/local/bin/

# Make executable
chmod +x /usr/local/bin/lamboot-monitor.py

# Test it
lamboot-monitor.py

Automated Monitoring

Create a systemd timer for periodic checks:

# /etc/systemd/system/lamboot-monitor.timer
cat > /etc/systemd/system/lamboot-monitor.timer << 'EOF'
[Unit]
Description=LamBoot VM boot health check

[Timer]
OnBootSec=60
OnUnitActiveSec=300

[Install]
WantedBy=timers.target
EOF

# /etc/systemd/system/lamboot-monitor.service
cat > /etc/systemd/system/lamboot-monitor.service << 'EOF'
[Unit]
Description=LamBoot VM boot health monitor

[Service]
Type=oneshot
ExecStart=/usr/local/bin/lamboot-monitor.py --alert-webhook http://your-webhook-url
EOF

systemctl daemon-reload
systemctl enable --now lamboot-monitor.timer

Webhook Integration

The monitor sends JSON alerts to any webhook endpoint:

{
  "alert": "lamboot-crash-loop",
  "timestamp": "2026-03-27T12:00:00",
  "vms": [
    {
      "vmid": 102,
      "name": "broken-vm",
      "state": "CrashLoop",
      "crash_count": 5,
      "last_entry": "fedora-6.12.0",
      "status": "critical"
    }
  ]
}

Compatible with: Slack incoming webhooks, PagerDuty, Grafana OnCall, ntfy.sh, and custom HTTP endpoints.

How NVRAM Monitoring Works

[Guest VM]                              [Proxmox Host]
LamBoot writes UEFI NVRAM vars   ->   OVMF stores to efidisk0
  - LamBootState                        (qcow2 or raw image)
  - LamBootCrashCount                          |
  - LamBootLastEntry                           v
  - LamBootTimestamp              lamboot-monitor.py reads the
                                  efidisk image file directly
                                  (no qemu-nbd needed for raw
                                   OVMF_VARS.fd files)
                                           |
                                           v
                                  Searches for LAMBOOT vendor
                                  GUID bytes in the variable
                                  store, extracts values
                                           |
                                           v
                                  Assesses health:
                                    OK = BootedOK, count=0
                                    WARN = Booting (recent)
                                    CRIT = CrashLoop or high count
                                           |
                                           v
                                  Alert via webhook / JSON / table

The monitoring requires no guest agent and is completely non-intrusive.

Hookscript: Automated VMID and Monitoring

The LamBoot hookscript integrates with Proxmox's VM lifecycle to automatically inject VMID into SMBIOS OEM strings and capture boot health data.

Installation

# On the Proxmox host:
cp lamboot-hookscript.pl /var/lib/vz/snippets/
chmod +x /var/lib/vz/snippets/lamboot-hookscript.pl

# Enable for a VM:
qm set 201 --hookscript local:snippets/lamboot-hookscript.pl

What It Does

PhaseAction
pre-startAuto-injects lamboot.vmid=VMID into the VM's SMBIOS OEM strings (if not already set). Checks previous boot health and logs warnings for crash loop state.
post-startLogs VM start event.
post-stopCaptures boot health from OVMF NVRAM to fleet log. Calls lamboot-monitor.py for health assessment.

Automatic VMID Injection

On every VM start, the hookscript checks if lamboot.vmid= is in the VM's args: config line. If not, it adds:

args: -smbios type=11,value=lamboot.vmid=201

This means:

  • No manual qm set --args needed: the hookscript handles it
  • Works with cloned VMs: each clone gets its correct VMID automatically
  • Non-destructive: existing args are preserved, VMID is appended

Fleet Log

The hookscript appends health assessments to /var/log/lamboot/fleet.jsonl (one JSON record per line). This provides a historical record of boot health across all VMs with the hookscript enabled.

GUI System Information

When running in a Proxmox VM with VMID configured, the LamBoot boot menu header shows:

  • VM ID (large text, top right): for example, VM 201
  • Hypervisor: auto-detected via CPUID (shows KVM for Proxmox)
  • Build info: Secure Boot state, filesystem driver count

This information is gathered automatically: the VMID from SMBIOS OEM strings (injected by the hookscript), the hypervisor from CPUID, and Secure Boot from UEFI variables.


OVMF VARS Deployment for Secure Boot (Config 4: direct, no shim)

Audience: Proxmox VE operators deploying LamBoot across VM fleets without touching guests' shim or MOK.

Config: Config 4 from the Secure Boot deployment guide.

Outcome: Secure-Boot-enabled VMs that trust LamBoot directly via firmware db, with no shim chain, no MokManager dance, and no guest interaction.

What this is

OVMF_VARS_lamboot.fd is a 540 KB binary: a UEFI variable store in the 4MB OVMF format, with these keys pre-populated in firmware NVRAM:

  • PK: Microsoft's Platform Key (preserved from stock Debian OVMF)
  • KEK: Microsoft's and Debian's Key Exchange Keys
  • db: Microsoft UEFI CA 2011, Microsoft Windows Production PCA 2011, plus LamBoot's signing cert

Because LamBoot's cert is in db, the firmware validates LamBoot binaries directly. No shim. No MOK. Microsoft keys are retained so Windows guests and distro shims continue to work, so this file is safe to use for any guest, not just Linux.

Everything Windows and shim need to boot is untouched. The only behavioural change from stock OVMF VARS is that LamBoot now boots. LamBoot's signing applies from v0.10.0, which was the first GPG-signed release.

Common pitfall: --add-mok alone is not enough

Operators rolling their own VARS file with virt-fw-vars (instead of using the pre-built OVMF_VARS_lamboot.fd) sometimes enroll LamBoot's cert into the MOK list with --add-mok, expecting that to be sufficient. For direct boot it is insufficient:

  • MOK (Machine Owner Key) is consulted by shim, not by the firmware. If LamBoot is launched directly (no shim in the chain), the firmware sees no db entry that signs LamBoot, rejects the binary, and falls back to whatever other Boot#### entries exist (typically the distro's shim).
  • db is consulted by the firmware itself. A cert in db makes LamBoot directly bootable under SB with no shim wrapper.

The two paths that work under SB:

PathCert locationTool flag
Config 3: chained behind shimMOKvirt-fw-vars --add-mok (or mokutil --import from inside the guest)
Config 4: direct, no shimfirmware dbvirt-fw-vars --add-db (this document; OVMF_VARS_lamboot.fd does this for you)

Mixing the two (for example enrolling to MOK only and expecting LamBoot to boot directly) silently fails by falling back to a different Boot#### entry. Diagnose with efibootmgr post-boot: BootCurrent will not match LamBoot's Boot####.

This pitfall was observed during early sprint testing on a Fedora workstation: virt-fw-vars --add-mok alone left LamBoot un-bootable directly under SB; adding --add-db for the same cert fixed it.

When to use this

Choose Config 4 (this document) over Config 3 (shim + MOK) when:

  • You are deploying LamBoot across many VMs and want zero per-guest interaction
  • You control the Proxmox host
  • You want boot to "just work" with no operator steps inside the guest

Choose Config 3 instead when:

  • The VM runs on a hypervisor you do not control (bare metal, cloud, and similar)
  • You want to validate the same trust path a distro end-user would experience

Both configs require a signed LamBoot binary; the difference is in how the firmware trusts it.

Prerequisites

On the Proxmox host:

  • Proxmox VE 7.x or 8.x (tested on 8.x)
  • Root shell access
  • The target VM exists and is shut down
  • Target VM uses bios: ovmf and has an efidisk0 (Secure Boot must be configured in advance if you want SB enforcement)

Files you need on the Proxmox host:

  • OVMF_VARS_lamboot.fd from your LamBoot release tarball (or dist/OVMF_VARS_lamboot.fd from the dev tree)

Copy it onto the Proxmox node:

scp dist/OVMF_VARS_lamboot.fd root@pve:/var/lib/vz/snippets/

/var/lib/vz/snippets/ is a convenient location that exists by default on every Proxmox install; any other directory works.

Identifying the target VM's efidisk

From the Proxmox host:

qm config <VMID> | grep -E 'bios|efidisk'

Expected output:

bios: ovmf
efidisk0: <storage>:<volume>,efitype=4m,pre-enrolled-keys=1,size=1M

Three things to check:

  1. bios: ovmf: must be present. If bios: seabios, this VM uses BIOS firmware and Secure Boot does not apply; use Config 1 (unsigned install) instead.
  2. efitype=4m: must be 4m, not the old 64k format. The OVMF_VARS_lamboot.fd file is 4MB format only.
  3. <storage>:<volume>: tells you which Proxmox storage backend holds the efidisk. This drives the next section.

If efitype=4m is not set, the VM was created with an older OVMF format. Recreate the efidisk:

qm shutdown <VMID>
qm set <VMID> --delete efidisk0
qm set <VMID> --efidisk0 <storage>:1,efitype=4m,pre-enrolled-keys=0

Writing the VARS file by storage backend

The efidisk0 is a 528 KB volume stored differently depending on <storage>'s backend type. pvesm status shows each storage's type.

ZFS-backed storage (zfspool)

Efidisk is a ZFS volume (zvol) exposed as a block device at /dev/zvol/<pool>/vm-<VMID>-disk-N.

qm shutdown <VMID>                                 # make sure VM is off
zfs list -t volume | grep vm-<VMID>-disk           # find the zvol (look for ~528K size)

# Write the VARS file over the zvol. The zvol is already sized for OVMF VARS,
# so dd truncates our 540K input safely if needed.
dd if=/var/lib/vz/snippets/OVMF_VARS_lamboot.fd \
   of=/dev/zvol/<pool>/vm-<VMID>-disk-N \
   bs=1M conv=notrunc status=progress
sync

qm start <VMID>

LVM / LVM-thin storage (lvm, lvmthin)

Efidisk is a logical volume at /dev/<vg>/vm-<VMID>-disk-N.

qm shutdown <VMID>
lvs | grep vm-<VMID>-disk                          # find the efidisk LV

dd if=/var/lib/vz/snippets/OVMF_VARS_lamboot.fd \
   of=/dev/<vg>/vm-<VMID>-disk-N \
   bs=1M conv=notrunc status=progress
sync

qm start <VMID>

Directory storage (dir)

Efidisk is a raw file at /var/lib/vz/images/<VMID>/vm-<VMID>-disk-N.raw.

qm shutdown <VMID>
ls -la /var/lib/vz/images/<VMID>/                  # confirm the file exists

cp /var/lib/vz/snippets/OVMF_VARS_lamboot.fd \
   /var/lib/vz/images/<VMID>/vm-<VMID>-disk-N.raw
sync

qm start <VMID>

Ceph RBD storage (rbd)

Efidisk is an RBD image in a Ceph pool.

qm shutdown <VMID>
rbd -p <pool> ls | grep vm-<VMID>-disk             # find the efidisk image

rbd import --image-format 2 --dest-pool <pool> \
    /var/lib/vz/snippets/OVMF_VARS_lamboot.fd \
    vm-<VMID>-disk-efi-tmp

# Replace in-place: delete the old image (after confirming), rename new one
rbd -p <pool> rm vm-<VMID>-disk-N
rbd -p <pool> mv vm-<VMID>-disk-efi-tmp vm-<VMID>-disk-N

qm start <VMID>

(Ceph operators may prefer rbd import --image-format 2 followed by an update to /etc/pve/qemu-server/<VMID>.conf to reference the new image name. Use whichever workflow fits your backup cadence.)

Other backends

For storage types not listed (iSCSI, NFS, ZFS-over-iSCSI, and similar), the general pattern is:

  1. qm shutdown <VMID>
  2. Determine how vm-<VMID>-disk-N is exposed on the host filesystem
  3. Write OVMF_VARS_lamboot.fd byte-for-byte over that backing storage with dd, cp, or the backend-native tool
  4. qm start <VMID>

The Proxmox wiki's Storage: Raw Files page lists how each backend names and exposes volumes.

Alternative: in-place modification (preserve existing VARS state)

The previous section replaces the entire efidisk0 with the pre-built OVMF_VARS_lamboot.fd template. That is the right approach for fleet deployment where every VM should have identical canonical VARS.

For an already-running individual VM where you want to keep its existing UEFI state (BootOrder, existing Boot#### entries such as Debian's \EFI\debian\shimx64.efi, enrolled MOK entries like DKMS module signing keys and NVIDIA Module Signing, customized PK/KEK) use in-place modification instead: read the existing VARS off the efidisk, append LamBoot's cert to db with virt-fw-vars --inplace --add-db, write it back.

When to use this path vs template replace

Template replaceIn-place modify
Fleet deployment from scratch
Preserves per-VM BootOrder, Boot#### entries
Preserves enrolled MOK (DKMS, NVIDIA module signing, …)
Preserves customized PK/KEK (e.g., Debian's PK/KEK)
Requires virt-firmware on Proxmox host(only if generating template)✓ (always)
Snapshot/rollback safetytied to template integrityper-modification ZFS snapshot
Idempotent (re-running is a no-op if cert already in db)byte-identical writesyes, with verification helper

Rule of thumb: use template replace if you have a fleet template you trust; use in-place modify if the VM has been running for a while and accumulated state worth keeping.

The procedure (ZFS-backed example)

VMID=108
ZFS_DS="MonsterStore/vm-${VMID}-disk-1"          # adjust to your pool/disk
ZVOL="/dev/zvol/${ZFS_DS}"
CERT="/path/to/keys/db.der"                       # LamBoot signing cert
GUID="4c414d42-4f4f-5400-0000-000000000001"       # LamBoot vendor GUID

qm shutdown "$VMID"

# 1. ZFS snapshot for rollback safety (instant on ZFS)
zfs snapshot "${ZFS_DS}@pre-lamboot-inject-$(date +%Y%m%d-%H%M%S)"

# 2. Read existing VARS off the zvol
SIZE=$(blockdev --getsize64 "$ZVOL")              # 4194304 for 4M efidisk
COUNT_MB=$(( SIZE / 1048576 ))
dd if="$ZVOL" of=/tmp/vars.bak bs=1M count="$COUNT_MB" status=none
cp /tmp/vars.bak /tmp/vars.new

# 3. Append LamBoot cert to db (preserves all existing variables)
virt-fw-vars --inplace /tmp/vars.new --add-db "$GUID" "$CERT"

# 4. VERIFY before writing back (see verification section)
# ...

# 5. Write modified VARS back to the zvol
dd if=/tmp/vars.new of="$ZVOL" bs=1M count="$COUNT_MB" conv=notrunc status=none
sync

qm start "$VMID"

Rollback: zfs rollback ${ZFS_DS}@pre-lamboot-inject-<timestamp> while VM is shut down.

For non-ZFS storage backends, replace step 1's zfs snapshot with the equivalent (LVM snapshot, cp for directory storage, RBD snapshot, and similar) and steps 2 and 5's dd with the appropriate read/write mechanism for that backend.

Verification: virt-fw-vars --print does not show cert subjects

A common verification mistake is piping virt-fw-vars --print through grep "LamBoot Release Signing Key" after --add-db. This will fail even when the cert is correctly added, because --print outputs structural information (variable names, blob sizes) but never decodes X.509 cert subjects:

db                  : blob: 4225 bytes      # ← size only, no subject string

The right verification is --extract-certs followed by openssl x509:

verify_lamboot_cert_in_db() {
    local vars_file="$1"
    local expected_sha256="$2"     # e.g. "513a22b6f16a5a13aeebb8da1bfb3e96..."
    local guid="$3"                # e.g. "4c414d42-4f4f-5400-0000-000000000001"

    local tmpdir
    tmpdir=$(mktemp -d /tmp/lamboot-verify-XXXXXX)

    ( cd "$tmpdir" && virt-fw-vars --input "$vars_file" --extract-certs >/dev/null 2>&1 ) \
        || { rm -rf "$tmpdir"; return 1; }

    local cert_file
    cert_file=$(find "$tmpdir" -maxdepth 1 -name "db-${guid}-*.pem" -print -quit)
    [[ -f "$cert_file" ]] || { rm -rf "$tmpdir"; return 1; }

    local fp
    fp=$(openssl x509 -in "$cert_file" -noout -fingerprint -sha256 \
         | sed 's/^.*Fingerprint=//' | tr -d ':' | tr '[:upper:]' '[:lower:]')

    rm -rf "$tmpdir"
    [[ "$fp" == "$expected_sha256" ]]
}

This asserts on bytes, not human-readable substrings, which is robust against virt-fw-vars output format changes and against subject-string typos.

A secondary sanity check is to verify the db blob grew by the expected amount:

virt-fw-vars --input /tmp/vars.bak --print | grep '^db '
# db                  : blob: 3143 bytes
virt-fw-vars --input /tmp/vars.new --print | grep '^db '
# db                  : blob: 4225 bytes

Delta = 1082 bytes for a typical 1KB DER cert plus the 44-byte EFI_SIGNATURE_LIST and owner GUID overhead. A delta of zero means --add-db silently no-op'd; a non-1KB delta means something other than a cert went in.

1MB efidisk anomaly on older Proxmox VMs

Some VMs created on older Proxmox versions, or restored from older backups, have efitype=4m in their config but a 1MB-allocated efidisk on disk:

efidisk0: <storage>:vm-N-disk-X,efitype=4m,pre-enrolled-keys=1,size=1M

The OVMF VARS structure (~540KB) fits in 1MB, so the VM still boots fine, but there is almost no slack for additional variable storage. Adding LamBoot's cert (≈1KB) is comfortable, but adding several certs, or larger BootOrder/MOK growth, can run out of space.

Best practice: expand to 4MB before in-place injection.

# ZFS-backed example:
qm shutdown "$VMID"
zfs snapshot "${ZFS_DS}@pre-lamboot-expand-$(date +%Y%m%d-%H%M%S)"
zfs set volsize=4M "$ZFS_DS"
udevadm settle
qm set "$VMID" --efidisk0 "<storage>:vm-${VMID}-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M"
# Then proceed with the in-place procedure.

The existing ~540KB of OVMF VARS content stays at the start of the now-4MB zvol; the new 3.5MB is zero-padded. Pre-enrolled keys, BootOrder, and existing Boot#### entries are all preserved.

Use template replacement only on a VM you are willing to reset. Replacing the efidisk with a fresh OVMF_VARS_4M.ms.fd wipes BootOrder and all Boot#### entries; the VM may no longer boot until you recreate the distro boot entry via grub-install from a rescue environment. For a VM whose state you want to preserve, use in-place modification.

Live data: aibox (Debian 13, ZFS, ext2 /boot)

End-to-end exercise of in-place modification plus Config 3 with db pre-enrolled. VM 108 on a Proxmox host, ZFS-backed efidisk, Debian 13 trixie guest with /boot=ext2.

Pre-install state:

  • efidisk0: AB:vm-108-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M (1MB allocated, the efidisk anomaly above)
  • ZFS zvol MonsterStore/vm-108-disk-1, blockdev size 1048576
  • Existing UEFI state non-trivial: Debian PK (not Microsoft's), MS UEFI CA + MS PCA + Debian's KEK in db, MokList already enrolled with DKMS module signing key + NVIDIA Module Signing (per --extract-certs on a backup of the pre-modification VARS)

Expand step:

  • zfs set volsize=4M MonsterStore/vm-108-disk-1 → new size 4194304
  • qm set 108 --efidisk0 AB:vm-108-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
  • All existing variables, BootOrder, and MokList preserved (verified by --extract-certs before and after)

Inject step:

  • dd zvol → vars.bak (4194304 bytes)
  • virt-fw-vars --inplace vars.new --add-db 4c414d42-4f4f-5400-0000-000000000001 db.der
  • db blob grew 3143 → 4225 bytes (delta 1082 = expected for 1038-byte DER cert + EFI_SIGNATURE_LIST overhead)
  • Verification by --extract-certs produced db-4c414d42-4f4f-5400-0000-000000000001-LamBootReleaseSigningKey2026.pem with sha256 fingerprint matching expected 51:3A:22:B6:F1:6A:5A:13:…:14:2
  • dd vars.new → zvol; round-trip re-read confirms cert on disk

Install step (lamboot-install --signed, in-guest):

  • Auto-detected shim at /boot/efi/EFI/debian/shimx64.efi; deployed Config 3 layout
  • /boot/efi/EFI/LamBoot/ contains: shimx64.efi (957KB Debian shim), grubx64.efi (LamBoot signed binary), lambootx64.efi (also LamBoot signed; same sha256), policy.toml, db.der, drivers/, modules/, reports/
  • 5 BLS entries generated for kernels 6.12.{69,73,74+1,85,88}+deb13-amd64
  • efibootmgr --create made Boot0001* LamBoot → \EFI\LamBoot\shimx64.efi; inserted at head of BootOrder by default

Observed first boot:

  • BootCurrent: 0001: firmware loaded LamBoot's shim, not Debian's shim
  • LamBoot ran, discovered 5 entries in 1000 ms, selected bls-debian-6.12.88
  • Kernel 6.12.88+deb13-amd64 booted normally; Debian came up clean
  • boot.json reports entry_type: linux_legacy: kernel loaded via firmware LoadImage on this build

This run confirms the in-place modification plus pre-enrolled-db hybrid path on Debian 13 with SB, a ZFS efidisk, and an ext2 /boot, populating a previously-blank cell in the test matrix.

After swapping VARS

Boot the VM. From the guest:

mokutil --sb-state
# expected: SecureBoot enabled

# LamBoot's cert should now appear in firmware db:
sudo apt install efitools  # or equivalent
sudo efi-readvar -v db | grep -A2 'LamBoot'
# expected: Subject: C=US, ST=IL, O=Lamco Development, OU=LamBoot,
#           CN=LamBoot Release Signing Key 2026

Now install LamBoot with the direct-boot path (no shim, no MOK):

sudo lamboot-install --signed --no-shim

Reboot. LamBoot loads directly: firmware validates its signature against the db entry you installed, hands off, and the LamBoot splash appears.

Regenerating OVMF_VARS_lamboot.fd with production keys

The release tarball ships OVMF_VARS_lamboot.fd pre-built. If you need to rebuild it (fleet key rotation, test builds, custom cert composition), use tools/build-ovmf-vars.sh:

# Requires: pip install virt-firmware (or a venv with it)
# Requires: /usr/share/OVMF/OVMF_VARS_4M.ms.fd from the ovmf package

cd ~/lamboot-dev
./tools/build-ovmf-vars.sh --cert keys/db.crt --output dist/OVMF_VARS_lamboot.fd

The script takes the stock Microsoft-enrolled Debian OVMF VARS template and appends LamBoot's cert to the db variable. Microsoft keys are preserved.

If you are scripting around virt-fw-vars directly instead of using build-ovmf-vars.sh, use --add-db for direct LamBoot boot under SB. --add-mok alone is not sufficient for direct boot; see the common pitfall section for why.

Note on key rotation: when the LamBoot db key rotates (planned 2029), OVMF_VARS_lamboot.fd must be regenerated and re-deployed to every Config 4 VM. Plan for a maintenance window or roll out alongside existing update workflows.

Rollback

If LamBoot fails to boot and you need the VM back on its original bootloader:

qm shutdown <VMID>

# Restore stock Proxmox OVMF VARS (Microsoft-only db):
qm set <VMID> --delete efidisk0
qm set <VMID> --efidisk0 <storage>:1,efitype=4m,pre-enrolled-keys=1

qm start <VMID>

Stock Proxmox OVMF VARS has Microsoft keys and no LamBoot cert, so Windows and shim-based guests continue to boot normally. LamBoot binaries are rejected; the \EFI\LamBoot\ tree can then be cleaned up from inside the guest with lamboot-install --remove.

Fleet automation

For large deployments, wrap the template-replace procedure in a helper script:

#!/bin/bash
# deploy-lamboot-vars.sh — swap a VM's efidisk to OVMF_VARS_lamboot.fd
set -e
VMID="$1"
VARS="/var/lib/vz/snippets/OVMF_VARS_lamboot.fd"
[ -f "$VARS" ] || { echo "$VARS not found"; exit 1; }
[ -n "$VMID" ] || { echo "Usage: $0 <VMID>"; exit 1; }

qm shutdown "$VMID"

# Locate and identify efidisk storage — adapt to your backend
EFIDISK=$(qm config "$VMID" | awk -F: '/^efidisk0:/ {print $2}' | cut -d, -f1)
# ... (add per-backend write logic here; pattern from the storage-backend section)

qm start "$VMID"

Iterate across a VMID list to deploy the fleet. Expect each VM's efidisk swap to take a few seconds.

libvirt / virt-manager (brief)

The same OVMF_VARS_lamboot.fd works for libvirt. Edit the domain XML:

<os firmware='efi'>
  <nvram template='/usr/share/OVMF/OVMF_VARS_4M.ms.fd'>/var/lib/libvirt/qemu/nvram/<domain>_VARS.fd</nvram>
  <firmware>
    <feature enabled='yes' name='secure-boot'/>
    <feature enabled='yes' name='enrolled-keys'/>
  </firmware>
</os>

Replace the <nvram> target with OVMF_VARS_lamboot.fd:

virsh destroy <domain>
cp /path/to/OVMF_VARS_lamboot.fd /var/lib/libvirt/qemu/nvram/<domain>_VARS.fd
virsh start <domain>

Full libvirt coverage is beyond the scope of this document.

Operator tooling: lamboot-pve-ovmf-vars

For Proxmox host operators who prefer a subcommand-style UX over invoking build-ovmf-vars.sh directly, the companion toolkit ships lamboot-pve-ovmf-vars in the lamboot-toolkit-pve RPM subpackage.

lamboot-pve-ovmf-vars is a mirror of tools/build-ovmf-vars.sh in the LamBoot repo: canonical source stays in the repo; the mirror is regenerated at toolkit release-build time. Edit tools/build-ovmf-vars.sh in the repo, then re-run the toolkit's mirror script, rather than editing the mirror directly.

Subcommands (from the mirror):

  • build: build OVMF_VARS_lamboot.fd with LamBoot's db cert pre-enrolled
  • verify: inspect an existing VARS file for the expected enrollment
  • show: print enrolled keys in human-readable form

The tool exists for discoverability in the broader lamboot-toolkit-pve suite. Operators already running tools/build-ovmf-vars.sh from the release tarball need not change anything. Tool-level documentation lives with the lamboot-tools repository (the OVMF-VARS helper is a sub-component of the PVE setup tooling).


Guest-Integration Layer (fw_cfg metadata and boot-health capture)

This layer is the host-side software on a Proxmox VE node that interacts with LamBoot-running guest VMs. It is distinct from and orthogonal to installing LamBoot as the host node's own bootloader. The hookscript and monitor described here are version-pinned host-side artifacts; the surrounding LamBoot release is 0.15.2.

What this layer does:

  1. Injects per-VM, per-fleet metadata into each guest at VM start via QEMU's fw_cfg interface, so LamBoot inside the guest can read its identity (VMID, fleet ID, role) without any agent inside the guest.
  2. Captures boot-health data from each guest's UEFI variables at VM stop, so the host has a fleet-wide rolling record of LamBoot's self-reported LamBootState, LamBootCrashCount, LamBootLastEntry, LamBootTimestamp, and LamBootVersion.
  3. Surfaces crash-loop detection at the host log layer so an operator can act on a CrashLoop guest before the next boot.

This layer is purely host-side. It is not a LamBoot-on-host install, and it requires no guest OS agent (the guest does not need to know this layer exists). Each Proxmox node runs its own instance, reading its own /etc/lamboot/fleet.toml and watching only the VMs hosted on that node.

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│ Proxmox host (e.g. pve2)                                             │
│                                                                      │
│  /etc/lamboot/fleet.toml         ← operator-edited; v1 schema        │
│           │                                                          │
│           ▼                                                          │
│  /var/lib/vz/snippets/                                               │
│  lamboot-hookscript.pl           ← invoked by qm at each lifecycle   │
│           │                                                          │
│   pre-start → reads fleet.toml, reads /etc/pve/qemu-server/<vmid>    │
│               .conf, writes /var/lib/lamboot/<vmid>.json,            │
│               appends snapshot to fleet.jsonl                        │
│   post-stop → invokes lamboot-monitor.py on the just-stopped VM,     │
│               captures boot health from OVMF_VARS, appends to        │
│               fleet.jsonl                                            │
│           │                                                          │
│           ▼                                                          │
│  /var/lib/lamboot/<vmid>.json    ← exposed to guest via fw_cfg       │
│  /var/log/lamboot/hookscript.log ← human-readable audit trail        │
│  /var/log/lamboot/fleet.jsonl    ← structured rolling boot-health log│
└────────────────────────────────┬─────────────────────────────────────┘
                                 │
                                 │ qm starts the VM, QEMU exposes:
                                 │   -fw_cfg name=opt/lamboot/config,
                                 │           file=/var/lib/lamboot/<vmid>.json
                                 ▼
┌──────────────────────────────────────────────────────────────────────┐
│ Guest VM (e.g. 364)                                                  │
│                                                                      │
│  LamBoot reads fw_cfg blob at startup                                │
│   → identifies its own VMID + fleet ID + role                        │
│   → renders these in the GUI menu header                             │
│   → logs them in /boot/EFI/LamBoot/reports/boot.json                 │
│                                                                      │
│  LamBoot also writes to its UEFI variables (vendor GUID              │
│  4c414d42-4f4f-5400-0000-000000000001):                              │
│    LamBootState         (u8: 0=Fresh, 1=Booting, 2=BootedOK, 3=Crash)│
│    LamBootCrashCount    (u8)                                         │
│    LamBootLastEntry     (UTF-8)                                      │
│    LamBootTimestamp     (epoch seconds)                              │
│    LamBootVersion       (u32 packed)                                 │
│                                                                      │
│  (Persisted in the VM's efidisk0 OVMF_VARS file across reboots.)     │
└──────────────────────────────────────────────────────────────────────┘

Two host-side processes do real work; one is a Perl hookscript driven by Proxmox's lifecycle hooks, the other is a Python boot-health reader called from inside the hookscript. Everything else is plain config files and append-only logs.

Components inventory

File on hostSource in dev repoModePurpose
/var/lib/vz/snippets/lamboot-hookscript.pltools/lamboot-hookscript.pl0755Proxmox --hookscript target; called as <vmid> <phase>
/usr/local/bin/lamboot-monitor.pytools/lamboot-monitor.py0755Reads OVMF_VARS, emits JSON boot-health record per VM
/etc/lamboot/fleet.toml(operator-authored, schema v1)0644Fleet identity + per-VM role assignment + per-field injection toggles
/var/lib/lamboot/(created at install)0755 dirPer-VM JSON state files + any staged certs/markers
/var/lib/lamboot/<vmid>.json(rewritten on every pre-start)0644Output to guest via fw_cfg
/var/log/lamboot/(created at install)0755 dirLogs
/var/log/lamboot/hookscript.log(appended by hookscript)0644Human-readable timestamped lifecycle audit
/var/log/lamboot/fleet.jsonl(appended by hookscript + monitor)0644Structured boot-health records, one JSON object per line

This layer ships no systemd units and runs no daemons. The hookscript runs only when Proxmox invokes it; the monitor runs only when the hookscript calls it.

Dependencies installed by Proxmox by default that the layer relies on: perl >= 5.30, python3 >= 3.11 (for tomllib; on older Proxmox where 3.11+ is not default, tomli is the fallback), qemu-nbd (used by lamboot-monitor.py to mount OVMF_VARS images), and pvesm (Proxmox storage tool, for resolving efidisk0 storage spec to a file path).

Installation procedure

Idempotent. Re-running these steps overwrites in place; no _old.bak files accumulate.

Copy the host-side files

# As root on the Proxmox node:
install -Dm0755 lamboot-hookscript.pl /var/lib/vz/snippets/lamboot-hookscript.pl
install -Dm0755 lamboot-monitor.py    /usr/local/bin/lamboot-monitor.py
install -d -m0755 /etc/lamboot /var/lib/lamboot /var/log/lamboot

The hookscript MUST live under /var/lib/vz/snippets/ for Proxmox to resolve local:snippets/lamboot-hookscript.pl. Storing it elsewhere breaks qm set --hookscript ... lookup.

Verify prerequisites

perl -c /var/lib/vz/snippets/lamboot-hookscript.pl   # → "syntax OK"
python3 -c "import tomllib"                          # → no error on 3.11+
python3 -m py_compile /usr/local/bin/lamboot-monitor.py

If python3 -c "import tomllib" fails (older Python on Proxmox 7.x), install the python3-tomli apt package; the hookscript automatically falls back.

Author /etc/lamboot/fleet.toml

See the schema section for the full schema. The minimum useful config is:

schema = "v1"

[fleet]
id = "lamco-pve2"
description = "pve2.a.lamco.io test fleet"

[roles]
"364" = "lamboot-archinstall-target"

Attach the hookscript and fw_cfg args to each LamBoot guest

For each guest VMID <N> where LamBoot is the bootloader:

qm set <N> --hookscript local:snippets/lamboot-hookscript.pl
qm set <N> --args "-fw_cfg name=opt/lamboot/config,file=/var/lib/lamboot/<N>.json"

qm set --args is append-only-by-the-operator: re-running with a different value replaces the whole args line. If the VM already had other QEMU CLI args, snapshot the existing value first and concatenate:

EXISTING=$(qm config <N> | sed -n 's/^args: //p')
FWCFG="-fw_cfg name=opt/lamboot/config,file=/var/lib/lamboot/<N>.json"
qm set <N> --args "${EXISTING:+${EXISTING} }${FWCFG}"

Important: qm set --args takes effect on the next start of the VM, because QEMU receives its command line at process exec time. A running VM keeps its existing -fw_cfg (or lack of one) until cold-stopped and restarted; qm reboot is not enough, since it reboots the guest OS inside the same QEMU process. Use qm stop <N> && qm start <N> to pick up new args.

Optionally seed an initial JSON state file

Without this, the first VM start that picks up the new args will have /var/lib/lamboot/<N>.json written by the hookscript's pre-start 1 second before QEMU's fw_cfg open call. That is fine in practice but is visible-to-an-impatient-debugger as a brief race. To eliminate it:

/var/lib/vz/snippets/lamboot-hookscript.pl <N> pre-start

This is the same code path Proxmox triggers; running it manually is idempotent and writes the JSON exactly as the next real pre-start would.

/etc/lamboot/fleet.toml schema v1

Authoritative source: the parser in lamboot-hookscript.pl (lines ~165 to 207 for the loader, ~230 to 254 for role determination). The Python side (lamboot-monitor.py) reads the same file but only uses [fleet].id.

Top-level

schema = "v1"             # REQUIRED. Hookscript silently returns {} if mismatched.
                          # Allowed values: "v1" (string) OR omitted entirely.
                          # If you set schema = "v2" or anything else, the
                          # hookscript treats the file as empty.

[fleet]: fleet identity (REQUIRED to populate fleet_id in JSON)

[fleet]
id          = "lamco-pve2"            # Free-form; surfaces in guest JSON + logs
description = "pve2 test fleet"       # Optional; not surfaced to guest

[roles]: explicit VMID to role-name map

[roles]
"364" = "lamboot-archinstall-target"
"365" = "lamboot-archinstall-target"

Keys are VMID strings (quoted). Values are arbitrary role names. An explicit [roles] entry wins over tag-based matching. Per-VM, the role appears in the per-VM JSON as "role": "<value>" and in fleet.jsonl as the role field of the monitor record (if the monitor is run later for that VM; currently the monitor does not read the role table, it only writes from the OVMF_VARS readback).

[tags]: tag-based role assignment (fallback when [roles] does not list the VM)

[tags]
"lamboot-target"       = ["lamboot", "uefi-test"]
"production-customer"  = ["prod", "customer"]

The hookscript reads tags: from /etc/pve/qemu-server/<vmid>.conf (Proxmox's per-VM tags field; semicolon- or comma-separated), and for each role name in [tags] (sorted alphabetically for determinism), checks if any of its tag list appears in the VM's tags. First match wins.

This is useful when you have many VMs and do not want to enumerate them all in [roles]. Tag matching is bypassed entirely if the VM has an explicit [roles] entry.

[hookscript]: per-field injection toggles (default: all on)

[hookscript]
inject_vmid     = true   # Default: true. False → omit "vmid" from JSON.
inject_fleet_id = true   # Default: true. False → omit "fleet_id" from JSON.
inject_role     = true   # Default: true. False → omit "role" from JSON + skip role determination.

If all three are false, the hookscript logs "All inject_* flags disabled in fleet.toml [hookscript]; skipping JSON refresh" and writes nothing: the guest then sees whatever was in the JSON file from the previous run (or no file at all on first start). Set all three to true or omit the [hookscript] table to use the default-all-on behavior.

Schema validation

The hookscript does best-effort validation:

  • TOML parse failure → empty config → role, fleet_id come back null/empty.
  • schema = "v2" or any non-v1 value → empty config (same as above).
  • Missing [fleet]fleet_id field in JSON is null but JSON is still written.
  • Missing [roles] AND [tags]role field in JSON is null.
  • Malformed [roles] (not a hash) → silently ignored, falls through to tag matching.

No errors are surfaced to the operator beyond /var/log/lamboot/hookscript.log. Use journalctl -u pve-cluster -n 50 or check the hookscript.log to diagnose; the script never blocks a VM start, by design.

Per-VM JSON schema v1 (what the guest sees)

Path on host: /var/lib/lamboot/<VMID>.json
Exposed to guest at: QEMU fw_cfg blob opt/lamboot/config
Read by guest from: /sys/firmware/qemu_fw_cfg/by_name/opt/lamboot/config/raw (after the guest kernel mounts qemu_fw_cfg; LamBoot reads via the FwCfg DMA interface directly, before any kernel)

Example (with all injects on):

{
  "schema_version": "v1",
  "vmid": "364",
  "hostname": "pve2",
  "fleet_id": "lamco-pve2",
  "role": "lamboot-archinstall-target",
  "written_by": "lamboot-hookscript 0.8.4",
  "written_at": "2026-05-29T00:20:58Z",
  "tags_at_setup": []
}

Field semantics

FieldTypeAlways present?Source
schema_versionstring "v1"yeshardcoded in hookscript
vmidstringonly if inject_vmid (default on)VMID passed as $ARGV[0]
hostnamestringyeshostname -s from POSIX uname
fleet_idstring or nullonly if inject_fleet_id[fleet].id from fleet.toml
rolestring or nullonly if inject_role[roles]."<VMID>" or tag match
written_bystringyeslamboot-hookscript $HOOKSCRIPT_VERSION
written_atRFC 3339 UTCyesstrftime("%Y-%m-%dT%H:%M:%SZ", gmtime)
tags_at_setuparray of stringsyestags: from /etc/pve/qemu-server/<vmid>.conf at pre-start time

tags_at_setup is intentionally the VM's tag list (from its Proxmox config), NOT the role-tag list from fleet.toml. The name reflects that these are the operator-set tags at the moment of this pre-start.

Lifecycle event matrix

Proxmox calls the hookscript with two positional args: <vmid> and <phase>. The hookscript dispatches on phase:

PhaseWhenHookscript workSide effects
pre-startAfter config validation, before QEMU execRefresh /var/lib/lamboot/<vmid>.json; call lamboot-monitor.py to capture previous boot's health (because pflash file is still readable; OVMF_VARS has not been touched yet by this boot)New JSON, log line + JSONL line. If previous boot was CrashLoop, log line includes WARNING.
post-startAfter QEMU is alive and responsiveLog line only ("VM started")log line
pre-stopBefore graceful stopNo-op. Included for completeness.none
post-stopAfter QEMU has exitedCall lamboot-monitor.py to capture the just-completed boot's health (OVMF_VARS at last-flush state)log line + JSONL line

post-start is intentionally a no-op for now: there is no use case yet that requires firing on the QEMU-alive transition. It is reserved.

Note the asymmetry between pre-start and post-stop monitor invocations:

  • pre-start monitor reads = "the boot we just left ended like X." If you boot, then qm stop, then qm start again, the pre-start of the second qm start reads the same flush that qm stop's post-stop already captured. That is the design: pre-start gives you a fresh-from-pflash readout in case the previous post-stop missed (host crash, abrupt termination).
  • post-stop monitor reads = "this boot ended like X." This is the canonical signal; the pre-start one is the safety net.

fw_cfg interface (host and guest contract)

QEMU's fw_cfg lets the host expose arbitrary blobs to the guest at known selector names. The host side:

qm set <N> --args "-fw_cfg name=opt/lamboot/config,file=/var/lib/lamboot/<N>.json"

becomes, in the QEMU command line:

... -fw_cfg name=opt/lamboot/config,file=/var/lib/lamboot/<N>.json ...

QEMU reads the file at process start and exposes it under the fw_cfg selector opt/lamboot/config. The opt/ prefix is the QEMU-reserved namespace for guest-OS-supplied configuration (guaranteed not to collide with QEMU's own fw_cfg entries).

How the guest accesses it

Before kernel: LamBoot reads via the UEFI FwCfg protocol exposed by OVMF (the firmware that backs the VM). OVMF probes the host's QEMU fw_cfg device on every boot; if opt/lamboot/config is present, LamBoot will see it and can parse the JSON.

After kernel: The Linux kernel exposes fw_cfg at /sys/firmware/qemu_fw_cfg/by_name/opt/lamboot/config/raw. cat that file inside the running guest to see exactly what LamBoot would have read at boot.

Why fw_cfg and not SMBIOS

An earlier hookscript version (pre-0.8.4) injected metadata via SMBIOS strings using qm set --smbios=.... Two problems:

  • qm set requires the VM config lock, which Proxmox holds during the pre-start lifecycle event, deadlocking.
  • SMBIOS strings are length-limited and have an awkward enum-of-fields interface.

The fw_cfg approach (file specified in args, content rewritten via plain cat/write on the file path) sidesteps both: no config-lock contention, the blob can be arbitrary size, and the host can update it without touching the VM config at all once the args line is in place.

Why a file, not a string fw_cfg

QEMU's -fw_cfg name=NAME,string=VALUE form embeds the value in the QEMU command line, which means rewriting it requires qm set (back to the config-lock problem). The file=PATH form makes QEMU read the file at exec time, so the host can rewrite the file's contents between process exits without touching VM config. This gives all-host-side mutability for free.

lamboot-monitor.py operation

Invoked by the hookscript on pre-start and post-stop. Also runnable manually: lamboot-monitor.py [--json] [--alert-webhook URL] [--threshold N].

What it does

  1. Scans /etc/pve/qemu-server/*.conf for VMs with bios: ovmf and an efidisk0: entry.
  2. For each match, resolves the efidisk0 storage spec to a file path via pvesm path.
  3. If the VM is stopped (or we are called for a stopped VMID), reads the OVMF_VARS file directly. If running, uses qemu-nbd to safely read the variables from the live VARS image without disrupting the VM.
  4. Parses the UEFI variable store (varstore.dat) for variables with the LamBoot vendor GUID 4c414d42-4f4f-5400-0000-000000000001:
    • LamBootState (u8 enum: 0=Fresh, 1=Booting, 2=BootedOK, 3=CrashLoop)
    • LamBootCrashCount (u8)
    • LamBootLastEntry (UTF-8 string)
    • LamBootTimestamp (epoch seconds)
    • LamBootVersion (u32, packed major.minor.patch)
  5. Computes an overall status field: "healthy" (state=BootedOK, crash_count below threshold), "warning" (crash_count >= threshold), "critical" (state=CrashLoop).
  6. Emits a JSON record per VM (one VM with --vmid <N>, or all OVMF VMs without filter).

Output shape

Per VM:

{
  "vmid": 364,
  "name": "arch-btrfs-lbi",
  "state": "BootedOK",
  "crash_count": 0,
  "last_entry": "Arch Linux (7.0.10-arch1-1)",
  "timestamp": "2026-05-29T00:51:34Z",
  "version": "0.11.14",
  "status": "healthy",
  "qmp_status": "running"
}

When invoked by the hookscript on pre-start / post-stop, the JSON goes to stdout; the hookscript reads it and appends it as one line to /var/log/lamboot/fleet.jsonl.

Caveats

  • Reading OVMF_VARS of a running VM via qemu-nbd requires the VM storage to be a block device or qcow2: this works for local-lvm, local-zfs, local (qcow2). NFS-backed storage with raw images is trickier; the script falls back to refusing the read if qemu-nbd errors.
  • The variable values reflect the last flush by OVMF, which on most setups is at every guest write to efivars. There is a small window between a guest write and the host-visible flush.

Observability

/var/log/lamboot/hookscript.log: human audit trail

One line per hookscript invocation. Format:

[2026-05-28T19:20:58] VM 364 (pre-start): Refreshed /var/lib/lamboot/364.json (fleet=lamco-pve2 role=lamboot-archinstall-target)
[2026-05-28T19:20:58] VM 364 (pre-start): VM starting
[2026-05-28T19:21:03] VM 364 (post-start): VM started
[2026-05-28T19:45:12] VM 364 (post-stop): VM stopped, capturing boot health
[2026-05-28T19:45:13] VM 364 (post-stop): Boot health captured

Tail this when debugging. There is no rotation; it is the operator's responsibility (a logrotate snippet is not shipped yet; see the troubleshooting section).

/var/log/lamboot/fleet.jsonl: structured rolling log

One JSON object per line. Each line is a monitor record (see the monitor output shape). Append-only. Suitable for ingestion into log-shipping pipelines (Loki, Vector, and similar) or for jq analysis:

# Show last 10 records for VM 364
grep '"vmid": *364' /var/log/lamboot/fleet.jsonl | tail -10 | jq

# Count crash-loops in the fleet over the log's lifetime
jq -c 'select(.status == "critical")' /var/log/lamboot/fleet.jsonl | wc -l

/var/lib/lamboot/<vmid>.json: current-state snapshot

Always exactly the data the guest will see on its next boot. Cheap to cat for debugging:

cat /var/lib/lamboot/364.json | jq

Troubleshooting

LamBoot does not appear in boot menu

# Check if UEFI entry exists
efibootmgr -v

# Re-create entry
efibootmgr -c -d /dev/vda -p 1 -l '\EFI\LamBoot\lambootx64.efi' -L 'LamBoot'

BLS entries not found

# Verify entries exist
ls /boot/efi/loader/entries/

# If /boot is on ext4, install the ext4 driver
cp ext4_x64.efi /boot/efi/EFI/LamBoot/drivers/

VM stuck in CrashLoop

On the Proxmox host:

# Check status
lamboot-monitor.py --vmid 102

# To reset: stop VM, clear NVRAM by removing and re-adding efidisk0
# Or: boot from rescue media and clear the NVRAM variable

Monitor cannot read OVMF_VARS

# Check permissions
ls -la /var/lib/vz/images/100/

# pvesm path may need root access
sudo lamboot-monitor.py --vmid 100

Symptom: role is empty in /var/lib/lamboot/<vmid>.json

Three causes, in order of frequency:

  1. fleet.toml schema mistake. The role table is [roles] "364" = "..." (flat string-keyed), NOT [vms.364] role = "...". The hookscript looks at fleet->{roles}{$vmid}, not nested tables.
  2. inject_role = false in [hookscript]. Either explicitly set it to true or remove the line (default is on).
  3. TOML parse error. Run python3 -c "import tomllib; tomllib.load(open('/etc/lamboot/fleet.toml','rb'))" to surface the parse error. The hookscript silently returns an empty config on parse failure.

Symptom: hookscript.log is empty / qm start does not invoke the script

  • Verify qm config <N> | grep hookscript: shows the line. If absent, rerun the qm set --hookscript local:snippets/lamboot-hookscript.pl command.
  • Verify the script lives at /var/lib/vz/snippets/lamboot-hookscript.pl exactly (not under a different snippets storage). Proxmox resolves the local:snippets/ prefix against /var/lib/vz/snippets/ for the local storage; if your snippets are on a different storage, change the qm set line to match (<storage>:snippets/<file>).
  • Verify the script is executable: ls -la /var/lib/vz/snippets/lamboot-hookscript.pl (should be -rwxr-xr-x).

Symptom: pre-start JSON written but guest LamBoot does not see the fleet_id

  • Verify the -fw_cfg ... file=... argument actually made it into the QEMU command line: ps -ef | grep "kvm -id <N>" and look for -fw_cfg name=opt/lamboot/config,file=/var/lib/lamboot/<N>.json.
  • If absent, check qm config <N> | grep args: and either set the args line or fix the existing one.
  • If the VM was already running when you set the args, the running QEMU process does not pick up the new args. qm reboot is not enough. qm stop <N> && qm start <N> is required.
  • Inside the booted guest (after kernel), confirm fw_cfg sees the blob: cat /sys/firmware/qemu_fw_cfg/by_name/opt/lamboot/config/raw. If the file is missing, the kernel did not mount fw_cfg: load the qemu_fw_cfg module with modprobe qemu_fw_cfg.

Symptom: lamboot-monitor.py errors with "qemu-nbd: failed to attach"

  • Storage backend does not support nbd export from the live VM. Either stop the VM and re-run the monitor against the stopped image, or accept that this VM's pre-start monitor invocation will fail silently (the hookscript catches monitor failures and logs them as Failed to capture boot health (monitor returned error)).
  • Some NFS-backed storages do not expose efidisk0 in a form qemu-nbd can read. This is a known limitation; see the limitations section.

Symptom: fleet.jsonl grows without bound

That is the current behavior. Rotation is the operator's responsibility. Drop-in /etc/logrotate.d/lamboot:

/var/log/lamboot/*.log /var/log/lamboot/*.jsonl {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Scope and known limitations

  1. NFS-storage efidisk reads via qemu-nbd are flaky. Monitor invocations for VMs whose efidisk0 is on NFS may fail silently. This is filed and tracked in lamboot-tools-dev SPEC-LAMBOOT-PVE-SETUP.
  2. post-start phase is a no-op. Reserved for future use; documented so anyone adding a post-start handler can do so without surprises.
  3. Log rotation is operator-supplied. Drop a logrotate snippet (see the troubleshooting section).
  4. fleet.toml uses best-effort validation. Misshapen TOML silently degrades to "empty config"; there is no startup-time check that tells the operator they have a typo. Formal validation is future work in SDS-2.
  5. Single-node scope. Each Proxmox node reads its own fleet.toml. If you live-migrate a VM between nodes, the target node's fleet.toml determines the role written to its JSON file; there is no cluster-wide single source of truth. This suits single-node and small-fleet operators; cluster operators should plan accordingly.
  6. The hookscript and monitor are version-pinned to 0.8.4 and current, respectively. They are updated manually rather than by apt upgrade lamboot on the host: they live under /var/lib/vz/snippets/ and /usr/local/bin/, both outside dpkg's tracked paths by design (so operator-customized variants survive package upgrades). Re-install manually after upgrading the source files.
  7. Per-node aggregation. fleet.jsonl is per-node. Aggregating across nodes is a future-work item; current operators do it via log-shipping into Loki/Vector or equivalent.
  8. fw_cfg blob is plaintext. Anything in /var/lib/lamboot/<N>.json is readable by any process inside the guest that can open /sys/firmware/qemu_fw_cfg/by_name/opt/lamboot/config/raw (effectively any privileged process). Keep secrets out of fleet.toml.

Validation procedure (end-to-end sanity check)

After install, this is the minimal sequence to confirm everything works:

# Pick a LamBoot guest VMID (e.g. 364).

# 1. Manually invoke pre-start (idempotent).
/var/lib/vz/snippets/lamboot-hookscript.pl 364 pre-start

# 2. Inspect the JSON the guest will see.
cat /var/lib/lamboot/364.json | jq
# Expect: schema_version=v1, vmid=364, fleet_id, role all populated.

# 3. Confirm the hookscript wrote audit log.
tail -3 /var/log/lamboot/hookscript.log
# Expect: lines for VM 364 (pre-start) including "Refreshed ...".

# 4. Cold-start the VM so it picks up the -fw_cfg arg.
qm stop 364 && qm start 364

# 5. After VM is booted, SSH in as a non-root user and read the blob.
ssh <user>@<vm-ip> cat /sys/firmware/qemu_fw_cfg/by_name/opt/lamboot/config/raw | jq
# Expect: the same JSON the host wrote. If the file path doesn't exist,
# `modprobe qemu_fw_cfg` first, then re-cat.

# 6. Stop the VM and confirm post-stop captured boot health.
qm stop 364
sleep 5
tail -1 /var/log/lamboot/fleet.jsonl | jq
# Expect: a record for vmid=364, status=healthy (or other), with the
# LamBoot state/crash_count/last_entry/timestamp fields populated.

If any of these do not match expectations, see the troubleshooting section.

Future expansion (scope-locked references)

This document covers what works today. Larger, planned expansions that would change the surface described here:

  • SDS-2 (lamboot-pve-setup proper tool): automates the per-guest hookscript and fw_cfg attachment across an entire fleet, handles TOML editing safely, and provides a lamboot-pve-setup status command. Scheduled later per operator decision.
  • Cluster-wide fleet.toml aggregation: replace per-node fleet.toml with a Proxmox cluster filesystem (/etc/pve/lamboot/fleet.toml) source of truth. Designed and pending implementation; it would resolve the single-node scope limitation above.
  • Fleet-wide rolling boot-health aggregation: Loki/Vector recipes, or a dedicated lamboot-fleet-status aggregator.

Document changes to the schema will accompany schema bumps in the hookscript (from v1 to v2 and beyond); old fleet.toml files at v1 will continue to be parsed by future hookscripts via the explicit schema_version check.


Related documentation