08/2026 - Hardened Arch Linux Workstation (Bare Metal) - Full Project Archive v1.1
Version: v1.1
Date: 2026-08-05
Host Context: archbtw (user: andrew)
Status: Operational/Known Good
Table of Contents
- Executive Summary
- Environment & Objectives
- Architecture Overview
- Storage Design (LUKS + Btrfs)
- Snapshot & Rollback Strategy
- Package Governance & Supply-Chain Controls
- Hardening Controls
- Automation Strategy (systemd)
- Operations Runbook
- Validation Evidence
- Incident/Troubleshooting Record (SMART service)
- Known Good Checkpoint
- Resume/Portfolio Extracts
- Lessons Learned
- v2 Improvement Backlog
- Command Reference
- Glossary
- Changelog
Executive Summary
This project documents the design and operation of a hardened, rollback-capable Arch Linux workstation built on bare metal. The implementation prioritizes stability, security, recoverability, and controlled automation:
- Encrypted storage with LUKS + Btrfs subvolume layout
- Snapshot-aware update workflow with fast rollback paths
- Layered host hardening (kernel, MAC, audit, firewall, malware monitoring)
- Safe maintenance automation via systemd timers/services
- Manual control retained for high-impact operational decisions
Outcome: a personal desktop operated with enterprise-style systems discipline.
Environment & Objectives
Environment
- OS: Arch Linux (bare metal)
- Workloads: gaming, multimedia, development, cybersecurity labs
- Filesystem: Btrfs
- Encryption: LUKS
- Snapshot stack: Snapper + snap-pac + grub-btrfs
- Automation: systemd timers/services
- Security stack: linux-hardened, AppArmor, auditd, clamav-daemon, least-privilege firewall/network policy
- Compatibility layer: Distrobox (primarily Fedora based containers)
Objectives
- Keep Arch flexible while minimizing operational risk
- Ensure quick rollback/recovery from update regressions
- Enforce least-privilege and layered controls
- Automate only low-risk, repeatable maintenance tasks
- Preserve manual authority for risky/destructive decisions
Architecture Overview
Design Principles
- Recovery-first change management
- Trust-boundary package governance
- Least-privilege and defense-in-depth
- Safe automation only
- Evidence-driven operations via logs and validation checkpoints
Core Control Planes
- State plane: Btrfs subvolumes + snapshots
- Change plane: pre/post update snapshots + rollback boot entries
- Security plane: hardened kernel + MAC + auditing + firewall + anti-malware
- Operations plane: timer-driven checks + weekly human review cadence
Storage Design (LUKS + Btrfs)
Subvolume Layout
- @root/system state
- @home/user data
- @snapshots/snapshot history
- @swap/swap specific isolation
Rationale
- Separation of concerns between OS state and user state
- Cleaner rollback and maintenance operations
- Better lifecycle control over snapshot retention and storage footprint
- Data-at-rest protection through LUKS encryption
Snapshot & Rollback Strategy
Components
- Snapper: snapshot creation/management
- snap-pac: pre/post pacman transaction snapshots
- grub-btrfs: snapshot visibility in bootloader entries
Behavior
- Automatic snapshots before and after system updates
- Manual snapshots available for operator-defined checkpoints
- Boot-time recovery path available via snapshot entries
Retention
- Snapshot trimming configured to maintain a small footprint (manual + auto limits), preventing sprawl while preserving rollback utility.
Package Governance & Supply-Chain Controls
Boundary Model
- pacman => system packages (trusted)
- Discover/Flatpak => user facing app layer (semi-trusted)
- fwupd => firmware lifecycle (trusted)
- AUR => explicitly disallowed on host baseline (untrusted)
Compatibility Strategy
- Distrobox used to access non-native tooling (primarily Fedora ecosystem) without polluting host package integrity.
Security Value
- Reduces exposure to unvetted host packages
- Keeps trust assumptions explicit and auditable
- Minimizes configuration drift across package channels
Hardening Controls
Implemented layered controls:
- linux-hardened kernel
- AppArmor policy enforcement
- auditd syscall monitoring/telemetry
- least-privilege firewall/network posture
- clamav-daemon background scanning (acceptable overhead on hardware)
Operationally, controls were tuned to retain day-to-day usability while increasing baseline resistance and observability.
Automation Strategy (systemd)
Automated (Safe/Low-Risk)
- SMART long-test orchestration schedule
- Btrfs scrub cadence
- Weekly health reporting
- Snapshot hooks around updates (via snap-pac integration)
Manual (High-Impact/Decision-Based)
- Update timing/approval
- Rollback execution decisions
- Security anomaly triage and interpretation
- Snapshot policy changes
Safe Doctrine
- Automate observability and hygiene.
- Do not automate destructive or high-blast-radius actions.
Operations Runbook
Weekly
- Review weekly health report
- Check critical logs (journalctl -p 0..4 -b)
- Check failed units (systemctl --failed)
- Confirm timers (systemctl list-timers --all)
- Review snapshot list/retention trend
- Perform updates (if planned) with snapshot awareness
- Post-update validation and checkpoint snapshot
Monthly
- Verify SMART cycle execution logs
- Verify scrub outcomes
- Review snapshot storage footprint
Quarterly
- Reassess hardening effectiveness/usability tradeoffs
- Revalidate firewall least-privilege assumptions
- Review audit signal quality
- Run rollback drill from known-good state
Validation Evidence (2026-08-05)
Confirmed
- SMART service final status:
- Result=success
- ExecMainStatus=0
- SMART timer:
- enabled
- active (waiting)
- next trigger: Tue 2026-09-01 02:20:28 EDT
- linked trigger target: smart-long-all.service
- Weekly report and scrub flows previously validated successful in session review
Interpretation
- Systemd automation for core maintenance is functioning as intended at checkpoint time.
Incident/Troubleshooting Record (SMART service)
Symptom Observed
- /usr/local/bin/smart-long-all.sh: line 10: date: unbound variable
Root Cause
- Timestamp echo line used malformed variable expansion ($date ...) under strict mode instead of command substitution.
Corrective Action
- Replaced malformed text with:
- $(date --iso-8601=seconds)
Post-Fix Verification
- Script syntax check passed
- Service start completed successfully
- Unit result fields showed success
- Journal confirmed successful execution path
Additional Expected Log Behavior
- When long self-tests are already running, smartctl may emit:
- "Can't start self-test without aborting current test ..."
- This is expected and non-fatal in this context.
Known Good Checkpoint
Checkpoint Date:
- 2026-08-05
Checkpoint Label (recommended)
- Known-good: maintenance timers/services fixed (2026-08-05)
Intended Use
- Reference rollback baseline after maintenance script/timer stabilization.
Resume/Portfolio Extracts
Two-Line Resume Version
Designed and operated a hardened Arch Linux bare-metal system using LUKS + Btrfs, Snapper/snap-pac/grub-btrfs rollback workflows, and strict package trust boundaries (pacman/Flatpak/fwupd, no AUR).
Implemented systemd-based safe automation and layered controls (linux-hardened, AppArmor, auditd, least-privilege firewall, ClamAV) to improve reliability, recovery speed, and endpoint security.
Five-Bullet Resume Version
- Architectured a security-first Arch Linux platform with encrypted LUKS+Btrfs and structured subvolumes (@, @home, @snapshots, @swap).
- Implemented resilient rollback lifecycle using Snapper + snap-pac + grub-btrfs with pre/post update snapshots and boot-time fallback.
- Enforced package trust boundaries across pacman, Flatpak, and fwupd; disallowed AUR on host baseline.
- Hardened endpoint with linux-hardened kernel, AppArmor, auditd, least-privilege firewall posture, and clamav-daemon.
- Automated low-risk maintenance via systemd timers/services while retaining manual authority for updates, rollback, and incident triage.
Lessons Learned
- Recovery-first architecture dramatically reduces rolling-release stress.
- Clear trust boundaries simplify supply-chain risk management.
- systemd automation is most effective when scoped to safe tasks.
- Shell strict mode improves safety but punishes small syntax mistakes.
- Historical failures in journald are acceptable when root cause is fixed and validated.
- Immediate known-good snapshots after fixes improve future incident response.
v2 Improvement Backlog
High Priority
- Improve SMART script behavior to explicitly "skip if test in progress"
- Add concise weekly dashboard summary script
Medium Priority
- Formalize periodic rollback drills with RTO target
- Add config backup/restore verification workflow
Low Priority
- Optional log centralization/retention enhancement
- Add diagrams/screenshots for archive readability
Command Reference
Service & Timer
- systemctl status
--no-pager - systemctl list-timers --all
- systemctl --failed
- systemctl show -p Result -p ExecMainStatus
Logs
- journalctl -u
-n 100 --no-pager - journalctl -p 0..4 -b
Script Validation
- bash -n /path/to/script.sh
Snapper
- snapper -c root list
- snapper -c root create -d "
"
Glossary
- LUKS: Linux disk encryption standard
- Btrfs subvolume: logical filesystem partitioning construct
- Snapper: snapshot lifecycle and rollback tooling
- snap-pac: pacman hooks for automatic updates
- grub-btrfs: exposes snapshots in GRUB boot menu
- AppArmor: Linux mandatory access control framework
- auditd: Linux audit daemon for syscall/event telemetry
- systemd timer: scheduled trigger for systemd services
Changelog
v1.1 - 2026-08-05
- Added concrete host context and verified operational evidence
- Documented SMART script fault and resolution
- Recorded successful service/timer state and next trigger
- Established explicit known-good checkpoint entry
v1.0 - 2026-08-05
- Initial documentation set and archive scaffolding