Stay informed with the latest news and updates from Dru Does Tech. If you are on social media, be
sure to follow us there for real time updates!
10/2026 - Price Increases Effective 11/01/2026
Due to rising costs across the board, we will be implementing a price increase for our services
effective November 1, 2026. We remain committed to providing high-quality tech support and
appreciate your understanding as we adjust our pricing to continue delivering the best service
possible.
Prices have been updated on our website to reflect the new rates. However, folks can still save
significantly by signing up for a yearly plan, as those have been drastically reduced from before
and remain the best value for our customers.
Please see our How It Works page for the latest pricing
information.
08/2026 - Hardened Arch Linux Workstation (Bare Metal) - Full Project Archive v1.1
Version: v1.1
Date: 2026-08-05
Host Context: archbtw (user: andrew)
Status: Operational/Known Good
Table of Contents
- Executive Summary
- Environment & Objectives
- Architecture Overview
- Storage Design (LUKS + Btrfs)
- Snapshot & Rollback Strategy
- Package Governance & Supply-Chain Controls
- Hardening Controls
- Automation Strategy (systemd)
- Operations Runbook
- Validation Evidence
- Incident/Troubleshooting Record (SMART service)
- Known Good Checkpoint
- Resume/Portfolio Extracts
- Lessons Learned
- v2 Improvement Backlog
- Command Reference
- Glossary
- Changelog
Executive Summary
This project documents the design and operation of a hardened, rollback-capable Arch Linux
workstation built on bare metal. The implementation prioritizes stability, security, recoverability,
and controlled automation:
- Encrypted storage with LUKS + Btrfs subvolume layout
- Snapshot-aware update workflow with fast rollback paths
- Layered host hardening (kernel, MAC, audit, firewall, malware monitoring)
- Safe maintenance automation via systemd timers/services
- Manual control retained for high-impact operational decisions
Outcome: a personal desktop operated with enterprise-style systems discipline.
Environment & Objectives
Environment
- OS: Arch Linux (bare metal)
- Workloads: gaming, multimedia, development, cybersecurity labs
- Filesystem: Btrfs
- Encryption: LUKS
- Snapshot stack: Snapper + snap-pac + grub-btrfs
- Automation: systemd timers/services
- Security stack: linux-hardened, AppArmor, auditd, clamav-daemon, least-privilege
firewall/network policy
- Compatibility layer: Distrobox (primarily Fedora based containers)
Objectives
- Keep Arch flexible while minimizing operational risk
- Ensure quick rollback/recovery from update regressions
- Enforce least-privilege and layered controls
- Automate only low-risk, repeatable maintenance tasks
- Preserve manual authority for risky/destructive decisions
Architecture Overview
Design Principles
- Recovery-first change management
- Trust-boundary package governance
- Least-privilege and defense-in-depth
- Safe automation only
- Evidence-driven operations via logs and validation checkpoints
Core Control Planes
- State plane: Btrfs subvolumes + snapshots
- Change plane: pre/post update snapshots + rollback boot entries
- Security plane: hardened kernel + MAC + auditing + firewall + anti-malware
- Operations plane: timer-driven checks + weekly human review cadence
Storage Design (LUKS + Btrfs)
Subvolume Layout
- @root/system state
- @home/user data
- @snapshots/snapshot history
- @swap/swap specific isolation
Rationale
- Separation of concerns between OS state and user state
- Cleaner rollback and maintenance operations
- Better lifecycle control over snapshot retention and storage footprint
- Data-at-rest protection through LUKS encryption
Snapshot & Rollback Strategy
Components
- Snapper: snapshot creation/management
- snap-pac: pre/post pacman transaction snapshots
- grub-btrfs: snapshot visibility in bootloader entries
Behavior
- Automatic snapshots before and after system updates
- Manual snapshots available for operator-defined checkpoints
- Boot-time recovery path available via snapshot entries
Retention
- Snapshot trimming configured to maintain a small footprint (manual + auto limits),
preventing sprawl while preserving rollback utility.
Package Governance & Supply-Chain Controls
Boundary Model
- pacman => system packages (trusted)
- Discover/Flatpak => user facing app layer (semi-trusted)
- fwupd => firmware lifecycle (trusted)
- AUR => explicitly disallowed on host baseline (untrusted)
Compatibility Strategy
- Distrobox used to access non-native tooling (primarily Fedora ecosystem) without polluting
host package integrity.
Security Value
- Reduces exposure to unvetted host packages
- Keeps trust assumptions explicit and auditable
- Minimizes configuration drift across package channels
Hardening Controls
Implemented layered controls:
- linux-hardened kernel
- AppArmor policy enforcement
- auditd syscall monitoring/telemetry
- least-privilege firewall/network posture
- clamav-daemon background scanning (acceptable overhead on hardware)
Operationally, controls were tuned to retain day-to-day usability while increasing baseline
resistance and observability.
Automation Strategy (systemd)
Automated (Safe/Low-Risk)
- SMART long-test orchestration schedule
- Btrfs scrub cadence
- Weekly health reporting
- Snapshot hooks around updates (via snap-pac integration)
Manual (High-Impact/Decision-Based)
- Update timing/approval
- Rollback execution decisions
- Security anomaly triage and interpretation
- Snapshot policy changes
Safe Doctrine
- Automate observability and hygiene.
- Do not automate destructive or high-blast-radius actions.
Operations Runbook
Weekly
- Review weekly health report
- Check critical logs (journalctl -p 0..4 -b)
- Check failed units (systemctl --failed)
- Confirm timers (systemctl list-timers --all)
- Review snapshot list/retention trend
- Perform updates (if planned) with snapshot awareness
- Post-update validation and checkpoint snapshot
Monthly
- Verify SMART cycle execution logs
- Verify scrub outcomes
- Review snapshot storage footprint
Quarterly
- Reassess hardening effectiveness/usability tradeoffs
- Revalidate firewall least-privilege assumptions
- Review audit signal quality
- Run rollback drill from known-good state
Validation Evidence (2026-08-05)
Confirmed
- SMART service final status:
- Result=success
- ExecMainStatus=0
- SMART timer:
- enabled
- active (waiting)
- next trigger: Tue 2026-09-01 02:20:28 EDT
- linked trigger target: smart-long-all.service
- Weekly report and scrub flows previously validated successful in session review
Interpretation
- Systemd automation for core maintenance is functioning as intended at checkpoint time.
Incident/Troubleshooting Record (SMART service)
Symptom Observed
- /usr/local/bin/smart-long-all.sh: line 10: date: unbound variable
Root Cause
- Timestamp echo line used malformed variable expansion ($date ...) under strict mode instead
of command substitution.
Corrective Action
- Replaced malformed text with:
- $(date --iso-8601=seconds)
Post-Fix Verification
- Script syntax check passed
- Service start completed successfully
- Unit result fields showed success
- Journal confirmed successful execution path
Additional Expected Log Behavior
- When long self-tests are already running, smartctl may emit:
- "Can't start self-test without aborting current test ..."
- This is expected and non-fatal in this context.
Known Good Checkpoint
Checkpoint Date:
Checkpoint Label (recommended)
- Known-good: maintenance timers/services fixed (2026-08-05)
Intended Use
- Reference rollback baseline after maintenance script/timer stabilization.
Resume/Portfolio Extracts
Two-Line Resume Version
Designed and operated a hardened Arch Linux bare-metal system using LUKS + Btrfs,
Snapper/snap-pac/grub-btrfs rollback workflows, and strict package trust boundaries
(pacman/Flatpak/fwupd, no AUR).
Implemented systemd-based safe automation and layered controls (linux-hardened, AppArmor, auditd,
least-privilege firewall, ClamAV) to improve reliability, recovery speed, and endpoint security.
Five-Bullet Resume Version
- Architectured a security-first Arch Linux platform with encrypted LUKS+Btrfs and structured
subvolumes (@, @home, @snapshots, @swap).
- Implemented resilient rollback lifecycle using Snapper + snap-pac + grub-btrfs with pre/post
update snapshots and boot-time fallback.
- Enforced package trust boundaries across pacman, Flatpak, and fwupd; disallowed AUR on host
baseline.
- Hardened endpoint with linux-hardened kernel, AppArmor, auditd, least-privilege firewall
posture, and clamav-daemon.
- Automated low-risk maintenance via systemd timers/services while retaining manual authority
for updates, rollback, and incident triage.
Lessons Learned
- Recovery-first architecture dramatically reduces rolling-release stress.
- Clear trust boundaries simplify supply-chain risk management.
- systemd automation is most effective when scoped to safe tasks.
- Shell strict mode improves safety but punishes small syntax mistakes.
- Historical failures in journald are acceptable when root cause is fixed and validated.
- Immediate known-good snapshots after fixes improve future incident response.
v2 Improvement Backlog
High Priority
- Improve SMART script behavior to explicitly "skip if test in progress"
- Add concise weekly dashboard summary script
Medium Priority
- Formalize periodic rollback drills with RTO target
- Add config backup/restore verification workflow
Low Priority
- Optional log centralization/retention enhancement
- Add diagrams/screenshots for archive readability
Command Reference
Service & Timer
- systemctl status --no-pager
- systemctl list-timers --all
- systemctl --failed
- systemctl show -p Result -p ExecMainStatus
Logs
- journalctl -u -n 100 --no-pager
- journalctl -p 0..4 -b
Script Validation
- bash -n /path/to/script.sh
Snapper
- snapper -c root list
- snapper -c root create -d ""
Glossary
- LUKS: Linux disk encryption standard
- Btrfs subvolume: logical filesystem partitioning construct
- Snapper: snapshot lifecycle and rollback tooling
- snap-pac: pacman hooks for automatic updates
- grub-btrfs: exposes snapshots in GRUB boot menu
- AppArmor: Linux mandatory access control framework
- auditd: Linux audit daemon for syscall/event telemetry
- systemd timer: scheduled trigger for systemd services
Changelog
v1.1 - 2026-08-05
- Added concrete host context and verified operational evidence
- Documented SMART script fault and resolution
- Recorded successful service/timer state and next trigger
- Established explicit known-good checkpoint entry
v1.0 - 2026-08-05
- Initial documentation set and archive scaffolding
01/2026 - New Site, New Service Plans!
We are excited to announce the launch of our new website and service plans designed to cater to both
individual and business needs. Whether you're looking for one-time support or ongoing assistance, we
have a plan for you!
10/2025 - Dru Does Tech Joins TBD!
We are thrilled to announce that Dru Does Tech has joined TBD, a local business network that meets
in various locations around Southern Maine. This partnership allows us to connect with other
businesses in the community and expand our reach to better serve our clients.
09/2025 - We're Open for Business!
Dru Does Tech is officially open for business effective 09-22-2025! We are excited to offer our
services to individuals and businesses in need of reliable tech support. We strive to focus on older
adults and small to medium-sized businesses, providing personalized solutions to meet their unique
needs.