diff --git a/dotfiles/agents/skills/disk-space-assessment/SKILL.md b/dotfiles/agents/skills/disk-space-assessment/SKILL.md new file mode 100644 index 00000000..4992bcd2 --- /dev/null +++ b/dotfiles/agents/skills/disk-space-assessment/SKILL.md @@ -0,0 +1,135 @@ +--- +name: disk-space-assessment +description: Measure and explain disk usage without deleting data, producing reusable compressed ncdu snapshots and an evidence-backed cleanup handoff. Use when disk is low, a filesystem is unexpectedly large, builds fail from space pressure, the user asks what is consuming space, or deeper iterative disk analysis is needed before cleanup. +--- + +# Disk Space Assessment + +Build a reproducible picture of disk usage. Stop at findings and proposed actions; use `disk-space-cleanup` when the user authorizes remediation. + +Read `references/ignore-paths.md` before scanning. Read `references/observed-heavy-hitters.md` when triaging this machine or interpreting familiar paths. + +## Required Artifact Contract + +Always create a reusable, timestamped `ncdu` export for every filesystem or major root assessed. Never make an interactive `ncdu` session, transient `/tmp` export, or `du` output the only record of an assessment. + +Use the local wrapper: + +```bash +safe_ncdu /home +safe_ncdu /nix/store +sudo -n env HOME=/home/imalison safe_ncdu / +``` + +`safe_ncdu` writes these durable artifacts under `~/.cache/ncdu/`: + +- `safe-ncdu--.json.zst`: compressed importable ncdu data +- `.excludes`: exact exclusions used +- `.meta`: scan root, timestamps, hostname, ncdu version, status, and artifact paths +- `latest-.json.zst` plus matching sidecar symlinks + +Keep timestamped files intact for iterative analysis. Report their absolute paths in the handoff. If a scan cannot finish, retain and report the failed manifest, mark the coverage gap, and use bounded probes as supplemental evidence. + +## Workflow + +1. Record filesystem pressure and topology. +2. Choose scan roots that cover the pressured filesystem without crossing mounts. +3. Create reusable `safe_ncdu` snapshots before any cleanup. +4. Analyze snapshots repeatedly with `top` and `open`; do not rescan for every question. +5. Attribute special stores such as Nix separately. +6. Produce findings, confidence, cleanup candidates, and artifact paths. + +## 1. Establish the Baseline + +```bash +df -h / +df -h /home +df -h /nix +findmnt -rn -o TARGET,FSTYPE,SOURCE,OPTIONS +``` + +Record used/free space and whether `/home`, `/nix`, or other large paths are separate mounts. Note active builds or services that may cause measurements to move during the assessment. + +## 2. Select Scan Roots + +Prefer one-filesystem coverage: + +- Scan `/` for root accounting. +- Scan separately mounted `/home` and `/nix/store` independently. +- Add a focused root such as `~/Projects` when the first snapshot identifies it as dominant. +- Use privileged root scans when unprivileged results undercount private service state. + +Inspect exclusions before a long scan: + +```bash +safe_ncdu excludes / +``` + +Update `references/ignore-paths.md` and the implementation of `safe_ncdu` together when a newly discovered remote, recursive, or special mount needs a permanent exclusion. + +## 3. Create the Snapshots + +Run scans early enough that cleanup does not destroy the evidence: + +```bash +safe_ncdu /home +safe_ncdu /nix/store +sudo -n env HOME=/home/imalison safe_ncdu / +``` + +If `safe_ncdu` is unavailable, source or run `/srv/dotfiles/dotfiles/lib/functions/safe_ncdu`. If `ncdu` itself is missing, use Nix temporarily rather than substituting a non-reusable interactive scan. + +Do not store the privileged scan in root's home. Set `HOME=/home/imalison` so all artifacts remain together and are available to later sessions. + +## 4. Analyze Iteratively + +Query the same export at multiple depths: + +```bash +safe_ncdu top ~/.cache/ncdu/latest-root.json.zst 30 +safe_ncdu top ~/.cache/ncdu/latest-root.json.zst 30 /home/imalison +safe_ncdu top ~/.cache/ncdu/latest-home.json.zst 30 /imalison/Projects +safe_ncdu open ~/.cache/ncdu/latest-root.json.zst +``` + +Use `du` only for bounded confirmation, live-change checks, or paths missing from the snapshot: + +```bash +timeout 30s du -xh --max-depth=1 "$HOME/.cache" 2>/dev/null | sort -h +timeout 30s du -xh --max-depth=1 "$HOME/.local/share" 2>/dev/null | sort -h +``` + +Distinguish physical allocation from logical/apparent size, especially for sparse files, hardlinked Nix store data, copy-on-write storage, and container layers. Do not sum overlapping Nix closure sizes as reclaim estimates. + +## 5. Attribute Nix Store Usage + +When `/nix/store` is large, first quantify whether garbage collection can help: + +```bash +nix_store_audit --top 30 +nix-store --gc --print-dead +nix-store --gc --print-roots +``` + +Use `/srv/dotfiles/dotfiles/lib/functions/find_store_path_gc_roots` and `nix why-depends` to explain why a large path is retained. Inspect `.direnv/flake-profile-*`, `result*` symlinks, system generations, and current/booted system closures. Prefer `nix_store_audit` over an initial `du -sh /nix/store`, which is slow and does not explain retention. + +## Assessment Handoff + +Return: + +- Baseline filesystem usage and scan time. +- Absolute paths to every snapshot, `.excludes`, and `.meta` artifact. +- Coverage and privilege limitations. +- Largest consumers with evidence from snapshot queries or store attribution. +- Ranked cleanup candidates, expected reclaim range, risk, and whether data is rebuildable. +- Volatile paths or active processes that cleanup must preserve. + +Do not delete, prune, garbage-collect, truncate, or mutate data in this skill. + +## Learning Loop + +After each assessment: + +1. Add new permanent scan exclusions to `references/ignore-paths.md` and `safe_ncdu`. +2. Add stable machine-specific diagnostic knowledge to `references/observed-heavy-hitters.md`. +3. Remove stale observations instead of accumulating an unbounded chronological log. diff --git a/dotfiles/agents/skills/disk-space-assessment/agents/openai.yaml b/dotfiles/agents/skills/disk-space-assessment/agents/openai.yaml new file mode 100644 index 00000000..510aff03 --- /dev/null +++ b/dotfiles/agents/skills/disk-space-assessment/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Disk Space Assessment" + short_description: "Build reusable evidence about disk usage" + default_prompt: "Use $disk-space-assessment to measure disk usage and create reusable ncdu snapshots." diff --git a/dotfiles/agents/skills/disk-space-assessment/references/ignore-paths.md b/dotfiles/agents/skills/disk-space-assessment/references/ignore-paths.md new file mode 100644 index 00000000..ad62303b --- /dev/null +++ b/dotfiles/agents/skills/disk-space-assessment/references/ignore-paths.md @@ -0,0 +1,21 @@ +# Ignore Paths for Disk Assessment + +Exclude these paths from `ncdu`/`du` scans because they are remote, recursive, separately mounted, or noisy. + +## Known Ignores + +- `$HOME/keybase` +- `$HOME/.cache/keybase` +- `$HOME/.local/share/keybase` +- `$HOME/.config/keybase` +- `/keybase` +- `/var/lib/railbird` +- `/run/user/*/doc` (FUSE portal mount; observed as `/run/user/1004/doc`) + +Discover candidates with: + +```bash +findmnt -rn -o TARGET,FSTYPE,SOURCE | rg '(keybase|fuse|rclone|s3|railbird)' +``` + +When adding an ignore, update `/srv/dotfiles/dotfiles/lib/functions/safe_ncdu` in the same change so recorded snapshots actually use it. diff --git a/dotfiles/agents/skills/disk-space-assessment/references/observed-heavy-hitters.md b/dotfiles/agents/skills/disk-space-assessment/references/observed-heavy-hitters.md new file mode 100644 index 00000000..6d74cf70 --- /dev/null +++ b/dotfiles/agents/skills/disk-space-assessment/references/observed-heavy-hitters.md @@ -0,0 +1,30 @@ +# Observed Disk-Usage Patterns + +Use these as hypotheses, not substitutes for a current snapshot. + +## Home and Project Trees + +- Rust `target/` directories under top-level projects, `.worktrees/*`, and `.claude/worktrees/*` have repeatedly consumed 25–100+ GiB in aggregate. +- `~/.cache/uv`, `~/.cache/pypoetry`, browser caches, Spotify, and Gradle caches have each reached multi-GiB sizes. +- `~/.local/share/picom/debug.log` has exceeded 15 GiB after verbose debugging or crashes. +- Trash and active build outputs are high-yield candidates but require lifecycle and process checks. + +## Service State + +- `/var/lib/private/gitea-runner` has exceeded 50 GiB and is invisible to an unprivileged root scan. +- `/var` has also been dominated by Docker, containers, Rancher, private state, and journals. +- Root accounting must include large sparse or allocated files such as `/swapfile` and ext4 reserved blocks; these may differ from ordinary directory totals. + +## Nix Store + +- A large store may be almost entirely live. In a 2026-07-10 audit, the Nix DB held 413.0 GiB logical NAR size with zero dead paths. +- The current system has directly pinned large ComfyUI model weights plus CUDA, GHC, Android, and Rust families. +- Hundreds of `.direnv` GC-root entries have retained more than 100 GiB outside the current-system closure. `railbird-mobile`, Railbird backend, taffybar, rocket-sense, and keepbook dev shells have been notable roots. +- `result*` symlinks, NixOS generations, and repo-local profiles can pin large closures. +- Closure sizes overlap. Compute unions or uniquely collectible sets; never sum per-profile closure sizes as a reclaim estimate. +- Physical `/nix/store` usage can be materially lower than logical NAR size after `nix-store --optimise` hardlinks duplicate files. + +## Volatility + +- Concurrent agent builds can move free space by tens of GiB during an assessment. +- Severe pressure may surface as SQLite I/O errors, missing compiler diagnostics, or opaque build-script failures rather than an explicit `ENOSPC` message. diff --git a/dotfiles/agents/skills/disk-space-cleanup/SKILL.md b/dotfiles/agents/skills/disk-space-cleanup/SKILL.md index fb3e47ae..f588c2fc 100644 --- a/dotfiles/agents/skills/disk-space-cleanup/SKILL.md +++ b/dotfiles/agents/skills/disk-space-cleanup/SKILL.md @@ -1,68 +1,75 @@ --- name: disk-space-cleanup -description: Investigate and safely reclaim disk space on this machine, especially on NixOS systems with heavy Nix, Rust/Haskell, Docker, and Podman usage. Use when disk is low, builds fail with no-space errors, /nix/store appears unexpectedly large, or the user asks for easy cleanup wins without deleting important data. +description: Safely reclaim disk space through reviewed, automated cleanup actions on this machine, especially for Nix, Rust build artifacts, containers, caches, and service state. Use when an assessment has identified cleanup candidates, the user asks to free space or automate remediation, or low-space recovery requires measured deletion of rebuildable data. --- # Disk Space Cleanup -Reclaim disk space with a safety-first workflow: investigate first, run obvious low-risk cleanup wins, then do targeted analysis for larger opportunities. +Act on evidence from `disk-space-assessment`. Automate guarded remediation, measure each action, and preserve user data and active work. -Bundled helpers: +Bundled resources: -- `scripts/rust_target_dirs.py`: inventory and guarded deletion for explicit Rust `target/` directories -- `references/rust-target-roots.txt`: machine-specific roots for Rust artifact scans -- `references/ignore-paths.md`: machine-specific excludes for `du`/`ncdu` +- `scripts/rust_target_dirs.py`: inventory and guarded deletion of explicit Cargo `target/` directories +- `references/rust-target-roots.txt`: machine-specific Rust scan roots +- `references/cleanup-patterns.md`: validated machine-specific remediation patterns -## Execution Default +## Required Input and Handoff -- Start with non-destructive investigation and quick sizing. -- Prioritize easy wins first (`nix-collect-garbage`, container prune, Cargo artifacts). -- Propose destructive actions with expected impact before running them. -- Run destructive actions only after confirmation, unless the user explicitly requests immediate execution of obvious wins. -- For Rust build artifacts, do not repeatedly ask for confirmation before deleting explicit directories literally named `target` after `rust_target_dirs.py delete` validates them. Cargo targets are rebuildable artifacts; when the user asks to clean Rust target directories, validate with the helper, delete with `--yes`, and report the reclaimed space. -- Capture new reusable findings by updating this skill before finishing. +Prefer a recent assessment containing baseline `df` output, reusable `ncdu` artifact paths, ranked candidates, and active-process caveats. If no reusable snapshot exists, use `disk-space-assessment` first. Even in an emergency, create a timestamped `safe_ncdu` snapshot of the affected filesystem or focused heavy root before deleting data whenever the scan can complete safely. + +Record before/after free space for every cleanup round. After material cleanup, create a new reusable snapshot through `disk-space-assessment` so the result can be compared without losing the pre-cleanup evidence. + +## Execution Policy + +- Present proposed actions with expected impact and risk before high-impact deletion. +- Run destructive actions only after confirmation unless the user explicitly requests immediate cleanup of obvious rebuildable artifacts. +- Prefer ownership-aware cleanup commands over raw recursive deletion. +- Check for active builds, services, mounts, and open files before deleting their state. +- Execute one category at a time and remeasure; stop when the user's space target is met. +- Never treat overlapping Nix closure sizes as additive reclaim estimates. + +For Rust artifacts, helper-validated explicit directories literally named `target` are rebuildable. When the user asks to clean Rust targets, inventory them, validate with the helper, delete with `--yes`, and report reclaimed space without repeatedly asking for confirmation. ## Workflow -1. Establish current pressure and biggest filesystems -2. Run easy cleanup wins -3. Inventory Rust build artifacts and clean the right kind of target -4. Investigate remaining heavy directories with `ncdu`/`du` -5. Investigate `/nix/store` roots when large toolchains still persist -6. Summarize reclaimed space and next candidate actions -7. Record new machine-specific ignore paths, Rust roots, or cleanup patterns in this skill +1. Read the assessment and record current free space. +2. Rank candidates by reclaim, reversibility, user impact, and confidence. +3. Check active use and run dry-run modes where available. +4. Execute the smallest sufficient cleanup round. +5. Remeasure free space and verify affected services or builds. +6. Produce a post-cleanup reusable snapshot after material changes. +7. Report actions, actual reclaim, residual candidates, and artifact paths. -## Step 1: Baseline - -Run a quick baseline before deleting anything: +## Baseline and Activity Checks ```bash df -h / df -h /home df -h /nix +ps aux | rg '(cargo|rustc|nix|docker|podman|gitea-runner)' || true ``` -Optionally add a quick home-level size snapshot: - -```bash -du -xh --max-depth=1 "$HOME" 2>/dev/null | sort -h -``` - -## Step 2: Easy Wins - -Use these first when the user wants fast, low-effort reclaiming: +Use `lsof`, service status, repository state, and worktree age when a candidate could be active. A directory that looks stale by mtime may still belong to a running build or concurrent agent. + +## Low-Risk Automated Wins + +Use dry-run or non-interactive privilege checks first: ```bash +nix-collect-garbage -d sudo -n nix-collect-garbage -d sudo -n docker system prune -a sudo -n podman system prune -a ``` Notes: -- Add `--volumes` only when the user approves deleting unused volumes. -- Re-check free space after each command to show impact. -- Prefer `sudo -n` first so cleanup runs fail fast instead of hanging on password prompts. -- If root is still tight after these, run app cache cleaners before proposing raw `rm -rf`: + +- Plain `nix-collect-garbage -d` may work through the daemon when `sudo` is unavailable. +- Add container `--volumes` only with explicit approval to delete unused volumes. +- Recheck `df` after each command. +- Fail fast on unavailable privilege; do not hang on a password prompt. + +Use application-aware cache cleaners when the assessment identifies those caches: ```bash uv cache clean @@ -71,200 +78,66 @@ yarn cache clean npm cache clean --force ``` -## Step 3: Rust Build Artifact Cleanup +Expect dependency redownloads. Close applications before clearing their profile caches. -Do not start with a blind `find ~ -name target` or with hard-coded roots that may miss worktrees. Inventory explicit `target/` directories first using the bundled helper and the machine-specific root list in `references/rust-target-roots.txt`. +## Rust Build Artifacts -Inventory the biggest candidates: +Inventory across configured project and worktree roots: ```bash python /srv/dotfiles/dotfiles/agents/skills/disk-space-cleanup/scripts/rust_target_dirs.py list --min-size 500M --limit 30 -``` - -Focus on stale targets only: - -```bash python /srv/dotfiles/dotfiles/agents/skills/disk-space-cleanup/scripts/rust_target_dirs.py list --min-size 1G --older-than 14 --output tsv ``` -Use `cargo-sweep` when the repo is still active and you want age/toolchain-aware cleanup inside a workspace: +For active workspaces, prefer age/toolchain-aware cleanup: ```bash nix run nixpkgs#cargo-sweep -- sweep -d -r -t 30 nix run nixpkgs#cargo-sweep -- sweep -r -t 30 -nix run nixpkgs#cargo-sweep -- sweep -d -r -i -nix run nixpkgs#cargo-sweep -- sweep -r -i ``` -Use direct `target/` deletion when inventory shows a discrete stale directory, especially for inactive repos or project-local worktrees. The helper only deletes explicit paths named `target` that are beneath configured roots and a Cargo project: +For inactive repos and stale worktrees, use guarded direct deletion: ```bash python /srv/dotfiles/dotfiles/agents/skills/disk-space-cleanup/scripts/rust_target_dirs.py delete /abs/path/to/target python /srv/dotfiles/dotfiles/agents/skills/disk-space-cleanup/scripts/rust_target_dirs.py delete /abs/path/to/target --yes ``` -Recommended sequence: +The helper rejects symlinks, paths outside configured roots, non-`target` names, and directories without a Cargo project above them. Re-run inventory after each round. Preserve any target involved in an active Cargo/Rust process. -1. Run `rust_target_dirs.py list` to see the largest `target/` directories across `~/Projects`, `~/org`, `/srv/dotfiles`, and other configured roots. -2. For active repos, prefer `cargo-sweep` from the workspace root. -3. For inactive repos, abandoned branches, and `.worktrees/*/target`, prefer guarded direct deletion of the explicit `target/` directory. -4. Re-run the list command after each deletion round to show reclaimed space. +If `python` is unavailable, run the helper with `nix run nixpkgs#python3 -- ...`. If a stale `~/.cargo/bin/cargo-sweep` shadows the NixOS binary, inspect `type -a cargo-sweep` before use. -Machine-specific note: +## Nix Remediation -- Project-local `.worktrees/*/target` directories are common cleanup wins on this machine and are easy to miss with the old hard-coded workflow. -- `cargo-sweep` is installed through the NixOS `code.nix` package set, but stale manually-installed binaries under `~/.cargo/bin` can shadow `/run/current-system/sw/bin/cargo-sweep`. If `cargo sweep` fails with a missing loader or `No such file or directory`, run `type -a cargo-sweep` and remove the stale `~/.cargo/bin/cargo-sweep` entry. -- `cargo-sweep sweep -i/--installed` can fail when `rustup toolchain list` contains stale toolchains whose `rustc` no longer exists. On this machine, `1.68.2-x86_64-unknown-linux-gnu` caused `failed to determine fingerprint ... 'rustc': No such file or directory`. -- `/home/imalison/Projects/codex/codex-rs/target` can be dominated by current-looking `target/debug/incremental` data that `cargo-sweep sweep -a` and `--maxsize` report as not removable. If it is stale and space pressure is high, use the guarded `rust_target_dirs.py delete ... --yes` workflow for that explicit target directory. -- `/home/imalison/Projects/hypr-workspace-history/target` is a small non-Cargo false positive; the guarded delete workflow correctly rejects it because there is no Cargo project above the directory. -- `nixos/imalison.nix` defines a daily user timer, `cargo-sweep-rust-targets.timer`, that runs `cargo-sweep sweep -r --hidden --maxsize 15GB` across `/home/imalison/Projects`, `/home/imalison/org`, and `/srv/dotfiles`. +Do not delete `/nix/store` paths directly. Act on retaining roots identified by the assessment: -## Step 4: Investigation with `ncdu` and `du` +- Remove stale project `.direnv` directories or `flake-profile-*` roots only after verifying the project/worktree is inactive. +- Remove stale `result*` symlinks when the referenced build output no longer needs to remain rooted. +- Remove obsolete system generations through Nix tooling. +- Run garbage collection after roots are removed and measure actual free-space change. +- Run `nix-store --optimise` when duplicate physical files are a demonstrated opportunity; it can be I/O intensive. -Avoid mounted or remote filesystems when profiling space. Load ignore patterns from `references/ignore-paths.md`. +Use `nix_store_audit --top 30` before and after a root-removal campaign. Treat its uniquely collectible estimates as planning evidence, not a promise of exact filesystem reclaim. -Use one-filesystem scans to avoid crossing mounts: +## Service and Cache Remediation -```bash -ncdu -x "$HOME" -sudo ncdu -x / -``` +Read `references/cleanup-patterns.md` before touching private service state, runner caches, logs, or ambiguous project artifacts. Preserve configuration, registration, credentials, SSH material, and active workspaces. -When excluding known noisy mountpoints: - -```bash -ncdu -x --exclude "$HOME/keybase" "$HOME" -sudo ncdu -x --exclude /keybase --exclude /var/lib/railbird / -``` - -If `ncdu` is missing, use: - -```bash -nix run nixpkgs#ncdu -- -x "$HOME" -``` - -For reusable, mount-safe snapshots on this machine, prefer the local wrapper: - -```bash -safe_ncdu / -sudo -n env HOME=/home/imalison safe_ncdu / -safe_ncdu /nix/store -safe_ncdu top ~/.cache/ncdu/latest-root.json.zst 30 /home/imalison -safe_ncdu open ~/.cache/ncdu/latest-root.json.zst -``` - -`safe_ncdu` writes compressed ncdu exports under `~/.cache/ncdu`, records the exclude list beside the export, excludes mounted descendants of the scan root, and supports follow-up `top` queries without rescanning. - -For quick, non-blocking triage on very large trees, prefer bounded probes: - -```bash -timeout 30s du -xh --max-depth=1 "$HOME/.cache" 2>/dev/null | sort -h -timeout 30s du -xh --max-depth=1 "$HOME/.local/share" 2>/dev/null | sort -h -``` - -Machine-specific heavy hitters seen in practice: - -- `~/.cache/uv` can exceed 20G and is reclaimable with `uv cache clean`. -- `~/.cache/pypoetry` can exceed 7G across artifacts, repository cache, and virtualenvs; inspect first, then use Poetry cache commands or targeted virtualenv removal. -- `~/.cache/google-chrome` can exceed 8G across multiple Chrome profiles; close Chrome before clearing profile cache directories. -- `~/.cache/spotify` can exceed 10G; treat as optional app-cache cleanup. -- `~/.gradle` can exceed 8G, mostly under `caches/`; prefer Gradle-aware cleanup and expect dependency redownloads. -- `~/.local/share/picom/debug.log` can grow past 15G when verbose picom debugging is enabled or crashes leave a stale log behind; if `picom` is not running, deleting or truncating the log is a high-yield low-risk win. -- `~/.local/share/Trash` can exceed several GB; empty only with user approval. -- `/var/lib/private/gitea-runner` can exceed 50G and is not visible to an unprivileged `ncdu /` scan; use `sudo -n env HOME=/home/imalison safe_ncdu /` when `/var` looks undercounted. - - Validated cleanup pattern: stop `gitea-runner-nix.service`, remove cache/work directories under `/var/lib/private/gitea-runner` (`.cache`, `.gradle`, `action-cache-dir`, `workspace`, stale nested `gitea-runner`, and nested `nix/.cache`/`nix/.local`), recreate `action-cache-dir`, `workspace`, and `.cache` owned by `gitea-runner:gitea-runner`, then restart the service. - - Preserve registration/config-like files such as `/var/lib/private/gitea-runner/nix/.runner`, `/var/lib/private/gitea-runner/nix/.labels`, `/var/lib/private/gitea-runner/.docker/config.json`, and SSH/Kube material. -- `~/Projects/*/target` directories can dominate home usage. Recent example candidates included stale `target/` directories under `scrobble-scrubber`, `http-client-vcr`, `http-client`, `subtr-actor`, `http-types`, `subtr-actor-py`, `sdk`, and `async-h1`. -- 2026-05-26 cleanup: deleting explicit Cargo-backed targets under `~/Projects/{keepbook,subtr-actor,rlru,rocket-sense,boxcars,rumno}` plus stale `subtr-actor/.worktrees/*/target` reclaimed about 65G by helper sizing and moved `/` from 100% used to 89% used. A final all-depth scan left no `~/Projects` Rust `target/` directories over 500M. -- 2026-05-26 cleanup: when `cargo test` is actively running in `~/Projects/subtr-actor`, leave `subtr-actor/target` alone and delete only inactive Cargo-backed targets. Deleting `keepbook`, `rlru`, `rocket-sense`, `rumno`, and stale `subtr-actor/.worktrees/*/target` reclaimed about 24.5G by helper sizing. -- 2026-05-26 cleanup: `~/Projects/nixpkgs/.worktrees/*/result` symlinks pinned several GiB of Nix closures, and clean registered nixpkgs worktrees were about 460M each. Removing stale `result` symlinks, running GC, and removing clean worktrees while preserving dirty ones moved `/` from 100% used to about 90% used. -- 2026-05-27 cleanup: under `~/Projects`, `hypr-workspace-history/target` can be a Rust-style build cache even though the guarded helper rejects it because no `Cargo.toml` is present; inspect and remove that explicit cache manually if present. Preserve `~/Projects/Hyprland/src/layout/target`, which is source code, not a build artifact. -- 2026-06-18 cleanup: deleting helper-validated Rust targets under `.worktrees/*/target` and `.claude/worktrees/*/target`, plus stale `~/Projects/lastfm-edit/target`, removed 24 target directories totaling 67.1G by helper sizing and moved `/` from 99% used to 90% used. Remaining large targets were top-level project caches under `keepbook`, `rlru`, `subtr-actor`, `rocket-sense`, `rocket-sense-pr-73-ci`, `rocket-sense-subtr-viewer`, `rocket-sense-controlled-plays`, and `boxcars`. -- 2026-06-23 cleanup: deleting helper-validated Rust targets over 100M under `~/Projects` removed 21 target directories totaling 59.6G by helper sizing; `rocket-sense/.worktrees/missed-event-capture/target` was recreated once during verification and deleted again after no active Cargo/Rust process was found. Clearing rebuildable Cargo registry/git caches and removing stale `~/.rustup/toolchains/1.68.2-x86_64-unknown-linux-gnu` moved `/` from 99% used with 17G free to 91% used with 81G free. In this shell, `python`, `python3`, `cargo`, `rustup`, `ps`, and `cargo-sweep` were not on `PATH`; using `nix run nixpkgs#python3 -- ...` worked for the helper, and `nix shell nixpkgs#procps nixpkgs#ripgrep --command bash -lc 'ps ...'` worked for process checks. -- 2026-06-28 cleanup: deleting helper-validated Rust targets over 500M under `~/Projects` removed 25 target directories totaling 102.6G by helper sizing and moved `/` from 98% used with 23G free to 86% used with 126G free. The largest wins were top-level `rlru`, `subtr-actor`, `rocket-sense`, `keepbook`, and `boxcars` targets plus many `rocket-sense` and `subtr-actor` `.worktrees/*/target` and `.claude/worktrees/*/target` directories. A follow-up scan over 100M found no remaining helper-validated Rust targets. -- 2026-07-03 cleanup (from within a sandboxed agent shell): `sudo -n` was unavailable ("no new privileges" flag), so `sudo nix-collect-garbage -d`, `docker`/`podman` prune were all out. Sudo-free wins still worked: plain `nix-collect-garbage -d` talked to the daemon and freed 17.0 GiB (2777 store paths), and helper-validated Rust target deletions under `~/Projects` (`keepbook`, `rlru`, and several `rocket-sense/.claude/worktrees/*/target` + `.worktrees/*/target`) reclaimed ~27G, moving `/home` from 100% used with 86M free to 95% with 47G free. Left `~/Projects/lastfm-edit/target` (17.4G) intact deliberately: it was in active use by the running session/build and a concurrent agent, and enough space had already been freed. Lesson: when only ~80M is free, disk-write failures surface as nondeterministic Rust build errors ("could not compile … due to 1 previous error" with no diagnostic text, `disk I/O error` on `~/.cache/nix/fetcher-cache-v4.sqlite`, and openssl-sys build-script failures) rather than obvious "No space left" messages. -- 2026-07-03 investigation: `/nix/store` was 503G while `nix-collect-garbage -d --dry-run` found only 42 dead paths (~1.0G by DB nar size), so pressure was mostly live roots. The current system pinned a 118G closure including ComfyUI, `comfyui-models`, `Qwen-Rapid-AIO-NSFW-v23.safetensors` (26.5G), and `lustifySDXLNSFW_v20-inpainting.safetensors` (6.5G). Other live contributors included `~/Projects/lastfm-edit/target` (36G, no active `cargo`/`rustc` process found), `~/Projects/rocket-sense/.claude/worktrees` (11G of duplicated agent worktrees), 12 `rocket-sense` `result*` symlinks pinning about 10G of direct Nix outputs, and 35 `.direnv/flake-profile-*` GC roots, with `railbird-mobile`'s shell closure around 31G. - -## Step 5: `/nix/store` Deep Dive - -When `/nix/store` is still large after GC, inspect root causes instead of deleting random paths. - -Useful commands: - -```bash -nix path-info -Sh /nix/store/* 2>/dev/null | sort -h | tail -n 50 -nix-store --gc --print-roots -``` - -Avoid `du -sh /nix/store` as a first diagnostic; it can be very slow on large stores. - -For repeated GHC/Rust toolchain copies: - -```bash -nix path-info -Sh /nix/store/* 2>/dev/null | rg '(ghc|rustc|rust-std|cargo)' -nix-store --gc --print-roots | rg '(ghc|rust)' -``` - -Resolve why a path is retained: - -```bash -/srv/dotfiles/dotfiles/lib/functions/find_store_path_gc_roots /nix/store/ -nix why-depends -``` - -Common retention pattern on this machine: - -- Many `.direnv/flake-profile-*` symlinks under `~/Projects` and worktrees keep `nix-shell-env`/`ghc-shell-*` roots alive. -- Old taffybar constellation repos under `~/Projects` can pin large Haskell closures through `.direnv` and `result` symlinks. Deleting `gtk-sni-tray`, `status-notifier-item`, `dbus-menu`, `dbus-hslogger`, and `gtk-strut` and then rerunning `nix-collect-garbage -d` reclaimed about 11G of store data in one validated run. -- `find_store_path_gc_roots` is especially useful for proving GHC retention: many large `ghc-9.10.3-with-packages` paths are unique per project, while the base `ghc-9.10.3` and docs paths are shared. -- NixOS system generations and a repo-root `nixos/result` symlink can pin multiple Android Studio and Android SDK versions. Check `/nix/var/nix/profiles/system-*-link`, `/run/current-system`, `/run/booted-system`, and `/srv/dotfiles/nixos/result` before assuming Android paths are pinned by project shells. -- `~/Projects/railbird-mobile/.direnv/flake-profile-*` can pin large Android SDK system images. Removing stale direnv profiles there is a more targeted first step than deleting Android store paths directly. -- 2026-05-27 Railbird GHC audit: the Railbird backend flake did not explicitly reference Haskell, but its dev shell had derivation-time GHC edges through `inputs.secrets.devShells.${system}.default -> agenix -> shellcheck -> ShellCheck -> ghc` and through `shell-packages.nix`'s `rdma-core -> pandoc-cli -> ghc`. Railbird Mobile had similar non-app-code GHC edges through `inputs.secrets`/`agenix` and `nixGLIntel -> shellcheck`. The `railbird/gql` and `railbird-mobile/src/gql` shells did not show GHC edges in their derivation graphs, only Rust/Cargo build tooling from packages such as `just`. -- For a repeatable `/nix/store` `ncdu` snapshot without driving the TUI, export and inspect it: - -```bash -ncdu -0 -x -c -o /tmp/nix-store.ncdu.json.zst /nix/store -zstdcat /tmp/nix-store.ncdu.json.zst | jq 'def sumd: if type=="array" then ((.[0].dsize // 0) + ([.[1:][] | sumd] | add // 0)) elif type=="object" then (.dsize // 0) else 0 end; .[3] | sumd' -``` - -- `nix-store --gc --print-dead` plus the Nix SQLite database is a fast way to estimate immediate GC wins before deleting anything: - -```bash -nix-store --gc --print-dead > /tmp/nix-dead-paths.txt -printf '%s\n' '.mode list' '.separator |' 'create temp table dead(path text);' \ - '.import /tmp/nix-dead-paths.txt dead' \ - 'select count(*), sum(narSize) from ValidPaths join dead using(path);' \ - | nix shell nixpkgs#sqlite --command sqlite3 /nix/var/nix/db/db.sqlite -``` - -- Quantify before acting: - -```bash -find ~/Projects -type l -path '*/.direnv/flake-profile-*' | wc -l -find ~/Projects -type d -name .direnv | wc -l -nix-store --gc --print-roots | rg '/\\.direnv/flake-profile-' | awk -F' -> ' '{print $1 \"|\" $2}' \ - | while IFS='|' read -r root target; do \ - nix-store -qR \"$target\" | rg '^/nix/store/.+-ghc-[0-9]'; \ - done | sort | uniq -c | sort -nr | head -``` - -- If counts are high and the projects are inactive, propose targeted `.direnv` cleanup for user confirmation. +Prefer service-aware sequences: stop the service, clean only validated cache/work paths, recreate required directories with correct ownership, restart, and verify health. ## Safety Rules -- Do not delete user files directly unless explicitly requested. -- Prefer cleanup tools that understand ownership/metadata (`nix`, `docker`, `podman`, `cargo-sweep`) over `rm -rf`. -- For Rust build artifacts, deleting an explicit directory literally named `target` is acceptable when it is discovered and validated by the bundled helper; Cargo will rebuild it. Do not double-check with the user after helper validation when the active request is Rust target cleanup. -- Present a concise “proposed actions” list before high-impact deletes. -- If uncertain whether data is needed, stop at investigation and ask. +- Do not delete user documents, source code, credentials, volumes, or unknown data without explicit approval. +- Never delete a path solely because its name resembles a cache or build directory. +- Do not clean targets used by running builds or concurrent agents. +- Preserve dirty worktrees and registration/configuration files. +- Stop when evidence is insufficient and return to `disk-space-assessment`. -## Learning Loop (Required) +## Learning Loop -Treat this skill as a living playbook. +After each cleanup: -After each disk cleanup task: - -1. Add newly discovered mountpoints or directories to ignore in `references/ignore-paths.md`. -2. Add newly discovered Rust repo roots in `references/rust-target-roots.txt`. -3. Add validated command patterns or caveats discovered during the run to this `SKILL.md`. -4. Keep instructions practical and machine-specific; remove stale guidance. +1. Add stable, validated remediation knowledge to `references/cleanup-patterns.md`. +2. Add new Rust repository roots to `references/rust-target-roots.txt`. +3. Update assessment-side diagnostic patterns separately; do not turn this skill into a usage-history log. +4. Remove stale guidance and keep automation guarded and reproducible. diff --git a/dotfiles/agents/skills/disk-space-cleanup/agents/openai.yaml b/dotfiles/agents/skills/disk-space-cleanup/agents/openai.yaml index 7f76fbcb..f24d57ec 100644 --- a/dotfiles/agents/skills/disk-space-cleanup/agents/openai.yaml +++ b/dotfiles/agents/skills/disk-space-cleanup/agents/openai.yaml @@ -1,3 +1,4 @@ interface: display_name: "Disk Space Cleanup" - short_description: "Find safe disk-space wins on NixOS hosts" + short_description: "Automate safe, measured disk-space cleanup" + default_prompt: "Use $disk-space-cleanup to act on assessed disk-space candidates and verify reclaimed space." diff --git a/dotfiles/agents/skills/disk-space-cleanup/references/cleanup-patterns.md b/dotfiles/agents/skills/disk-space-cleanup/references/cleanup-patterns.md new file mode 100644 index 00000000..e496ad31 --- /dev/null +++ b/dotfiles/agents/skills/disk-space-cleanup/references/cleanup-patterns.md @@ -0,0 +1,40 @@ +# Validated Cleanup Patterns + +Use only after a current assessment confirms the path is significant. + +## Rust and Worktrees + +- Stale Cargo-backed targets in top-level projects, `.worktrees/*/target`, and `.claude/worktrees/*/target` have repeatedly reclaimed 25–100+ GiB. +- Preserve targets used by active builds. Concurrent agents may recreate deleted outputs immediately. +- `hypr-workspace-history/target` has been a removable Rust-style cache without a nearby `Cargo.toml`, so the guarded helper rejects it; inspect manually. +- Preserve `~/Projects/Hyprland/src/layout/target`, which is source code rather than build output. +- Clean registered git worktrees only after preserving dirty ones. Stale nixpkgs worktree `result` symlinks may pin Nix closures. + +## Nix Roots and Store Optimization + +- Removing conservatively stale `.direnv` roots followed by `nix-store --gc` has reclaimed about 15 GiB in a validated run. +- With `auto-optimise-store = false`, `nix-store --optimise` once reclaimed about 60 GiB by hardlinking duplicate store files. Measure current conditions before assuming similar impact. +- Old taffybar constellation repos and `railbird-mobile` direnv profiles have pinned large Haskell or Android closures. +- NixOS generations and `/srv/dotfiles/nixos/result` can pin Android Studio and SDK versions. +- A stale rustup toolchain can break `cargo-sweep --installed`; remove it only after confirming the toolchain is unusable and unwanted. + +## Gitea Runner + +For `/var/lib/private/gitea-runner`, a validated sequence is: + +1. Stop `gitea-runner-nix.service`. +2. Remove only assessed cache/work directories such as `.cache`, `.gradle`, `action-cache-dir`, `workspace`, stale nested `gitea-runner`, and nested `nix/.cache` or `nix/.local`. +3. Recreate `action-cache-dir`, `workspace`, and `.cache` owned by `gitea-runner:gitea-runner`. +4. Restart the service and verify it. + +Preserve `.runner`, `.labels`, `.docker/config.json`, SSH material, Kubernetes material, and other registration/configuration state. + +## Logs and App Caches + +- Delete or truncate `~/.local/share/picom/debug.log` only after confirming picom is not actively writing it and correcting the verbose logging cause. +- Empty Trash only with explicit approval. +- Close Chrome, Spotify, Gradle daemons, and similar applications before targeted cache cleanup. + +## Severe Pressure + +When only megabytes remain, run the smallest safe cleanup first. Build tools and SQLite may fail opaquely before reporting `ENOSPC`. Avoid commands that need substantial temporary disk space until a safe margin has been restored. diff --git a/dotfiles/agents/skills/disk-space-cleanup/references/ignore-paths.md b/dotfiles/agents/skills/disk-space-cleanup/references/ignore-paths.md deleted file mode 100644 index e55dd730..00000000 --- a/dotfiles/agents/skills/disk-space-cleanup/references/ignore-paths.md +++ /dev/null @@ -1,31 +0,0 @@ -# Ignore Paths for Disk Investigation - -Use this file to track mountpoints or directories that should be excluded from `ncdu`/`du` scans because they are remote, special-purpose, or noisy. - -## Known Ignores - -- `$HOME/keybase` -- `$HOME/.cache/keybase` -- `$HOME/.local/share/keybase` -- `$HOME/.config/keybase` -- `/keybase` -- `/var/lib/railbird` -- `/run/user/*/doc` (FUSE portal mount; machine-specific example observed: `/run/user/1004/doc`) - -## Discovery Commands - -List mounted filesystems and spot special mounts: - -```bash -findmnt -rn -o TARGET,FSTYPE,SOURCE -``` - -Target likely remote/special mounts: - -```bash -findmnt -rn -o TARGET,FSTYPE,SOURCE | rg '(keybase|fuse|rclone|s3|railbird)' -``` - -## Maintenance Rule - -When a disk cleanup run encounters a mount or path that should be ignored in future runs, add it here immediately with a short note. diff --git a/dotfiles/lib/functions/nix_store_audit b/dotfiles/lib/functions/nix_store_audit new file mode 100755 index 00000000..0ffb101e --- /dev/null +++ b/dotfiles/lib/functions/nix_store_audit @@ -0,0 +1,192 @@ +#!/usr/bin/env bash + +set -euo pipefail + +top=20 + +usage() { + cat <<'EOF' +Usage: nix_store_audit [--top N] + +Read-only audit of /nix/store size and retention. Reports: + - filesystem pressure, valid/dead paths, and system closure sizes + - largest individual store paths + - broad ecosystem totals (not additive) + - direnv closure footprint outside the current system + - largest direnv profile and result-link closures + +Sizes from the Nix database are logical NAR sizes, not filesystem block usage. +EOF +} + +while (($#)); do + case "$1" in + --top) + top="${2:?--top requires a positive integer}" + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + printf 'Unknown argument: %s\n' "$1" >&2 + usage >&2 + exit 2 + ;; + esac +done + +if [[ ! "$top" =~ ^[1-9][0-9]*$ ]]; then + printf '%s\n' '--top must be a positive integer' >&2 + exit 2 +fi + +for command_name in awk comm cut date df find mktemp nix nix-store sort wc xargs; do + if ! command -v "$command_name" >/dev/null 2>&1; then + printf 'Required command is missing: %s\n' "$command_name" >&2 + exit 1 + fi +done + +db=/nix/var/nix/db/db.sqlite +if [[ ! -r "$db" ]]; then + printf 'Cannot read the Nix database: %s\n' "$db" >&2 + exit 1 +fi + +if command -v sqlite3 >/dev/null 2>&1; then + sqlite_bin=$(command -v sqlite3) +else + sqlite_bin=$(find /nix/store -mindepth 3 -maxdepth 3 -type f -path '*/bin/sqlite3' -print -quit 2>/dev/null || true) +fi + +if [[ -n "${sqlite_bin:-}" ]]; then + sql() { + "$sqlite_bin" -readonly "$db" "$@" + } +else + printf 'sqlite3 is not installed or already present in /nix/store; using a temporary nix shell.\n' >&2 + sql() { + nix shell nixpkgs#sqlite --command sqlite3 -readonly "$db" "$@" + } +fi + +tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/nix-store-audit.XXXXXX") +trap 'rm -rf "$tmpdir"' EXIT + +roots="$tmpdir/roots.txt" +current="$tmpdir/current.txt" +booted="$tmpdir/booted.txt" +dead="$tmpdir/dead.txt" +direnv_targets="$tmpdir/direnv-targets.txt" +direnv_closure="$tmpdir/direnv-closure.txt" +direnv_outside="$tmpdir/direnv-outside-current.txt" +booted_outside="$tmpdir/booted-outside-current.txt" +ranked_roots="$tmpdir/ranked-roots.txt" +ranked_root_pairs="$tmpdir/ranked-root-pairs.txt" +ranked_targets="$tmpdir/ranked-targets.txt" +ranked_target_sizes="$tmpdir/ranked-target-sizes.txt" + +nix-store --gc --print-roots 2>/dev/null > "$roots" +nix-store --gc --print-dead 2>/dev/null | sort -u > "$dead" +nix-store -qR /run/current-system 2>/dev/null | sort -u > "$current" +nix-store -qR /run/booted-system 2>/dev/null | sort -u > "$booted" +comm -23 "$booted" "$current" > "$booted_outside" + +awk -F' -> ' ' + { + gsub(/^"|"$/, "", $1) + gsub(/^"|"$/, "", $2) + if ($1 ~ /\.direnv\//) print $2 + } +' "$roots" | sort -u > "$direnv_targets" + +if [[ -s "$direnv_targets" ]]; then + xargs -r nix-store -qR < "$direnv_targets" 2>/dev/null | sort -u > "$direnv_closure" +else + : > "$direnv_closure" +fi +comm -23 "$direnv_closure" "$current" > "$direnv_outside" + +printf '# Nix store audit\n\n' +printf 'Generated: %s\n\n' "$(date --iso-8601=seconds)" + +printf '## Pressure and roots\n\n' +df -h / | awk 'NR == 1 || NR == 2' +printf '\n' +sql ".mode list" ".separator |" \ + "create temp table dead(path text);" \ + ".import $dead dead" \ + "select count(*) || ' valid paths | ' || printf('%.1f GiB logical NAR size', sum(narSize)/1073741824.0) from ValidPaths;" \ + "select count(*) || ' dead paths | ' || printf('%.2f GiB immediately GC-eligible', coalesce(sum(narSize), 0)/1073741824.0) from ValidPaths join dead using(path);" + +current_size=$(nix path-info -Sh /run/current-system 2>/dev/null | awk '{print $(NF-1) " " $NF}') +booted_size=$(nix path-info -Sh /run/booted-system 2>/dev/null | awk '{print $(NF-1) " " $NF}') +printf '%s current-system closure\n' "$current_size" +printf '%s booted-system closure\n' "$booted_size" + +sql ".mode list" ".separator |" \ + "create temp table x(path text);" \ + ".import $booted_outside x" \ + "select printf('%.2f GiB booted-only versus current', coalesce(sum(narSize), 0)/1073741824.0) from ValidPaths join x using(path);" + +awk -F' -> ' ' + { + gsub(/^"|"$/, "", $1) + if ($1 ~ /^\/proc\//) kind="runtime /proc roots" + else if ($1 ~ /\.direnv\/flake-profile-/) kind="direnv profile roots" + else if ($1 ~ /\.direnv\//) kind="other direnv roots" + else if ($1 ~ /\/result[^/]*$/) kind="result links" + else if ($1 ~ /home-manager/) kind="home-manager roots" + else if ($1 ~ /^\/nix\/var\/nix\/profiles|^\/run\//) kind="system/profile roots" + else kind="other roots" + count[kind]++ + } + END { for (kind in count) printf "%7d %s\n", count[kind], kind } +' "$roots" | sort -nr + +printf '\n## Largest individual store paths\n\n' +sql "select printf('%7.2f GiB', narSize/1073741824.0) || ' ' || path from ValidPaths order by narSize desc limit $top;" + +printf '\n## Broad named-path totals\n\n' +printf 'These categories are filename-based lower bounds and overlap; do not add them.\n\n' +sql "select 'model weights', printf('%.1f GiB',sum(narSize)/1073741824.0), count(*) from ValidPaths where path like '%.safetensors' union all select 'Android', printf('%.1f GiB',sum(narSize)/1073741824.0), count(*) from ValidPaths where lower(path) like '%android%' union all select 'CUDA/cuDNN/cuBLAS', printf('%.1f GiB',sum(narSize)/1073741824.0), count(*) from ValidPaths where lower(path) like '%cuda%' or lower(path) like '%cudnn%' or lower(path) like '%cublas%' union all select 'GHC', printf('%.1f GiB',sum(narSize)/1073741824.0), count(*) from ValidPaths where lower(path) like '%-ghc-%' or lower(path) like '%-ghc-with%' union all select 'Rust toolchains', printf('%.1f GiB',sum(narSize)/1073741824.0), count(*) from ValidPaths where lower(path) like '%-rustc-%' or lower(path) like '%-rust-std-%' or lower(path) like '%-rust-stable-%' or lower(path) like '%-cargo-%' union all select 'source snapshots', printf('%.1f GiB',sum(narSize)/1073741824.0), count(*) from ValidPaths where path like '%-source';" + +printf '\n## Current system versus direnv-only footprint\n\n' +printf '%s unique direnv root targets\n' "$(wc -l < "$direnv_targets")" +sql ".mode list" ".separator |" \ + "create temp table c(path text); create temp table d(path text);" \ + ".import $current c" \ + ".import $direnv_closure d" \ + "select printf('%.2f GiB current-system closure by DB NAR size', coalesce(sum(narSize), 0)/1073741824.0) from ValidPaths join c using(path);" \ + "select printf('%.2f GiB union of direnv-rooted closures', coalesce(sum(narSize), 0)/1073741824.0) from ValidPaths join d using(path);" \ + "delete from d;" \ + ".import $direnv_outside d" \ + "select printf('%.2f GiB direnv closure outside current system (upper bound, other roots may retain paths)', coalesce(sum(narSize), 0)/1073741824.0) from ValidPaths join d using(path);" \ + "select 'direnv-only Android', printf('%.1f GiB',sum(narSize)/1073741824.0) from ValidPaths join d using(path) where lower(path) like '%android%' union all select 'direnv-only CUDA family', printf('%.1f GiB',sum(narSize)/1073741824.0) from ValidPaths join d using(path) where lower(path) like '%cuda%' or lower(path) like '%cudnn%' or lower(path) like '%cublas%' union all select 'direnv-only GHC', printf('%.1f GiB',sum(narSize)/1073741824.0) from ValidPaths join d using(path) where lower(path) like '%-ghc-%' or lower(path) like '%-ghc-with%' union all select 'direnv-only Rust', printf('%.1f GiB',sum(narSize)/1073741824.0) from ValidPaths join d using(path) where lower(path) like '%-rustc-%' or lower(path) like '%-rust-std-%' or lower(path) like '%-rust-stable-%' or lower(path) like '%-cargo-%' union all select 'direnv-only sources', printf('%.1f GiB',sum(narSize)/1073741824.0) from ValidPaths join d using(path) where path like '%-source';" + +awk -F' -> ' ' + { + gsub(/^"|"$/, "", $1) + gsub(/^"|"$/, "", $2) + if ($1 ~ /\.direnv\/flake-profile-/ || $1 ~ /\/result[^/]*$/) print $1 "|" $2 + } +' "$roots" | sort -u > "$ranked_root_pairs" + +cut -d'|' -f2 "$ranked_root_pairs" | sort -u > "$ranked_targets" +if [[ -s "$ranked_targets" ]]; then + xargs -r nix path-info -Sh < "$ranked_targets" 2>/dev/null \ + | awk '{print $1 "|" $(NF-1) $NF}' > "$ranked_target_sizes" +else + : > "$ranked_target_sizes" +fi + +awk -F'|' ' + NR == FNR { size[$1]=$2; next } + $2 in size { print size[$2] "\t" $1 } +' "$ranked_target_sizes" "$ranked_root_pairs" > "$ranked_roots" + +printf '\n## Largest direnv profile and result-link closures\n\n' +printf 'These closure sizes heavily overlap; do not add them.\n\n' +sort -hr "$ranked_roots" | head -n "$top" || true diff --git a/dotfiles/lib/functions/safe_ncdu b/dotfiles/lib/functions/safe_ncdu index 1a9a6beb..69aba5bb 100755 --- a/dotfiles/lib/functions/safe_ncdu +++ b/dotfiles/lib/functions/safe_ncdu @@ -10,6 +10,7 @@ Usage: Creates a compressed ncdu export while avoiding mounted descendants of ROOT. Default ROOT is /. Default OUTPUT is ~/.cache/ncdu/safe-ncdu--.json.zst. +Records matching .excludes and .meta sidecars for reproducible analysis. EOF } @@ -90,9 +91,10 @@ function _safe_ncdu_scan { shift done - _safe_ncdu_require ncdu findmnt realpath awk sed date mkdir tee sort uniq || return 1 + _safe_ncdu_require ncdu findmnt realpath awk sed date mkdir dirname tee sort uniq hostname || return 1 - local root_real root_name out_dir latest excludes_file + local root_real root_name out_dir latest excludes_file meta_file + local started_at finished_at host ncdu_version scan_status exit_code root_real="$(realpath -m "$root")" || return 1 root_name="$(_safe_ncdu_root_name "$root_real")" out_dir="$HOME/.cache/ncdu" @@ -101,10 +103,29 @@ function _safe_ncdu_scan { if [[ -z "$output" ]]; then output="$out_dir/safe-ncdu-${root_name}-$(date +%Y%m%d-%H%M%S).json.zst" fi + output="$(realpath -m "$output")" || return 1 + mkdir -p "$(dirname "$output")" excludes_file="${output}.excludes" + meta_file="${output}.meta" _safe_ncdu_excludes "$root_real" | sort -u | tee "$excludes_file" >/dev/null + started_at="$(date --iso-8601=seconds)" + host="$(hostname)" + ncdu_version="$(ncdu --version 2>&1 | head -n 1)" + { + echo "format_version=1" + echo "status=in_progress" + echo "root=$root_real" + echo "started_at=$started_at" + echo "finished_at=" + echo "hostname=$host" + echo "ncdu_version=$ncdu_version" + echo "snapshot=$output" + echo "excludes=$excludes_file" + echo "exit_code=" + } >| "$meta_file" + local -a exclude_args local exclude exclude_args=() @@ -115,12 +136,34 @@ function _safe_ncdu_scan { echo "safe_ncdu: scanning $root_real" echo "safe_ncdu: writing $output" echo "safe_ncdu: excludes recorded in $excludes_file" - ncdu -0 -x -c "${exclude_args[@]}" -o "$output" "$root_real" || return $? + echo "safe_ncdu: metadata recorded in $meta_file" + ncdu -0 -x -c "${exclude_args[@]}" -o "$output" "$root_real" + exit_code=$? + finished_at="$(date --iso-8601=seconds)" + if [[ $exit_code -eq 0 ]]; then + scan_status="complete" + else + scan_status="failed" + fi + { + echo "format_version=1" + echo "status=$scan_status" + echo "root=$root_real" + echo "started_at=$started_at" + echo "finished_at=$finished_at" + echo "hostname=$host" + echo "ncdu_version=$ncdu_version" + echo "snapshot=$output" + echo "excludes=$excludes_file" + echo "exit_code=$exit_code" + } >| "$meta_file" + [[ $exit_code -eq 0 ]] || return $exit_code latest="$out_dir/latest-${root_name}.json.zst" ln -sfn "$output" "$latest" echo "safe_ncdu: latest symlink $latest" ln -sfn "$excludes_file" "${latest}.excludes" + ln -sfn "$meta_file" "${latest}.meta" } function _safe_ncdu_open {