Failure Log & Post-Mortems
A transparent log of real engineering mistakes, root-cause analyses, immediate fixes, and operational lessons.
Spent two full days debugging CUDA installation errors and driver mismatch errors on Linux.
I assumed my laptop GPU supported CUDA acceleration without checking its architecture specifications first.
Ran `lspci` and verified hardware parameters. Switched strategy to CPU-quantized GGUF models via Ollama instead.
"Always verify physical hardware compatibility before spending hours configuring low-level driver stack dependencies."
Accidentally executed a database drop command on a local PostgreSQL development instance containing active schema test data.
Had multiple terminal tabs open and ran a destructive reset script without verifying which terminal session was active.
Rebuilt schema migrations script from Git version control and established explicit `.env` database safety flags.
"Automate database migrations and seeding scripts through code rather than manually executing destructive raw SQL in open terminals."
Broke Linux boot process three times in one week, triggering `grub rescue>` prompt on startup.
Attempted manual kernel parameter tweaks and partition table resizing without backing up the EFI boot partition.
Booted from a live Arch USB installer, chrooted into system drive, restored GRUB config, and updated `/etc/fstab` UUIDs.
"Understand bootloaders before tweaking kernel flags, and always keep a bootable Linux USB drive within arm's reach."
Pushed an API secret key in an early commit to a public GitHub repository.
Relied on temporary manual file exclusions instead of maintaining a proper `.gitignore` file before git commit.
Revoked and rotated the exposed API key immediately, purged commit history using git filter-repo, and added automated pre-commit secret scanners.
"Secrets belong in environment variables (`.env`) from line one of a project. Never commit unvetted code to public branches."
Attempted to load an unquantized 13B parameter model into 8GB of system RAM, causing kernel panic and system lockup.
Underestimated raw FP16 model memory overhead and system swap memory limitations.
Implemented strict 4-bit (Q4_K_M) quantization pipelines and restricted model size to 8B parameters max on host RAM.
"Calculate memory budgets before initializing intensive workloads. Operating systems will enforce hard limits when RAM runs out."