Simple AHCI (SATA) driver w/ synchronous read interface

Initialize and configure the AHCI device. This exposes a very simple
interface to synchronously (blocking) read N contiguous sectors from
disk into a memory buffer. This is probably not a complete setup but
it's good enough to work with the QEMU emulated disk.

Makefile

@@ -44,7 +44,7 @@ CXXFLAGS:=$(_CFLAGS) \
 	-fno-exceptions \
 	-I$(KERNEL_SRC_DIR) \
 	-I$(KERNEL_SRC_DIR)/arch/$(ARCH)
-QEMU_FLAGS:=-m 4G -M q35
+QEMU_FLAGS:=-nodefaults -m 4G -M q35 -vga std
 
 # libgcc contains some useful logic that may be used implicitly by
 # gcc/clang (e.g., __divdi3 for unsigned long long operations on a
@@ -153,6 +153,10 @@ ifneq ($(KVM),)
 override QEMU_FLAGS+=-accel kvm
 endif
 
+ifneq ($(QMONITOR),)
+override QEMU_FLAGS+=-monitor unix:qemu-monitor-socket,server,nowait
+endif
+
 ################################################################################
 # Bootloader-specific config
 ################################################################################
@@ -244,7 +248,7 @@ $(BOOTABLE_DISK): $(BOOTLOADER) $(KERNEL_FS)
 $(BOOTABLE_DISK_TEST): $(BOOTLOADER) $(KERNEL_FS_TEST)
 	scripts/install_bootloader.py -b $(BOOTLOADER) -k $(KERNEL_FS_TEST) -o $@
 
-.PHONY: docs run runi gdb clean cleanall
+.PHONY: docs run runi gdb qmonitor clean cleanall
 docs:
 	doxygen
 
@@ -268,6 +272,10 @@ gdb:
 	    -ex 'set confirm on' \
 	    -ex 'set print asm-demangle on'
 
+# Attach to QEMU monitor (from https://unix.stackexchange.com/a/476617):
+qmonitor:
+	socat -,echo=0,icanon=0 unix-connect:qemu-monitor-socket
+
 # Note that `make clean` will only clean the build directory for the
 # current build variant. To remove all build variants, use `make
 # cleanall`.

README.md

@@ -4,6 +4,48 @@ now that I have a slightly better idea of what the bootloader does.
 
 Focusing on x86 first, will maybe update this to be for x86_64 later.
 
+High-level roadmap for HmmOS:
+- Bootloader
+  - [X] Simple boot protocol with bootloader requests
+  - [X] Kernel loader
+	- [X] With FAT32 support
+	- [ ] With ELF support
+- Memory management
+  - [X] Page frame allocator
+  - [ ] Slab allocator
+  - [ ] vmalloc
+- Userspace processes
+  - [X] Kernel threads
+  - [X] Thread scheduling
+  - [ ] ELF loader
+  - [ ] Syscalls
+  - [ ] Shared libraries
+  - Sample programs
+    - [ ] Shell
+	- [ ] Simple libc implementation
+- Filesystem
+  - Filesystem drivers
+    - [ ] FAT32
+	- [ ] ext2
+  - [ ] VFS layer
+- Device drivers (very simple)
+  - [X] Serial port
+  - [X] BIOS text mode display
+  - [X] ACPI
+  - [ ] Keyboard
+  - [ ] AHCI (SATA)
+  - [ ] Graphics
+  - [ ] UDP
+  - [ ] TCP
+- SMP
+  - [ ] Locking primitives
+- Tooling
+  - [X] Debugging in GDB
+  - [X] clang+gcc tooling
+  - [X] Testing in QEMU w/ optional KVM virtualization
+  - [X] Unit test framework
+  - [ ] Kernel module system with dependencies
+
 ## Setup
 This project is mainly built with clang/LLVM toolchains in mind. This
 includes `clang`, `lld`, `clangd`, and `clang-format`. You'll need at

notes/fs.md

@@ -8,7 +8,7 @@ itself and have a purely in-memory layout.
 The initial goals for the HmmOS filesystem are, in order:
 1. write the HmmOS kernel and userspace programs to a FAT filesystem
 2. implement basic FAT support in the bootloader (to find the kernel)
-3. write a basic SATA driver that can read/write one block at a time
+3. write a basic disk driver that can read/write one block at a time
 4. implement FAT support in the kernel to load userspace programs
 5. (later) write an ELF loader and store the binaries in ELF form
    rather than flat binaries
@@ -59,13 +59,44 @@ filesystem similar to the kernel to be dynamically loaded at runtime.
 
 Let's assume the kernel image is written in the root directory with
 name "KERNEL.BIN". We need to be able to locate and read this file
-into memory.
+into memory. For simplicity, the bootloader will maintain its own
+minimal FAT32 implementation for the sole purpose of loading the
+kernel binary.
+
+## disk interfaces
+
+(may split this out into another notes file if this gets too long)
+
+- **SATA (Serial AT Attachment)**: common physical interface for hard
+  drives. Can be accessed using the ATA (older) or AHCI (newer)
+  software interfaces.
+- **PATA (Parallel AT Attachment)**: older than SATA, can be accessed
+  using the ATA interface.
+- **IDE (Integrated Drive Electronics)**: original name for PATA
+- **ATAPI (ATA Packet Interface)**: extensions to ATA to support more
+  devices other than hard drives (e.g. optical drives, SCSI devices)
+- **AHCI (Advanced Host Controller Interface)**: encapsulates SATA
+  devices and provides a standard PCI interface. Supports all of the
+  ATA functionality and more.
+- **PCI (Peripheral Component Interconnect)**: a high-performance bus
+  for peripherals; can be used to enumerate devices.
+- **HBA (Host Bus Adapter)**: the AHCI controller. Usually this refers
+  to a physical extension card or onboard chip that connects external
+  storage to a system bus. NICs are similar for networking.
+- **FIS (Frame Information Structure)**: The packets used to
+  communicate between the CPU and the HBA.
+
+For HmmOS, we'll start with a simple AHCI driver. We can start with
+PIO-mode (slow but simple) and then implement DMA.
 
 ### TODO
-- [X] write the HmmOS kernel to a FAT filesystem
-- [ ] implement basic FAT support in the bootloader
-- [ ] write a basic SATA driver that can read/write one block at a
-      time
-- [ ] implement FAT support in the kernel to load userspace programs
-- [ ] (later) write an ELF loader and store the binaries in ELF form
-      rather than flat binaries
+- [X] write the HmmOS kernel to a FAT32 filesystem
+- [X] implement basic FAT32 support in the bootloader
+- [ ] write a basic AHCI driver in the kernel
+- [ ] implement FAT32 support in the kernel to load userspace programs
+
+Further TODO items
+- [ ] write an ELF loader and store the binaries in ELF format
+- [ ] VFS layer
+- [ ] ext2 support
+- [ ] NVMe support

notes/qemu.md

@@ -0,0 +1,73 @@
+# qemu notes
+I had a few notes about QEMU scattered throughout, now that we've
+gotten a little more intimate I think this deserves its own page
+
+## x86 quirks
+Both of the following make my life easier when running in QEMU, but
+make it harder for me to handle the expected case for real hardware.
+
+- *A20 is set on boot*: To be fair, the OSDev wiki states that this is
+  very non-standardized and there are a myriad of ways to
+  enable/disable the A20 line depending on the hardware. In QEMU's
+  case, [it was already set on
+  boot](https://forum.osdev.org/viewtopic.php?f=1&t=26256).
+- *No data segment offset limit checks*: In real mode, I noticed I can
+  write to 0x0000:0xB8000 without having to use unreal
+  mode. Similarly, when in protected mode/unreal mode, I was unable to
+  trigger the segment check (which should cause a
+  [GPF](https://wiki.osdev.org/Exceptions#General_Protection_Fault))
+  even if I manually set the limits low. It seems that QEMU doesn't
+  perform the limit checks ([at least as of
+  2019](https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg06518.html)).
+  I do get GPFs for the code segment though.
+- *PCI vendor ID 0x1234*: According to [this blog
+  post](https://web.archive.org/web/20200416081308/https://www.kraxel.org/blog/2020/01/qemu-pci-ids/). Indeed
+  0x1234 doesn't show up on the [list of PCI
+  vendors](https://devicehunt.com/all-pci-vendors).
+
+## QEMU/KVM bugs/weirdness
+- See "hmm.md#slow\_memory\_speed" for a QEMU emulation performance issue.
+- See "hmm.md#KVM memory invalidation heisenbug" for what seems to be a
+KVM bug.
+- See "demos.md#locking" which demonstrates that increment (a++)
+  appears to be atomic in QEMU emulation
+
+Probably if I tried hard enough both of these can be linked to
+QEMU/KVM code but I haven't found the time to look deeper.
+
+## default QEMU setup
+The default OS setup uses a Q35 architecture (based on [Intel's ~2009
+ICH9 architecture](https://wiki.qemu.org/Features/Q35)), which treats
+drivers as AHCI/SATA by default rather than IDE.
+
+We use QEMU emulation by default unless you pass `-accel kvm` (`make
+run KVM=1`). It's good to test with both emulation modes to shake out
+differences in emulation (just as running with both GCC and clang is a
+good idea).
+
+## useful debugging flags
+These all feature in some form in the Makefile:
+
+- `-serial stdio` (default): useful for emulating a serial port and
+  receiving output on the host machine
+- `-display none -serial stdio` (`make run TEST=...`): useful for
+  running tests since the OS doesn't take any input
+- `-d int,cpu_reset -no-reboot` (`make run SHOWINT=1`): logging of
+  interrupts and cpu reset (i.e. during a triple fault). `-no-reboot`
+  prevents automatic rebooting during triple-fault
+- `-S -s` (`make runi`): start gdbserver at localhost:1234 and wait
+  for GDB
+- `-monitor unix:qemu-monitor-socket,server,nowait` (`make run
+  QMONITOR=1`): allows you to [connect to the QEMU
+  monitor](https://wiki.qemu.org/Features/Q35) via socket. Some useful
+  monitor commands:
+  - `info tlb`: when debugging memory mapping issues
+  - `info qtree`: to [dump emulated device
+    info](https://serverfault.com/a/913622)
+- `-trace enable=ahci_*` or similar. See `qemu-system-i386 -trace ?`
+  to view all trace tags.
+
+The GDB and qmonitor `make` commands come in pairs: one to start the
+VM/gdbserver/monitor server, and the other to start gdb/QEMU monitor:
+- `make runi`/`make gdb`
+- `make run QMONITOR=1`/`make qmonitor`

notes/random.md

@@ -88,29 +88,6 @@ cross-compiling](https://stackoverflow.com/a/72254698), but maybe a
 need will arise in the future. For now enforcing clang >= 14 is good
 enough for this project.
 
-## QEMU x86 quirks
-Both of the following make my life easier when running in QEMU, but
-make it harder for me to handle the expected case for real hardware.
-
-- *A20 is set on boot*: To be fair, the OSDev wiki states that this is
-  very non-standardized and there are a myriad of ways to
-  enable/disable the A20 line depending on the hardware. In QEMU's
-  case, [it was already set on
-  boot](https://forum.osdev.org/viewtopic.php?f=1&t=26256).
-- *No data segment offset limit checks*: In real mode, I noticed I can
-  write to 0x0000:0xB8000 without having to use unreal
-  mode. Similarly, when in protected mode/unreal mode, I was unable to
-  trigger the segment check (which should cause a
-  [GPF](https://wiki.osdev.org/Exceptions#General_Protection_Fault))
-  even if I manually set the limits low. It seems that QEMU doesn't
-  perform the limit checks ([at least as of
-  2019](https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg06518.html)).
-  I do get GPFs for the code segment though.
-- *PCI vendor ID 0x1234*: According to [this blog
-  post](https://web.archive.org/web/20200416081308/https://www.kraxel.org/blog/2020/01/qemu-pci-ids/). Indeed
-  0x1234 doesn't show up on the [list of PCI
-  vendors](https://devicehunt.com/all-pci-vendors).
-
 ## .clangd configuration
 Clangd configuration is ... not the best. First of all, it's
 confusing. It can be configured via `compile_flags.txt`,

notes/resources.md

@@ -23,6 +23,8 @@ going to bother linking OSDev wikis since those are implied.
 ## Memory
 - Paging strategy for x86: https://forum.osdev.org/viewtopic.php?p=312121#p312121
 - _Understanding the Linux Virtual Memory Manager_ by Mel Gorman
+- _A Primer on Memory Consistency and Cache Coherence_ by Nagarajan et al.
+- ioremap()/memremap(): https://lwn.net/Articles/653585/
 
 ## MBR
 - CHS decoding: https://thestarman.pcministry.com/asm/mbr/PartTables.htm#Decoding
@@ -39,7 +41,13 @@ going to bother linking OSDev wikis since those are implied.
 - QEMU internals blog: https://airbus-seclab.github.io/qemu_blog/
 - Setting breakpoints in QEMU w/ KVM enabled: https://forum.osdev.org/viewtopic.php?p=315671&sid=fa474a99fd3a40ba5cf2c2fafd7e44a9#p315671
 - Q35/ICH9 architecture: https://wiki.qemu.org/Features/Q35
+- Attaching QEMU monitor to running VM instance: https://unix.stackexchange.com/a/476617
 
 ## filesystem
 - `mtools` to manipulate filesystem image without mounting: https://stackoverflow.com/a/29798605
 - design of the FAT filesystem: https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system
+
+## AHCI
+- ATA/ATAPI command spec: https://people.freebsd.org/~imp/asiabsdcon2015/works/d2161r5-ATAATAPI_Command_Set_-_3.pdf
+- SATA AHCI spec: https://www.intel.com/content/www/us/en/io/serial-ata/serial-ata-ahci-spec-rev1-3-1.html
+- PIO, third-party DMA, and bus mastering (first-party DMA). A good first introduction to DMA: http://www.tweak3d.net/articles/howbusmaster/