Skip to content

Latest commit

 

History

History
227 lines (194 loc) · 12.6 KB

README.md

File metadata and controls

227 lines (194 loc) · 12.6 KB

FPGA Board Support

The supported boards are Alveo X3522PV (X3) and VCU118. Both have 256 KiB BRAM and 128 KiB L1D. X3 has a 16 KiB L1I and a 2 MiB URAM L2 and exposes 1 GiB of DDR4; VCU118 has a 32 KiB L1I and a 4 MiB URAM L2 and exposes 2 GiB of DDR4. Both L2s are four-way with 64-byte lines. X3 has 10GBASE-R Ethernet; VCU118 has a 1 Gb/s RJ-45 interface and 10GBASE-R on QSFP1 lane 0. Both NIC interfaces use coherent DMA. Both builds perform plain misaligned loads and stores to cached DDR in hardware by default (MISALIGNED_HW). VCU118 also has SPI flash boot into OpenSBI and U-Boot. See the FPGA guide for X3 usage and the VCU118 guide for its board integration and usage.

The X3 has eight L1D read slots and eight L1D request endpoints, so up to seven load misses and one store or LR acquisition are in flight to the L2 at once, and eight load queue entries. Its L2 coherence home has eight fills with three demand seats each. A demand fill uses one seat for its initiating request; a prefetch fill starts with all three seats free. The home has two writeback slots and a stream prefetcher that reads ahead of the hart's data and instruction miss streams.

Architecture Overview

X3 clocks, memory, loading, and debug connections

Module Role
xilinx_frost_subsystem.sv CPU and BRAM, JTAG software loader, BSCAN debug, reset timers
x3/x3_frost.sv Board clocks, DDR and NIC integration, reset sequencing
x3/x3_ddr_init.sv Initializes the exposed DDR region with valid ECC
x3/x3_ecc_reader.sv Polls the DDR4 controller's ECC registers for FROST's memory error unit
x3/x3_nic_gty.sv Ethernet transceiver, MAC clocks, transceiver reset supervisor
x3/x3_cpu_clock_gty.sv CPU clock from a dedicated GTY transmitter
x3/constr/x3.xdc Pin and timing constraints
x3/constr/cpu_clock_boundary.tcl Debug transport clock/CDC and optional CPU clock-boundary constraints, applied after synthesis

x3/x3_frost.f lists the board RTL. The Vivado flow generates the loader, DDR, and transceiver IP from fpga/build/build_step.tcl, x3_ddr_bd.tcl, and x3_gty_ip.tcl.

The cache hierarchy's bridge issues 512-bit AXI transactions, one 64-byte L2 line each, with region-relative addresses. A SmartConnect merges them with the DDR JTAG master and the ECC reader and crosses the CPU, CPU/4, and DDR clocks. The CPU port is as wide as the controller's 512-bit interface; only the JTAG master's transfers are widened. CPU address 0x80000000 maps the first 1 GiB. ECC management sits at region offset 0x40000000, reachable from the DDR JTAG master and the ECC reader.

After calibration, x3_ddr_init zeroes the region so every word has valid ECC (about 0.1 seconds). It writes whole 512-bit controller words, one per beat: a partial-word write would make the controller read the uninitialized rest of the word to recompute ECC. The CPU and the DDR loader leave reset once the CPU clock runs and calibration and initialization are complete; the loader, which runs on the CPU/4 clock, takes that reset through a synchronizer on it. Check for ECC errors with fpga/ddr_ecc/ddr_ecc_status.py.

Once initialization is done, x3_ecc_reader polls the ECC registers about once a millisecond and reports corrected and uncorrectable errors to FROST's memory error unit (hw/rtl/cpu_and_mem/memory_error_unit.sv), with the CPU physical address of the 64-byte line the controller captured. It translates the controller's row, column and bank back to an address with the controller's mapping for this board; the controller's ECC words interleave across the line's 8-byte words, so the line is as precise as a capture gets. It clears only the controller's CE status bit, so the controller captures the next address, and writes CE_CNT back to 1 once it reaches 128, before the count saturates at 255. It never writes CE_CNT to 0 and never clears the UE bit, so ddr_ecc_status.py still fails a run that had an error. The controller gives a register write priority over its own update in that cycle, so an uncorrectable error in the cycle of the reader's CE clear is lost to the controller's status, though the run's CE_CNT still fails the check and a CPU read of the word still reports an error response.

Clock Generation

Domain Clock
CPU 161.1328125 MHz Ethernet reference × 2 = 322.265625 MHz (GTY X0Y29 TXOUTCLK)
Loader, UART, reset timers CPU/4 = 80.56640625 MHz
DDR reference Independent 300 MHz
Ethernet TX / recovered RX GTY user clocks, about 161.13 MHz
GTY reset controllers 300 MHz input / 2 = 150 MHz, independent of both transceivers and the link

The CPU clock comes from its own GTY channel, X0Y29 in the NIC's quad: a CPLL at 3.22265625 GHz from the Ethernet reference clock, a 6.4453125 Gb/s transmitter that carries no data (held in electrical idle), and TXOUTCLK from the transmitter's programmable divider (the CPLL clock ÷ 10). Two BUFG_GTs make the CPU and CPU/4 clocks. The XDC roots them at X2Y7 (Vivado would otherwise root them at the transceiver, on the device's right edge), and the guided placement candidate moves the root to X1Y9. The channel shares only the reference clock buffer with the NIC, so a NIC PHY reset, which resets the NIC's QPLL0 and channel, leaves the CPU clock running. A small supervisor on the 150 MHz clock restarts the channel's reset sequence if it has not finished 100 ms after power good or if the CPLL loses lock, and holds the CPU in reset until the clock is back.

build.py --cpu-clock-div N sets the board top's CPU_CLK_DIV generic, which divides the CPU and CPU/4 clocks by N; the DDR and Ethernet clocks do not change. N = 2 divides in the BUFG_GTs. N = 3 and 4 need CPU/4 dividers of 12 and 16, beyond a BUFG_GT's 8, so those builds take the CPU clock from an MMCM on the 300 MHz input instead (300 MHz / 8 × 34.375 / 4N); the CPU clock channel still runs. Load software with the matching clock. The PERF_COUNTERS generic includes the profiling counters (--perf-counters). The X3 build includes the vector unit (VECTOR, at a VLEN of 256 bits); the loader's board list for each vector test program names the boards whose builds include it.

On X3, Linux reboot requests SYSRESET and re-enters the retained DDR images; they can be stale or hang in OpenSBI, so reload them for a fresh boot. poweroff halts with power still applied.

JTAG-based software loading

The loader writes images through two JTAG-to-AXI masters, one for low BRAM and one for DDR (jtag_axi_ddr):

  1. A first BRAM write puts the CPU in reset.
  2. The DDR image (sw_ddr.txt), if any, is written to DDR. Periodic BRAM writes keep the CPU in reset meanwhile.
  3. The BRAM image (sw.txt) is written.

Every BRAM write restarts a 27-bit counter on the CPU/4 clock, and the CPU leaves reset when the counter runs out: about 1.67 seconds after the last write at 322.265625 MHz. A separate 16-bit counter holds the loader and CPU in reset briefly after board reset, until the clocks are stable. The board reset reaches the loader, its counters, and the BRAM programming port through a synchronizer on the CPU/4 clock, and frost synchronizes resets into both clock domains.

RISC-V debug over BSCAN (OpenOCD)

Debug shares the FPGA's own TAP. BSCANE2 USER3 carries dtmcs, USER4 carries dmi, and the Vivado debug hub keeps USER1. The subsystem sets DEBUG_JTAG_TAP=0. The six-bit IR uses IDCODE 0x09, DTMCS 0x22, and DMI 0x23; the OpenOCD configurations are in fpga/debug/. Only one process can own the cable, so stop Vivado's hardware server before starting OpenOCD, and the reverse.

I/O Connections

Signal Direction Pin Description
i_sysclk_p Input AK23 300 MHz differential clock (positive)
i_sysclk_n Input AL23 300 MHz differential clock (negative)
o_uart_tx Output AP24 UART transmit for debug console
i_uart_rx Input AR24 UART receive for debug console input
i_nic_refclk_p / i_nic_refclk_n Input P9 / P8 161.1328125 MHz Ethernet reference clock (MGTREFCLK0, quad 231)
o_nic_txp / o_nic_txn Output J7 / J6 NIC transceiver TX (GTY X0Y28, DSFP28 cage labelled 2, lane 1)
i_nic_rxp / i_nic_rxn Input K4 / K3 NIC transceiver RX (GTY X0Y28)
o_cpu_clock_txp / o_cpu_clock_txn Output H5 / H4 CPU clock transceiver TX (GTY X0Y29), electrical idle
i_cpu_clock_rxp / i_cpu_clock_rxn Input J2 / J1 CPU clock transceiver RX (GTY X0Y29), unused

The UART console uses 115200 baud, 8N1.

Adding Support for New Boards

To support another Xilinx FPGA board:

  1. Create a subdirectory named after the board.
  2. Start from the x3_frost.sv wrapper.
  3. Adapt the clock generation: set the MMCM parameters for the board's input clock, drive the CPU clock from CLKOUT0, and derive the /4 clock with a global clock divider suitable for the FPGA family.
  4. Instantiate xilinx_frost_subsystem. For a board with DDR, also instantiate its ddr_subsys block design, wire the cache bridge's AXI port to it, and pass ENABLE_CACHED_TIER=1 and USE_BEHAVIORAL_DDR=0. Hold the CPU in reset until DDR calibration (mem_ok) and, if the memory has ECC, until the region has been written. The cached tier includes a configurable UltraRAM L2 (L2_CACHE_BYTES, 2 MiB by default), so the FPGA needs enough UltraRAM for its selected capacity. A BRAM-only board leaves the cached tier off and needs no DDR block design.
  5. Write a constraint file with the board's pin assignments, including the DDR pins unless a MIG .prj or board interface supplies them.
  6. Add the subsystem to the board's .f file list.
  7. Add the board's FPGA part and its has_ddr and has_gty capabilities to board_build_configs in fpga/build/build_step.tcl. The Tcl flow derives the board's wrapper, file-list, constraint, and IP-creation procedure names from its name. A board with DDR also needs fpga/build/<board>_ddr_bd.tcl for its ddr_subsys block design; the CPU port's S00_AXI range sets the memory size advertised to Linux. A board with a NIC transceiver needs fpga/build/<board>_gty_ip.tcl for its transceiver wizard core.
  8. Register the board in the FPGA tools:
    • BOARD_CONFIG in fpga/build/build.py: clock, family, and synthesis directive.
    • BOARD_INFO in fpga/build/extract_timing_and_util_summary.py.
    • BOARD_CONFIG in fpga/load_software/load_software.py: clock, CoreMark iterations, and DDR support.
    • BOARD_VENDOR_INFO in fpga/common/hw_target.py, and DEFAULT_TARGETS, DEFAULT_SERIALS, and DEFAULT_TIMEOUTS in fpga/common/hw_defaults.py.
    • supported_boards in fpga/program_bitstream/program_bitstream.tcl, for direct Tcl use. The Python programmer, regression, soak, and sweep tools take their board lists from the registries above.
  9. Calibrate the board's CoreMark-PRO hardware_iterations in sw/apps/software_registry.py, and set hardware_timeout_minimums if setup takes longer than the board's default timeout. Optionally record score gates in BASELINE_SCORES in fpga/hw_regression.py.
  10. Add the board to this README.

Before the first build, check that:

  • the MMCM VCO frequency and dividers produce the target CPU clock;
  • the timing constraints match the input clock period;
  • the I/O standards match the board's bank voltages;
  • the DDR controller IP matches the board's soldered or SODIMM memory.

A non-Xilinx FPGA would need a different subsystem, because xilinx_frost_subsystem uses Xilinx IP and BSCANE2 primitives.

Board-level AXI and MAC plug-in ports

frost exposes three compile-time integration parameters (all zero by default, which xilinx_frost_subsystem keeps for X3 and VCU118) together with the core-domain synchronous reset output o_soc_rst:

  • EXT_DMA_PORTS (with EXT_DMA_ID_BITS) appends coherent 256-bit AXI4 DMA slave ports to dma_interconnect, each isolated by an axi_register_slice.
  • EXTERNAL_MAC=1 omits the internal nic_mac_10g block and exports the primary nic_engine's core-clock 64-bit AXI4-Stream TX/RX and MAC sideband ports so a board wrapper can attach a different MAC.
  • EXT_AXIL_WINDOW=1 exposes a 16 KiB 64-bit AXI4-Lite master window at 0x4002_4000-0x4002_7FFF behind an axil_register_slice in both cached and BRAM-only builds.

See hw/rtl/lib/axi/README.md for port widths, coherence aperture rules, and reset contracts.