Table of Contents:
I. Introduction to UberDDR3 Project
As mentioned in our previous post: Introducing OpenIPHub, my next upcoming blogs will initially focus on my ongoing project: UberDDR3 (repository link here). DDR3? Like the DDR RAM we insert and upgrade in our computers? Yes, exactly! That same DDR3 (or DDR4 or DDR5) RAM used in our computers and laptops. So, are we creating the DDR3 memory chip itself? No! What we are designing with UberDDR3 is the controller used for, obviously, controlling those DDR3 memory chips.
The DDR technology, standing for Double-Data Rate, enables faster data transfer by doubling the rate at which data is transferred. It's a complex technology abstracted from the user by what is known as the "DDR3 controller." In our computers, this controller is part of the CPU processor, managing communication between the CPU and the memory device. This controller is the brain behind the operation, managing the timing, access, and management of data between the processor and the memory modules.
Figure 1. Block-level view of an FPGA Project Utilizing a DDR3 Memory
Why is there a need for creating this controller, you might ask? The DDR3 controller is critical because it handles the complex mechanisms required to properly communicate with a DDR3 memory device. As shown in Figure 1, the controller acts as the intermediary handling the communication between the processor core and the DDR3 chip. The DDR3 controller is responsible for tasks such as refreshing memory contents, managing read and write cycles, and ensuring data integrity through error-checking and correction mechanisms.
Memory controllers inside CPUs, like those from Intel, used to utilize the QuickPath Architecture for connecting with the actual processor cores. Basically, the QuickPath Architecture is a high-speed, packet-based communication system previously used in Intel processors to facilitate data transfer between the processor, memory, and other components within a computer. As shown on Figure 2, which is from the Intel® QuickPath Interconnect document, the processor has a point-to-point interconnection with an on-die memory controller which then interfaces with an external memory. We are trying to highlight here the existence of that memory controller acting as the go-between for the memory device and the processor cores.
Figure 2. Intel QuickPath Interconnect
II. Brief History
Why did I created this controller in the first place? Back in 2023, during my last year at the university, I was offered the opportunity to be part of a team designing a 10Gb Ethernet Switch. It was a small team, and it was my first time being involved in a funded project aimed at commercializing the product. My task was to create the DDR3 controller to interface with SO-DIMM DDR3 RAM, which would serve as the main memory for processing and routing the network packets.
Shown in Figure 3 is the custom hardware built, named KlusterLab board. On the lower left, you can see the four SFP+ ports for the switch. This project is packed with features; take a peek at the details from the ZipCPU blog. My task was to develop the controller for the DDR3 RAM, which would be inserted into that DDR3 SODIMM socket.
Figure 3. KlusterLab Board for the 10Gb Ethernet Switch Project
You might ask, is there really a need to create our own controller from scratch for this 10Gb Ethernet Switch? Aren't there other ready-to-use controllers? Well, yes, there are. If you've tinkered with FPGA projects before, especially using Xilinx ones (it's strange now to say "AMD FPGA"), you might have already used the Vivado MIG (Memory Interface Generator), which can generate a variety of memory controllers (specifically DDR2 and DDR3 controllers). But obviously, that's not completely open-sourced; the RTL code can be generated, but that's not something you can post on GitHub and collaborate to improve.
The generated IP by Vivado MIG are also specific to AMD FPGA boards, meanwhile an open-source work can be improved to be a general-purpose IP that supports multiple FPGA board vendors. And in my honest opinion, the Vivado MIG is convoluted. Actually, even after spending almost a year creating my own open-source DDR3 controller, the RTL generated by MIG looks like magic to me. And there are primitives used in the PHY interface, like PHASER_OUT and PHASER_IN, which are intentionally undocumented. If you're asking if that's good or bad, well it doesn't really matter if the goal is just to have a ready-to-use DDR3 controller IP for your next project. But when we're talking about open-source work, then it's problematic. An open-source design is OPEN, meant to be something anyone can use, understand, and collaborate to improve.
Figure 4. Vivado MIG Generating a DDR3 Controller IP
There are other DDR3 controllers that are open-sourced, and I will elaborate on some of them in the next blog post. However, the team for the 10Gb Ethernet switch project concluded that we needed a more robust DDR3 controller that is also open-source. Hence, I was tasked with this project.
2023 was also my final year at the university. I had to complete my capstone project, final exams, graduation, review and prepare for the national engineering licensure exam, and then attend job interviews for my first job, AND THEN work on this DDR3 controller project. The year 2023 might have been one of the busiest years of my life.
Here's a brief share of my timeline for implementing the DDR3 controller:
January: Analyzed JEDEC DDR3 specification.
February: Researched other open-sourced DDR3 controllers.
March-April: Implemented the reset and refresh sequence logic, and bank-tracking for pipelined accessing of memory banks.
April: Designed the PHY module and implemented FSM logic for bitslip training.
May: Implemented multi-lane read/write calibration FSM, then created the testbench using the DDR3 Verilog model from Micron.
June: Verified controller pipeline using formal verification and optimized pipeline stall control for higher request throughput.
July: Passed the SODIMM configuration simulation and integrated the controller to the KlusterLab board hardware for debugging.
August: Hardware debugging (this was a nightmare, I didn't have an idea at first whether the problem was with my design or the custom hardware, desperate days...).
September: Finally fixed all bugs, passed all hardware tests, and officially handed off the DDR3 controller to the 10Gb Ethernet switch team.
If any of these terms or processes seem overwhelming, don't worry. I will elaborate on the technical side of the implementation process in future posts, which might be of interest to digital designers out there. For now, let's talk about the current capability of this DDR3 controller that will be relevant to casual users.
III. Current capability of UberDDR3
This DDR3 controller was originally designed for use in the 10Gb Ethernet Project for an 8-lane x8 DDR3 SODIMM running at 400 MHz (clock of DDR3 PHY is 400MHz thus this is a DDR3-800 since double-data rate). This is a 4:1 memory controller (ratio of DDR3 clock to controller clock, thus controller clock can be 100MHz while DDR3 clock is 400MHz) with configurable timing parameters and mode registers allowing it to be configured to any DDR3 memory device (different DDR3 device can have different timings and settings for mode register which can be checked via datasheet). The user-interface is the basic Pipelined Wishbone (wishbone interface can be classic or pipelined, please refer to Wishbone B4 protocol document on the difference. But basically, pipelined version is used as the user interface to maximize pipelined throughput during burst operation).
This memory controller is optimized to maintain a high data throughput and continuous sequential burst operations. The controller handles the reset sequence, refresh sequence, mode register configuration, bank status tracking, timing delay tracking, command issuing, and the PHY's calibration. The PHY's calibration handles the bitslip training, read-DQ/DQS alignment via MPR (read calibration), write-DQ/DQS alignment via write leveling (write calibration), and also an optional internal read/write test.
The optional internal read/write tests simplify testing the memory controller without the need for an external CPU to manage the test sequence. These tests include a burst access, random access, and alternating read-write access tests. Only if no error is found on these tests will the calibration end and user can start accessing the wishbone interface. This design has been formally verified, and simulated using the Micron DDR3 model.
Shown in Figure 5 is the high-level view of the UberDDR3. We will dive deeper into the technical overview of the digital design at some later blogs.
Figure 5. Block Diagram Overview of UberDDR3 IP
IV. Upcoming features for the UberDDR3
The 10Gb Ethernet Switch project is complete, but are there future upgrades planned for the DDR3 Controller? YES! In September of 2023, we applied for funding from NLNet to enhance this DDR3 Controller. To cut a long story short, the proposal was accepted! The introduction to the proposal reads as follows:
In an age where software and AI are rapidly advancing, the demand for high-speed memory processing becomes crucial. The gap between software innovation and hardware capability highlights the need for more efficient memory solutions. The UberDDR3 project steps up for the open-source community, offering a robust DDR3 controller that aims to bridge this gap, providing performance comparable to proprietary DDR3 controllers. It represents a significant step toward independence from proprietary memory controllers and sets a new benchmark for open-source hardware capabilities.
The milestones for this project proposal are:
Create a website for future updates and knowledge sharing on UberDDR3 project, and create initial blog posts outlining the current capabilities of UberDDR3.
Integrate an AXI bus interface atop the existing Wishbone interface.
Implement the in-line ECC feature to enable error-correction capability.
Expand support for different data bus widths ranging from 8-bits to 72-bits.
Implement self-refresh mode to decrease power consumption.
Implement dual-rank interleaving to support dual-rank DDR3 DIMM.
Implement on-the-fly timing parameter configuration, enabling automatic configuration based on the inserted DDR3 DIMM type.
Implement thermal management to ensure reliability under varying thermal conditions.
Optimize design for higher clock speed exceeding 400MHz.
Port the controller to the open-source FPGA toolchain.
Port the controller to diverse FPGA architectures from firms like Gowin, Lattice, etc.
Deploy on CI system, make available DDR3 controller integrated in test designs for different boards as deployable bitstreams for function verification.
Implement a benchmarking SOC to measure read and write speed.
As you might have guessed, milestone number 1, involving creating a website for future updates and knowledge sharing on the UberDDR3, is actually this Open IP Hub site! As I mentioned in our previous post on Introducing OpenIPHub, even though this site was originally for the UberDDR3 project I will still continue running this site and publishing blogs for the foreseeable future. Creating blogs for the UberDDR3 project has shown me how enjoyable blogging is and the benefits it brings to my career by publishing my work publicly.
Oh, and you might be wondering why we keep saying "UberDDR3." Actually, the project's name was simply "DDR3 Controller" while the 10Gb Ethernet Switch project was ongoing, but once it was accepted by NLnet for funding, we decided to give it a more personal name.
The name we chose is UberDDR3. "Uber" comes from the German language, where it's spelled "über" and means "over," "above," or "across." It’s a fitting name for this ambitious project, as it aims to go above and beyond any previous open-source DDR3 controller.
So, there you have it, welcome to the UberDDR3 project. More blogs are coming to introduce you to the know-how for using this DDR3 controller IP in your next project, provide a technical overview of the RTL side of the controller, and dive deep into the upcoming features.
V. Who Would Benefit from the UberDDR3 Project
FPGA hobbyists, always on the lookout for new projects and challenges (like myself!), will find the UberDDR3 particularly appealing. Its open-source nature means anyone with an interest in digital design can tinker with, modify, and possibly collaborate to improve the controller.
For students learning electronics and digital design, the UberDDR3 can serve as an excellent educational tool. It provides hands-on experience in understanding how memory controllers work, offering a practical aspect to theoretical knowledge of today's memory controllers. Professionals in the field of digital design and electronic engineering can also leverage the UberDDR3 for commercial and research projects. Its high-performance capabilities make it suitable for applications requiring robust memory control (think high-speed data processing, real-time video processing, data packet processing for networking, and much more!). I believe the fact that this controller was originally designed for storing packets in a 10Gb Ethernet Switch speaks volumes about its capability and its potential for other heavy data processing projects.
That wraps up this post. Catch you in the next blog post!
