Configure FPGA/SPI Flash to Run Linux on LX9 MicroBoard, Part 1 – Preparation

This is Part 1 for the topic, to prepare configuration data for LX9 MicroBoard.

The FPGA design is done with ISE 13.2 based on AXI/Little Endian MicroBlaze 8.20a. The details were presented here at Run Linux on Avnet Spartan-6 LX9 MicroBoard.

The Linux kernel is built with Petalinux SDK 2.1. You may contact PetaLogix to obtain a license and SDK to try out. If you prefer Xilinx’s embedded Linux solution, you can find useful information from its Open Source Wiki. From configuration point of view, it makes no difference which one is used.

The keys to prepare the configuration data are:

• understand how SPI Flash is partitioned
• understand how each partition is used
• make sure proper sizes for partitions, especially for the partition where Linux is at

You may need build kernel more than once, if you don’t know the kernel size beforehand. For example in the second build, you can adjust partitions to fit Linux into flash.

Here’s Flash partitions used by Petalinux, and sizes are modified because of 11MB kernel. Note SPI Flash on the MicroBoard has 256 64KB blocks, total space is 16MB.

Partition	Name	Size (blocks)	Flash Address	Usage
0	fpga	2MB (32)	0x0	bitstream and fs-boot
1	boot	256KB (4)	0x200000	u-boot
2	bootenv	128KB (2)	0x240000	u-boot environment variables (not used here)
3	config	128KB (2)	0x260000	used by flatfsd to store persistent data, required but not used by configuration
4	image	12.5MB (200)	0x280000	Linux kernel and filesystem
5	spare	1MB (16)	0xf00000	remaining space (not used here)

Here’s how partitions are specified in kernel configuration (System Settings -> Flash Partition Table -> Image Partition Size).

System Settings

Flash Partition Table

Image Partition Size

Once the kernel is rebuilt, along with the outputs of FPGA bitstream from ISE/EDK, and U-Boot from XSDK, the following files are used to construct configuration data, and/or bootstrap Linux:

File	Description	Built by	Usage
system_bd.bmm	BRAM memory map	ISE/XPS	to generate download.bit by script d2m
mb_system_top.bit	design bitstream
fs-boot_0.elf	fs-boot	XSDK
u-boot.elf	u-boot executable	Petalinux SDK	not used by configuration, but useful to transfer images to Flash
u-boot-s.bin	u-boot relocatable binary		to generate sopc_lx9.mcs by script pgen.ub
image.ub	Linux kernel and filesystem binary

This is the script d2m, a wrapper of data2mem:

d2m

#!/bin/sh data2mem \ -bm mb_system_bd.bmm \ -p xc6slx9csg324-2 \ -bt mb_system_top.bit \ -bd fs-boot_0.elf \ tag microblaze_0 \ -o b download.bit

This is the script pgen.ub, an wrapper of promgen:

pgen.ub

#!/bin/sh promgen -w \ -p mcs \ -c FF \ -o sopc_lx9 \ -s 16384 \ -u 0000 download.bit \ -data_file up 200000 u-boot-s.bin \ -data_file up 280000 image.ub \ -spi

The following capture summarizes the procedure to generate configuration data.

Generate Configuration Data

If you would rather prepare PROM from iMPACT GUI, here’re the steps (Note, the files could be at any folder).
Launch iMPACT and select “Prepare a PROM File”.

Prepare PROM File

Select and Enter as following (highlighted).

Prepare PROM Steps

Add device bitstream.

Add Device Bitstream

Add 2 data files.
The first one is U-Boot, and make sure it is at proper address on Flash, in this case it is 0x200000.

U-Boot Flash Address

Add U-Boot

The second data file is Linux kernel and filesystem. Make sure the Flash address is set to 0x280000.

Kernel Flash Address

Add Kernel

Here’s the resulting data file assignment.

Data File Assignment

Proceed to generate configuration data.

Generate File

You can find scripts and source data files from here.

As a side note, when people move to high end FPGA solutions with processor integration, as Xilinx calls Zynq as EPP (Extensible Processing Platform), and Altera calls SoC FPGA with HPS (Hard Processor System), there is less or no need, in a sense, to configure FPGA from persistent storage such as Flash. The processor can easily make bitstreams and software images available to FPGA.
For large volume and low cost applications, soft-cores such as MicroBlaze and Nios are good solutions, and configuration is still needed.

Configure FPGA/SPI Flash to Run Linux on LX9 MicroBoard, Part 3 – Easy Way

This is Part 3 for the topic.
In Part 2 – Quick Way, network connection and tftp server, plus certain knowledge of U-Boot are needed. You can avoid them by just using iMPACT to program the FPGA/SPI Flash. Here’s how.

On the host with Xilinx design tools installed, navigate to the folder with following files.

File	Description	Usage
sopc_lx9.mcs	configuration data	program FPGA/SPI Flash
impact.cmd	programming script	for Xilinx USB II cable (6MHz)
impact.12Mh.cmd	programming script	for Xilinx USB II cable (12MHz)
impact.onboard.cmd	programming script	for Digilent Onbard JTAG

Program FPGA/SPI Flash by running iMPACT in batch mode.

USB II Batch Program

The Linux kernel and u-boot has a size about 10MB (still have some rooms left for 16MB Flash, I could see many potentials there), and FPGA/fs-boot has only 336KB. The final MCS data triples the size (in ASCII format).
It takes about 68 minutes on my Linux host, including verification. Program alone may take about 43 minutes, by using default JTAG clock speed, 6MHz.

USB II Program Done

With impact.12Mh.cmd, it takes about 35 minutes. iMPACT complains 24MHz is not supported.

Program at 12MHz

I also tried several times to program by using MicroBoard’s Onboard USB/JTAG. It is supposed to be running fast (in 15MHz?). By 1) modifying Avnet Tom’s script listed in LX9 board forum Remarks on AvtS6LX9MicroBoard_SW302_PetaLinux, and 2) by iMPACT GUI, but I was never able to go to the end. I remembered once I let it run overnight from GUI, it was less than half way through in the next morning, and I had to abort the process.
It could be I didn’t specify options correctly. I attach the programming script for Onboard JTAG, just for reference, but be warned, don’t try it, it may take forever to program.

If you’re more interested at programming through iMPACT GUI to have a good understanding what’s going on, here’re some screenshots.

Start iMPACT by choosing “Configure devices using Boundary-Scan (JTAG)”.

Configure Devices

Select “No” for “Auto Assign Configuration Files Query Dialog”.

No for Auto Assign

Right click the device from “Boundary Scan” tab to add SPI Flash.

Add SPI Flash

Select attached SPI Flash (N25Q128 1.8/3.3V).

Select SPI Flash

Right click added SPI Flash device and select “Program”.

Program

It achieves identical result as running the script in batch mode, and takes about the same time to finish.

If you’re interested, here’s the link for scripts and configuration data.

Configure FPGA/SPI Flash to Run Linux on LX9 MicroBoard, Part 2 – Quick Way

This is Part 2 for the topic, I will present Part 1 (Preparation) after Part 3 (Easy Way) is done.

Petalinux is using a 2-stage booting process. The First Stage boot, FS-Boot, is running from FPGA’s BRAM once powered on. It’s sole responsibility is to load the main system bootloader, i.e., U-Boot, from the flash to the larger external memory, and run U-Boot from the external memory.

One of the quick ways to configure FPGA is to copy binary images to the flash, instead of programming it, which could take significant time depends on programmer used.

In order to do that, networking connectivity and a mechanism to deliver and retrieve images are required.
As used by almost everyone, I’m using tftp. The host runs tftpd is connected to the LX9 MicroBoard via a router.

The following images are needed.

Image	Description	Usage
download.bit	bitstream plus fs-boot binary	FPGA and fs-boot image on flash
u-boot.elf	U-Boot in ELF format	to transfer u-boot-s.bin and image.ub to flash
u-boot-s.bin	relocatable U-Boot in binary format	U-Boot on flash
image.ub	Linux kernel and root filesystem in binary format	Linux on flash
impact.cmd	script for impact (not required)	transfer download.bit
xmd.ini	script for xmd (not required)	run u-boot.elf

I am using Platform Cable USB II programmer, usb/uart port is directly connected to the host via minicom.

Board Connection

Start tftp daemon, and point the working folder to where the above files are.

Start tftp

Next step is to program FPGA through impact. If you’re using Digilent’s HS1 or onboard JTAG connection, your script should be different.

Program FPGA

Then download and run U-Boot through xmd. If you’re using Digilent’s HS1 or onboard JTAG connection, your script should be different.

Download and Run U-Boot

If the flash is programmed already, stop the autoboot process from the uart port console, in order to move new U-Boot and Linux to the flash.

Stop Autoboot

Now it is time to copy Linux kernel and U-Boot to flash. But before doing that, IP addresses for the tftp server (host) and the board (target) need be set. Your IP addresses could be different the ones used below.
Here’s how to do it and copy the Linux kernel (image.ub) to the flash.

IP Setup and Copy Kernel

Then copy the U-Boot.

Copy U-Boot

Now if everything goes well, you can run Linux by either hitting the reset button or issuing boot command from U-Boot prompt.

Booting

Here’s the booted console, and SPI flash’s partitions.

Booted Console and SPI Flash Partitions

You can find listed binary images and scripts here.

Run Linux on Avnet Spartan-6 LX9 MicroBoard

AV-LX9 MicroBoard is a new low cost Spartan-6 LX9 device based development kit from Avnet. XC6SLX9
has 1,430 slices, or 5,720 LUTs, and 32 Block RAM. Rich features, such as 64MB DDR, 128Mb SPI Flash, 10/100 Ethernet PHY, USB-UART port, JTAG, LEDs, DIP switches and expansion ports, make the board an affordable and very useful experimental prototyping tool.

Thanks for Avnet’s Frank for the LX9 MicroBoard, I was lucky to be able to try out the board early. My main goal was to get Petalinux running on this this USB-stick form factor board, with httpd server serving web pages and certain peripherals to demonstrate user defined functionalities. After many weekends’ efforts, I was always couple hundreds of LUTs in short to fit designs of MicroBlaze with MMU into LX9 device, even after removing most peripherals, and enabling Legalizer placement algorithm.

Then Petalogix’s John offered me a Linux BSP, used for hobby purpose. This convinced me the goal was possible, and kept me trying my own SOPC designs.

Finally I got it working before the recent Avnet LX9 MicroBoard SpeedWay Design Workshops. The keys were to remove non-essential MicroBlaze features and not use MDM’s JTAG-based UART thus saves a AXI4-Lite connection. I am sure it can go further to use less resources and/or add more peripherals, but still has a stable Linux running on the board.

Here’s how the working design was done in details. I am using PetaLinux SDK 2.1 and ISE 13.2 for 32 bit Linux, starting design flow from Project Navigator.

Create a new project based on Avnet LX9 MicroBoard XBD.

Create New ISE Project

Select Embedded Processor for Source Type with New Source Wizard.

Select Source Type

When BSB starts, choose AXI flow.

AXI Flow

And configure below processor, cache and peripherals.

Processor, Cache and Peripheral Configuration

Once XPS project is created, configure MicroBlaze by double clicking the IP from System Assembly View.

MicroBlaze with MMU Configuration

After select MMU configuration, click Advanced button to fine tune. The following setups can be done by going back and forth to enable then disable certain features. Here’s General configuration.

MicroBlaze General Configuration

Disable all exceptions.

MicroBlaze Exception Configuration

Make sure instruction cache base address is setup properly. This is the location where Linux kernel starts.

MicroBlase Cache Configuration

Here’s MMU setup.

MicroBlaze MMU Configuration

Enable MDM so XMD can be used to debug.

MicroBlase Debug Configuration

No need for Processor Version Registers.

MicroBlase PVR Configuration

Here’s the Interconnection I used.

MicroBlaze Interconnect Configuration

Buses setup is the last MicroBlaze configuration.

MicroBlaze Buses Configuration

Now configure MDM by disabling JTAG UART.

MDM Configuration

Here’s XPS System Assembly View when design is done.

XPS System Assembly View

Return to Project Navigator when XPS is closed. Add constraints by Project > Add Copy of Source.

ISE Add Constraints

Double click Generate Top HDL Source to create HDL instantiation template for the MicroBlaze design.

ISE Generate Top HDL Source

Export hardware design to SDK.

ISE Export Hardware To SDK with Bitstream

If there are errors for placement and routing, try to set environment variable XIL_PAR_ENABLE_LEGALIZER to 1, and rerun PAR.

Here’s my XPS Synthesis Summary (estimated).

XPS Synthesis Summary

Here’s partial ISE Device Utilization Summary.

ISE Device Utilization Summary

Follow PetaLinux SDK Board Bringup Guide to configure software settings of the hardware project and fs-boot.

I am using the following XMD commands to configure FPGA and download Linux image.

This is the PetaLinux login screen.

PetaLinux Login Screen

And PetaLinux Process Status.

PetaLinux Process Status

Another fun experiment is to write code to turn on and off LEDs in certain patterns. 4 bit LED IP’s address is 0x40000000, to turn on all LEDs, you can use:

poke 0x40000000 0x0f

Let me know if you’re interested running my design, I have it here. If you like to try the commercial PetaLinux, please contact Petalogix.

Xilinx Moves to AXI Protocol Fast

With the release of ISE 12.4 earlier in the year, 13.1 in last month, and the first 7-series Kintex device in production, Xilinx is moving fast to AXI (Advanced eXtensible Interface) protocol. AXI is one of the interface standard from ARM’s AMBA (Advanced Microcontroller Bus Architecture).

Here’s what happening:

• Embedded Processor is in PowerPC to ARM transition
• IP is moving to a single interconnect standard (AXI4) for Embedded, DSP and Connectivity domains
• MicroBlaze is in PLB to AXI transition since v8

Coming ISE release will support AXI only 7-series devices. Virtex-6 and Spartan-6 have both PLB and AXI support.

PetaLogix released PetaLinux SDK 2.1 in April with full support of ISE 13.1 and AXI. I was able to bring up PetaLinux with AXI bus easily by following Board Bringup Guide.

PetaLinux with AXI

One important difference between PLB and AXI is, MicroBlaze with AXI is in little endian.

I can’t wait for Zynq-7000 EPP coming out. And MicroBlaze/AXI with PetaLinux is a good head start.

Linux Bring-Up on SoPC

System-on-Programmable-Chip is an emerging trend in the embedded world.
Here’s what I did to bring-up Petalinux on Xilinx Spartan-6 devkit SP605/SP601.
Petalinux is an embedded Linux distribution for Xilinx FPGA’s MicroBlaze softcore. Since SDK v1.3, the current release as of early 2011, Petalinux supports PowerPC440 hardcore.
Once you have the right hardware and software tools, such as devkit (SP605 or SP601), Xilinx ISE (12.4, 12.3 is fine), Petalinux SDK (v1.3) and a Linux host (a Xubuntu 10.4 VM on Macbook), it is surprising easy to bring-up Linux on FPGA. The idea is the same if you have a different hardware and software combination.

1. Install Xilinx ISE 12.4 for Linux. Petalinux SDK depends on Xilinx design suite to run.

I installed ISE (Embedded Edition) to a CIFS network share, which is a Journaled HFS+ external portable USB2 hard drive connected to Macbook (it is worth for another article to cover the topic, may be some other time). If ISE is installed on a virtual disk, it takes about 12G space for ISE, EDK, PlanAhead, LabTools and SDK (not all are necessary for our purpose).
Here’s how ISE 12.4 is setup.

ISE 12.4

Xilinx JTAG Platform Cable drive needs to be installed too. I installed CP210x USB/UART driver on Macbook host.

2. Install Petalinux SDK 1.3

The SDK should be installed on a Linux file system, because of some Linux specific symbolic links. I installed it to my Xubuntu’s second ext4 virtual disk.

Here’s how Xilinx and Petalinux (/media/users) are mounted:

Mounts

Here’s how SDK is setup. I installed it to /software/target/ because of the specific need for the projects I am working on.

SDK 1.3

3. Treakin Ubuntu
The following first two changes are needed for SDK under Ubuntu.

SDK scripts are using pushd/popd, so replace /bin/sh (link to /bin/dash) and point it to /bin/bash.

Create symbolic link /usr/bin/gmake, pointing to /usr/bin/make.

I also found a weird thing about SDK 1.3. It looks for some files from ISE 12.3, so create a symbolic link 12.3 and points it to 12.4 (See Picture ISE 12.4).

4. Design hardware
PetaLogix’s Board Bringup Guide is extremely useful, but not all that clear. To bring up Linux, it needs UART, system timer, DDR and MicroBlaze hardware, Ethernet and Flash are optional. UART, system timer and Ethernet must be interrupt driven.
Create a folder, say sopc, under $PETALINUX/hardware/user-platforms.
Launch Xilinx Platform Studio (enter $xps &) and choose Base System Builder wizard.

Base System Builder wizard

Create New XPS Project by specifying project file name, say sopc.xmp under $PETALINUX/hardware/user-platforms/sopc, and search path at $PETALINUX/hardware/edk_user_repository.

XMP Project

Keep “PLB system” as the default interconnect type, and move BSB wizard forward to “Board Select”. Choose “Spartan-6 SP605 Evaluation Platform” as the “Board Name”.

SP605

Move BSB wizard forward by using all default selections till “Peripheral Configuration”. Add xps_timer, check “Use Interrupt” option and choose “Two timers are present” for “Configuration Mode”.

xps timer

Check “Use Interrupt” option for Ethernet_MAC.

Ethernet

For RS232_Uart, select 115200 for “Baud Rate”, and check “Use Interrupt” option.

UART

Move to “Cache Configuration”, and check both “Instruction Cache” and “Data Cache”.

Cache

Move the wizard to the end to finish BSB, and go back to XPS. In “System Assembly View” tab, double click MicroBlaze instance to launch “MicroBlaze Configuration Wizard”. Select “Linux with MMU” configuration, and then “OK” button to finish the wizard.

Select “Software Platform Settings” from XPS “Software” menu. For “Software Platform” tab, select “petalinux” as “OS” under “OS & Library Settings” section.

Software Platform Petalinux

For “OS and Lib Configuration” tab, make the selections for peripheral as highlighted in the following screenshot. If FLASH is used in the design, make sure its value is properly set.

OS and Lib Configuration

Now the hardware design is done, time to build bitstream by selecting “Generate Bitstream” from “Hardware” menu.
Before doing that, and if FLASH is used, make sure FLASH’s base address must be after DDR3. It can be changed and verified from XPS “System Assembly View” tab under “Address”.
In my environment, it takes about 22 minutes to finish the build process. The generated bit file is located at: $PETALINUX/hardware/user-platforms/sopc/implementation/sopc.bit. It will be used to configure Spartan-6 FPGA on SP605 later.

5. Design software

Create software platform by using: petalinux-new-platform:
petalinux-new-platform -c <cpu-arch> -v <VENDOR> -p <PLATFORM> 

Create Software Platform

Architecture configuration, Linux kernel configuration, and DTS file, etc., are generated at: $PETALINUX/software/petalinux-dist/vendors/tcao/sopc.

Configure Software Platform:
petalinux-copy-autoconfig -v <VENDOR> -p <PLATFORM> PROJECT_FILE 

Configure Software Platform

FPGA configuration files and drivers are copied to proper folders in $PETALINUX/software/petalinux-dist/linux-2.6.x.

Configure Kernel and Applications.
petalinux-config-kernel
petalinux-config-apps 

Configure Kernel and Apps

Build Linux Kernel
cd $PETALINUX/software/petalinux-dist
make 

Build Kernel

It takes about 21 minutes to build the kernel. Kernel image is at:
$PETALINUX/software/petalinux-dist/images/image.elf.

6. Upload configuration and application
Now comes the fun part to configure FPGA and unload Linux kernel. Connect JTAG USB and USB/UART cables to the host, power on SP605 or SP601 board. I forward the JTAG USB connection to VM, and keep USB/UART connection to host (baud rate is 115200).

Configure FPGA
Navigate to the hardware project folder and launch XMD:
$xmd
XMD% fpga -f xxx.bit

FPGA Programming

Upload Linux Image
Within XMD, connect to Microblaze core, upload the Linux image and boot Linux:
XMD% connect mb mdm
XMD% dow xxx.elf
XMD% run

Linux Upload

From serial console, login as root.

Petalinux Console