SO2 Lab 11 - Kernel Development on ARM¶

Lab objectives¶

get a feeling of what System on a Chip (SoC) means
get familiar with embedded world using ARM as a supported architecture
understand what a Board Support Package means (BSP)
compile and boot an ARM kernel with Qemu using i.MX6UL platform as an example
get familiar with hardware description using Device Trees

System on a Chip¶

A System on a Chip (SoC) is an integrated circuit (IC) that integrates an entire system onto it. The components that can be usually found on an SoC include a central processing unit (CPU), memory, input/output ports, storage devices together with more sophisticated modules like audio digital interfaces, neural processing units (NPU) or graphical processing units (GPU).

SoCs can be used in various applications most common are:

consumer electronics (TV sets, mobile phones, video game consoles)
industrial computers (medical imaging, etc)
automotive
home appliances

The leading architecture for SoCs is ARM. Worth mentioning here is that there are also x86-based SoCs platforms. Another thing we need to keep an eye on is RISC-V an open standard instruction set architecture.

A simplified view of an ARM platform is shown in the image below:

We will refer as a reference platform at NXP's i.MX6UL platform, but in general all SoC's contain the following building blocks:

one or more CPU cores

a system bus

clock and reset module

PLL

OSC

reset controller

interrupt controller

timers

memory controller

peripheral controllers

I2C

SPI

GPIO

Ethernet (for network)

uSDHC (for storage)

USB

UART

I2S (for sound)

eLCDIF (for LCD Panel)

Here is the complete block diagram for i.MX6UL platform:

i.MX6UL Evaluation Kit board looks like this:

Other popular SoC boards:

Broadcom Raspberry Pi

Texas Instruments Beagle board

Odroid Xu4

Nvidia Jetson Nano

Board Support package¶

A board support package (BSP) is the minimal set of software packages that allow to demonstrate the capabilities of a certain hardware platform. This includes:

toolchain

bootloader

Linux kernel image, device tree files and drivers

root filesystem

Semiconductor manufacturers usually provide a BSP together with an evaluation board. BSP is typically bundled using Yocto

Toolchain¶

Because our development machines are mostly x86-based we need a cross compiler that can produce executable code for ARM platform.

We can build our own cross compiler from scratch using https://crosstool-ng.github.io/ or we can install one

$ sudo apt-get install gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf # for arm32
$ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu     # for arm64

There are several of toolchain binaries depending on the configuration:

With "arm-eabi-gcc" you have the Linux system C library which will make calls into the kernel IOCTLs, e.g. for allocating memory pages to the process.

With "arm-eabi-none-gcc" you are running on platform which doesn't have an operating system at all - so the C library is different to cope with that.

Compiling the Linux kernel on ARM¶

Compile the kernel for 32bit ARM boards:

# select defconfig based on your platform
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make imx_v6_v7_defconfig
# compile the kernel
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j8

Compile the kernel for 64bit ARM boards:

# for 64bit ARM there is a single config for all supported boards
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make defconfig
# compile the kernel
$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j8

Linux kernel image¶

The kernel image binary is named vmlinux and it can be found in the root of the kernel tree. Compressed image used for booting can be found under:

arch/arm/boot/Image, for arm32
arch/arm64/boot/Image, for arm64

$ file vmlinux
  vmlinux: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

$ file vmlinux
  vmlinux: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), statically linked, not stripped

Rootfs¶

The root filesystem (rootfs) is the filesystem mounted at the top of files hierarchy (/). It should contain at least the critical files allowing the system to boot to a shell.

root@so2$ tree -d -L 2
├── bin
├── boot
├── dev
├── etc
├── home
│   └── root
├── lib
│   └── udev
├── mnt
├── proc
├── sbin
│   └── init
├── sys
├── usr
│   ├── bin
│   ├── include
│   ├── lib
└── var

As for x86 we will make use of Yocto rootfs images. In order to download an ext4 rootfs image for arm32 one needs to run:

$ cd tools/labs/
$ ARCH=arm make core-image-minimal-qemuarm.ext4

Device tree¶

Device tree (DT) is a tree structure used to describe the hardware devices in a system. Each node in the tree describes a device hence it is called device node. DT was introduced to provide a way to discover non-discoverable hardware (e.g a device on an I2C bus). This information was previously stored inside the source code for the Linux kernel. This meant that each time we needed to modify a node for a device the kernel needed to be recompiled. This no longer holds true as device tree and kernel image are separate binaries now.

Device trees are stored inside device tree sources (.dts) and compiled into device tree blobs (.dtb).

# compile dtbs
$ make dtbs

# location for DT sources on arm32
$ ls arch/arm/boot/dts/
  imx6ul-14x14-evk.dtb imx6ull-14x14-evk.dtb bcm2835-rpi-a-plus.dts

# location for DT source on arm64
$ ls arch/arm64/boot/dts/<vendor>
  imx8mm-evk.dts imx8mp-evk.dts

The following image is a represantation of a simple device tree, describing board type, cpu and memory.

Notice that a device tree node can be defined using label: name@address:

label, is an identifier used to reference the node from other places

name, node identifier

address, used to differentiate nodes with the same name.

A node might contain several properties arranged in the name = value format. The name is a string and the value can be bytes, strings, array of strings.

Here is an example:

/ {
     node@0 {
          empty-property;
          string-property = "string value";
          string-list-property = "string value 1", "string value 2";
          int-list-property = <value1 value2>;

          child-node@0 {
                  child-empty-property;
                  child-string-property = "string value";
                  child-node-reference = <&child-node1>;
          };

          child-node1: child-node@1 {
                  child-empty-property;
                  child-string-property = "string value";
          };
   };
};

Qemu¶

We will use qemu-system-arm to boot 32bit ARM platforms. Although, this can be installed from official distro repos, for example:

sudo apt-get install -y qemu-system-arm

We strongly recommend using latest version of qemu-system-arm build from sources:

$ git clone https://gitlab.com/qemu-project/qemu.git
$ ./configure --target-list=arm-softmmu --disable-docs
$ make -j8
$ ./build/qemu-system-arm

Exercises¶

Important

We strongly encourage you to use the setup from this repository.

To solve exercises, you need to perform these steps:

prepare skeletons from templates
build modules
start the VM and test the module in the VM.

The current lab name is arm_kernel_development. See the exercises for the task name.

The skeleton code is generated from full source examples located in tools/labs/templates. To solve the tasks, start by generating the skeleton code for a complete lab:

tools/labs $ make clean
tools/labs $ LABS=<lab name> make skels

You can also generate the skeleton for a single task, using

tools/labs $ LABS=<lab name>/<task name> make skels

Once the skeleton drivers are generated, build the source:

tools/labs $ make build

Then, start the VM:

tools/labs $ make console

The modules are placed in /home/root/skels/arm_kernel_development/<task_name>.

You DO NOT need to STOP the VM when rebuilding modules! The local skels directory is shared with the VM.

Review the Exercises section for more detailed information.

Warning

Before starting the exercises or generating the skeletons, please run git pull inside the Linux repo, to make sure you have the latest version of the exercises.

If you have local changes, the pull command will fail. Check for local changes using git status. If you want to keep them, run git stash before pull and git stash pop after. To discard the changes, run git reset --hard master.

If you already generated the skeleton before git pull you will need to generate it again.

Warning

The rules for working with the virtual machine for ARM are modified as follows

# modules build
tools/labs $ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make build
# modules copy
tools/labs $ ARCH=arm make copy
# kernel build
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j8

0. Intro¶

Inspect the following locations in the Linux kernel code and identify platforms and vendors using ARM architecture:

32-bit: arch/arm/boot/dts

64-bit: arch/arm64/boot/dts

Use qemu and look at the supported platforms:

../qemu/build/arm-softmmu/qemu-system-arm -M ?

Note

We used our own compiled version of Qemu for arm32. See Qemu section for more details.

1. Boot¶

Use qemu to boot i.MX6UL platform. In order to boot, we first need to compile the kernel. Review Compiling the Linux kernel on ARM section.

Successful compilation will result in the following binaries:

arch/arm/boot/Image, kernel image compiled for ARM

arch/arm/boot/dts/imx6ul-14x14-evk.dtb, device tree blob for i.MX6UL board

Review Rootfs section and download core-image-minimal-qemuarm.ext4 rootfs. Run qemu using then following command:

../qemu/build/arm-softmmu/qemu-system-arm -M mcimx6ul-evk -cpu cortex-a7 -m 512M \
  -kernel arch/arm/boot/zImage -nographic  -dtb arch/arm/boot/dts/imx6ul-14x14-evk.dtb \
  -append "root=/dev/mmcblk0 rw console=ttymxc0 loglevel=8 earlycon printk" -sd tools/labs/core-image-minimal-qemuarm.ext4

Note

LCDIF and ASRC devices are not well supported with Qemu. Remove them from compilation.

$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make menuconfig
# set FSL_ASRC=n and DRM_MXSFB=n
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j8

Once the kernel is booted check kernel version and cpu info:

$ cat /proc/cpuinfo
$ cat /proc/version

2. CPU information¶

Inspect the CPU configuration for NXP i.MX6UL board. Start with arch/arm/boot/dts/imx6ul-14x14-evk.dts.

find cpu@0 device tree node and look for operating-points property.

read the maximum and minimum operating frequency the processor can run
$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq
$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq

3. I/O memory¶

Inspect I/O space configuration for NXP i.MX6UL board. Start with arch/arm/boot/dts/imx6ul-14x14-evk.dts and identify each device mentioned below.

$ cat /proc/iomem
  00900000-0091ffff : 900000.sram sram@900000
  0209c000-0209ffff : 209c000.gpio gpio@209c000
  021a0000-021a3fff : 21a0000.i2c i2c@21a0000
  80000000-9fffffff : System RAM

Identify device tree nodes corresponding to:

System RAM, look for memory@80000000 node in arch/arm/boot/dts/imx6ul-14x14-evk.dtsi. What's the size of the System RAM?

GPIO1, look for gpio@209c000 node in arch/arm/boot/dts/imx6ul.dtsi. What's the size of the I/O space for this device?

I2C1, look for i2c@21a0000 node in arch/arm/boot/dts/imx6ul.dtsi. What's the size of the I/O spaces for this device?

4. Hello World¶

Implement a simple kernel module that prints a message at load/unload time. Compile it and load it on i.MX6UL emulated platform.

# modules build
tools/labs $ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make build
# modules copy
tools/labs $ ARCH=arm make copy
# kernel build
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j8

5. Simple device¶

Implement a driver for a simple platform device. Find TODO 1 and notice how simple_driver is declared and register as a platform driver. Follow TODO 2 and add the so2,simple-device-v1 and so2,simple-device-v2 compatible strings in the simple_device_ids array.

Create two device tree nodes in arch/arm/boot/dts/imx6ul.dtsi under soc node with compatible strings so2,simple-device-v1 and so2,simple-device-v2 respectively. Then notice the behavior when loading simple_driver module.