'RISCV-vector clang causes illegal instruction in spike

I hope you can help me on my RISC-V issue.

I am currently experimenting with the toolchain support for RISC-V's vector (RVV) instructions. From what I found on the internet, the spec is currently frozen at v1.0. GCC has a support RVV, but it is not actively maintained anymore. LLVM on the other hand has support for RVV.

So I went ahead and set up a Docker container with: RISC-V tools (Repo at basic-rvv), spike (latest commit) and LLVM (latest commit).

Next, I compiled a sgemm example with the following command: clang -march=rv32gcv --target=riscv32 --sysroot=/usr/local/riscv32-unknown-elf --gcc-toolchain=/usr/local -O2 sgemm.c -o sgemm.elf. The command runs successfully and I get an elf file, which looks fine using objdump: it uses the vector instructions.

Now to my issue: Afterwards, I want to verify the binary with the instruction set simulator spike. Therefore, I ran: spike /usr/local/riscv32-unknown-elf/bin/pk sgemm.elf, which ends up in an execution of an illegal instruction (see below for the full error message). The following OP fails: 0xb2905457. I decoded the instruction with echo "DASM(0xb2905457)" | spike-dasm --> vfmacc.vf v8, v9, ft0, which looks fine to me.

I already went through the code of spike, and why it might fail, but I got lost.

Maybe you have an idea what's going wrong here? I have the feeling that my vector unit is misconfigured (setvl instructions). I hope you guys can give me some support on this!

Thanks very much in advance! Tim

Error message from spike:

bbl loader
z  00000000 ra 000103cc sp 7ffffd70 gp 00020810
tp 00000000 t0 00000020 t1 bf06fb33 t2 00000000
s0 00020090 s1 00020b54 a0 00000004 a1 00020000
a2 00020010 a3 00000004 a4 00020b94 a5 0000001c
a6 bfed957a a7 00020b94 s2 00000000 s3 00000000
s4 00000000 s5 00000000 s6 00000000 s7 00000000
s8 00000000 s9 00000000 sA 00000000 sB 00000000
t3 3ea13dab t4 bf4b3713 t5 3ea6844f t6 3fdfe3d3
pc 000103ea va/inst b2905457 sr 80006620
An illegal instruction was executed!
clangvectorizationriscvinstruction-setriscv32


Solution 1:[1]

You should try

spike --isa=rv64gcv --varch=vlen:128,elen:64 /usr/local/riscv32-unknown-elf/bin/pk sgemm.elf

Add --isa=rv64gcv --varch=vlen:128,elen:64 in the spike run command.

Solution 2:[2]

so I think over time I found a solution. However, I forgot to post it until now. In fact, there was something going wrong in building the Docker container and maybe also during the build process. The error was originating from an earlier point in executing the elf in spike.

I can only tell everyone to check spike -d $PK $ELF 2> debug.txt to see where things went wrong.

Anyhow, I attached the Dockerfile and the Makefile for anyone how might run into the same issue.

Dockerfile:

FROM gcc:11.2
RUN apt update
RUN apt install -y autoconf automake autotools-dev curl python3 gawk \
    build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev \
    libexpat-dev cmake vim device-tree-compiler libmpc-dev libmpfr-dev \
    gdb zsh tmux libgmp-dev && \
    apt clean -y && \
    apt autoremove -y

#environemnt
ARG CFLAGS=-D__riscv_compressed

#install toolchain
RUN mkdir riscv-gnu-toolchain && cd riscv-gnu-toolchain && \
    git clone https://github.com/riscv/riscv-gnu-toolchain . && \
    git fetch && \
    git checkout basic-rvv && \
    git submodule update --init --recursive && \
    ./configure --with-arch=rv32gc --with-abi=ilp32d && \
    make -j32 && \
    make install && \
    cd .. && \ 
    rm riscv-gnu-toolchain -rf

#install spike
RUN mkdir -p /build/spike/build /build/spike/repo && cd /build/spike && \
    git clone https://github.com/riscv/riscv-isa-sim.git repo && \
    cd /build/spike/build && \
    ../repo/configure --with-varch=vlen:128,elen:32 --with-isa=rv32imafcv && \
    make && make install && \
    cd /build && \
    rm /build/spike -rf

# install pk
RUN mkdir -p /build/pk/build /build/pk/repo && cd /build/pk && \
    git clone https://github.com/riscv/riscv-pk.git repo && \
    cd /build/pk/build && \ 
    ../repo/configure --host=riscv32-unknown-elf --with-arch=rv32gc CC=riscv32-unknown-elf-gcc --with-abi=ilp32d && \
    make && make install && \
    cd /build && \
    rm /build/pk -rf

# install llvm
RUN mkdir -p /build/llvm && cd /build/llvm && \
    git clone https://github.com/llvm/llvm-project . && \
    mkdir build && cd build && \ 
    cmake -G "Unix Makefiles" -DLLVM_TARGETS_TO_BUILD="RISCV" \
    -DLLVM_DEFAULT_TARGET_TRIPLE=riscv32-unknown-elf \
    -DCMAKE_BUILD_TYPE=Release -DDEFAULT_SYSROOT=/usr/local/riscv32-unknown-elf \ 
    -DLLVM_ENABLE_PROJECTS="clang;lld" ../llvm && \
    make -j32 && \
    make install && \
    cd /build && \
    rm /build/llvm -rf

WORKDIR /root

Makefile (assuming the only file is test_sgemm.c)

PREFIX = riscv32-unknown-elf
AR = $(PREFIX)-ar
CC = $(PREFIX)-gcc
CLANG = clang
MARCH = rv32gcv

INCLUDES = 
CFLAGS = -march=$(MARCH) --target=riscv32 \
    --sysroot=/usr/local/riscv32-unknown-elf \
    --gcc-toolchain=/usr/local \
    -g

OBJS = $(patsubst %.c, %_$(MARCH).o, $(wildcard *.c))

all: test_sgemm.elf

%.o: %.c $(DEPS)
    $(CLANG) -c -o $@ $< $(CFLAGS) $(INCLUDES)

%.elf: %.o
    $(CLANG) -o $@ $< $(CFLAGS)

Hope this helps you!

Thanks Tim

Solution 3:[3]

GCC now support RVV intrinsics and auto-vectorization now. You can checkout "riscv-gcc-rvv-next" branch in riscv-gcc directory and "riscv-binutils-2.38" in riscv-binutils directory. Then build up the whole toolchain.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Jeremy Caney
Solution 2 Tim H
Solution 3 tonyzhong

Related Questions