TCG translation cache functioning?

[coden-s]

Is the TCG translation cache functional in unicorn? I know that unicorn uses a relatively old version of QEMU (and therefore TCG), but I was wondering if there’s any known pre-existing issues with the translation cache in unicorn.
Using a basic matrix multiplication test to check the performance between QEMU and unicorn, we got average runtimes of 0.618s (unicorn) vs 0.028s (QEMU) for 64x64 matrices. The performance difference at larger matrix sizes seems to suggest that the cache isn’t working.
We also manually set the cache size to 0 by redefining DEFAULT_CODE_GEN_BUFFER_SIZE_1 in TCG’s translate-all.c (I’m not 100% sure that this is the right/only control for the cache, but it’s what we could find). The runtime was nearly identical when compared to the default cache size of 1GB, which further supports the hypothesis that it’s not working properly.

[PhilippTakacs]

There are several issues related to the TB Cache (i.e. #2258) so I would guess the Cache is working.

Can you share your test code (including the sample blob you emulate)?

Did you use the count parameter on uc_emu_start? This would lead to a single step emulation.

Did you register any hooks? Depending on the hook (code hook or memory hook) this might lead to a massive slowdown because unicorn needs to disable some optimization to be able to call this hooks.

[coden-s]

I'm not using the count parameter on uc_emu_start, but I still had some hooks registered which was definitely slowing things down.
However, it still seems slower that it should be. I made a new test which calculates Fibonacci numbers (similarly computationally intensive to matrix multiplication but without arguments and with fewer inconsistencies related to memory usage).

Like with my previous test, unicorn is faster than QEMU at smaller sizes but is 100x slower when n = 10000000 (unicorn = 5.121s, QEMU = 0.052s, native = 0.015s). Do you think this means that the translation cache is not working, or is there another explanation? Is there a more robust way to verify this?
It is worth noting that QEMU uses a statically linked .exe, while the unicorn test uses a dynamically linked .exe.

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

// aarch64-linux-gnu-gcc -o ../fib fib.c -O0

int main()
{
    int n = 10000000;
    uint64_t a = 0, b = 1, c;
    for (int i = 2; i <= n; i++)
    {
        c = a + b;
        a = b;
        b = c;
    }

    return 0;
}

#include <string.h>
#include <uc_log.h>
#include <fcntl.h>
#include <elf.h>
#include <math.h>
#include <unistd.h>
#include <sys/mman.h>
#include <unicorn/unicorn.h>

#define STACK_SIZE 0x1000
#define PAGE_SIZE 0x1000 // 4KB page size (4 mb)

static uint64_t code_addr = 0;
static void *code_ptr = NULL;
static void *stack_ptr = NULL;

int init_unicorn_a64(uc_engine **uc, uc_err *err, const char *file)
{
  int fd = open(file, O_RDONLY);
  if (fd < 0)
  {
    perror("Error opening file");
    return 1;
  }

  off_t file_size = lseek(fd, 0, SEEK_END);
  lseek(fd, 0, SEEK_SET);

  Elf64_Ehdr *ehdr =
      (Elf64_Ehdr *)mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
  if (ehdr == MAP_FAILED)
  {
    perror("Error mapping ELF file");
    close(fd);
    return 1;
  }

  if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) != 0 ||
      ehdr->e_machine != EM_AARCH64)
  {
    fprintf(stderr, "Error: Not a valid ARM ELF file\n");
    munmap(ehdr, file_size);
    close(fd);
    return 1;
  }

  // Allocate memory aligned to a page boundary for the AArch64 code
  size_t num_pages = (file_size + PAGE_SIZE - 1) / PAGE_SIZE;
  size_t total_size = num_pages * PAGE_SIZE;
  if (posix_memalign(&code_ptr, PAGE_SIZE, total_size) != 0)
  {
    perror("posix_memalign");
    return 1;
  }

  // Allocate memory aligned to a page boundary for the stack
  if (posix_memalign(&stack_ptr, PAGE_SIZE, STACK_SIZE) != 0)
  {
    perror("posix_memalign");
    return 1;
  }

  if (code_ptr == NULL)
  {
    perror("Error allocating memory for ELF file");
    munmap(ehdr, file_size);
    close(fd);
    return 1;
  }
  memcpy(code_ptr, ehdr, file_size);

  code_addr = (uint64_t)code_ptr;

  munmap(ehdr, file_size);
  close(fd);

  if ((*err = uc_open(UC_ARCH_ARM64, UC_MODE_ARM, uc)) != UC_ERR_OK)
  {
    fprintf(stderr, "Unicorn init failed: %s\n", uc_strerror(*err));
    return 1;
  }

  // make stack
  void *stack_ptr = malloc(STACK_SIZE);
  if (stack_ptr == NULL)
  {
    perror("[!] Error allocating memory for stack");
    return 1;
  }

  uint64_t sp = (uint64_t)stack_ptr + STACK_SIZE - 256;
  uc_reg_write(*uc, UC_ARM64_REG_SP, &sp);

  // map code
  if (uc_mem_map_ptr(*uc, 0x10000, 0xffffffffffff0000ULL, UC_PROT_ALL,
                     (void *)0x10000) != UC_ERR_OK)
  {
    perror("Error mapping host memory.\n");
    return 1;
  }

  return 0;
}

int emulate_function_a64(uc_engine *uc, uc_err err)
{
  // Hard coded function info:
  uint64_t code_start = 0x718;
  uint64_t code_len = 0x6c - 0x4;

  uint64_t start_addr = code_addr + code_start;
  err = uc_emu_start(uc, start_addr, start_addr + code_len, 0, 0);
  return 0;
}

int close_unicorn_a64(uc_engine **uc)
{
  if (*uc)
  {
    uc_close(*uc);
    *uc = NULL;
  }
  if (code_ptr)
  {
    free(code_ptr);
    code_ptr = NULL;
  }
  if (stack_ptr)
  {
    free(stack_ptr);
    stack_ptr = NULL;
  }
  return 0;
}

int main()
{
  // Initialize Unicorn Engine and emulator memory
  uc_engine *uc;
  uc_err err;

  const char *input_filename = AARCH64_INPUTS_PATH "fib";

  init_unicorn_a64(&uc, &err, input_filename);

  emulate_function_a64(uc, err);

  // Cleanup and free pointers
  close_unicorn_a64(&uc);
  return 0;
}

[PhilippTakacs]

I had only a quick look but I would guess the problem is the following line:

  if (uc_mem_map_ptr(*uc, 0x10000, 0xffffffffffff0000ULL, UC_PROT_ALL,
                     (void *)0x10000) != UC_ERR_OK)

This maps nearly the complete address space into uncorn, which makes memory handling a bit hard. You can try only mapping the two relevant memory regions (stack and code). I'll do some tests with your code in a few days.

Btw: mapping the complete memory out of your process might also cause some bugs, if the emulated code access memory from your programm.

[coden-s]

Here's an updated version of the code where I've tried splitting up the memory mapping.

Rerunning the test (with n = 10000000 in the sample blob), this version averages 4.08s, which is over a second faster than the original version but still over 75x worse than QEMU.

#include <string.h>
#include <uc_log.h>
#include <fcntl.h>
#include <elf.h>
#include <math.h>
#include <unistd.h>
#include <sys/mman.h>
#include <unicorn/unicorn.h>

#define PAGE_SIZE 0x1000 // 4KB page size

static void *code_ptr = NULL;

#define GUEST_CODE_BASE 0x100000
#define GUEST_STACK_BASE 0x200000
#define GUEST_STACK_SIZE 0x10000 // 64KB

int init_unicorn_a64(uc_engine **uc, uc_err *err, const char *file)
{
    int fd = open(file, O_RDONLY);
    if (fd < 0)
    {
        perror("Error opening file");
        return 1;
    }

    off_t file_size = lseek(fd, 0, SEEK_END);
    lseek(fd, 0, SEEK_SET);

    Elf64_Ehdr *ehdr =
        (Elf64_Ehdr *)mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
    if (ehdr == MAP_FAILED)
    {
        perror("Error mapping ELF file");
        close(fd);
        return 1;
    }

    if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) != 0 ||
        ehdr->e_machine != EM_AARCH64)
    {
        fprintf(stderr, "Error: Not a valid ARM ELF file\n");
        munmap(ehdr, file_size);
        close(fd);
        return 1;
    }

    // Allocate memory aligned to a page boundary for the AArch64 code
    size_t num_pages = (file_size + PAGE_SIZE - 1) / PAGE_SIZE;
    size_t total_size = num_pages * PAGE_SIZE;
    if (posix_memalign(&code_ptr, PAGE_SIZE, total_size) != 0)
    {
        perror("posix_memalign");
        return 1;
    }

    if (code_ptr == NULL)
    {
        perror("Error allocating memory for ELF file");
        munmap(ehdr, file_size);
        close(fd);
        return 1;
    }
    memcpy(code_ptr, ehdr, file_size);

    munmap(ehdr, file_size);
    close(fd);

    if ((*err = uc_open(UC_ARCH_ARM64, UC_MODE_ARM, uc)) != UC_ERR_OK)
    {
        fprintf(stderr, "Unicorn init failed: %s\n", uc_strerror(*err));
        return 1;
    }

    *err = uc_mem_map(*uc, GUEST_CODE_BASE, total_size, UC_PROT_READ | UC_PROT_EXEC);
    if (*err != UC_ERR_OK)
    {
        fprintf(stderr, "Failed to map code memory: %s\n", uc_strerror(*err));
        return 1;
    }

    *err = uc_mem_write(*uc, GUEST_CODE_BASE, code_ptr, file_size);
    if (*err != UC_ERR_OK)
    {
        fprintf(stderr, "Failed to write code to guest memory: %s\n", uc_strerror(*err));
        return 1;
    }

    *err = uc_mem_map(*uc, GUEST_STACK_BASE, GUEST_STACK_SIZE, UC_PROT_READ | UC_PROT_WRITE);
    if (*err != UC_ERR_OK)
    {
        fprintf(stderr, "Failed to map stack memory: %s\n", uc_strerror(*err));
        return 1;
    }

    uint64_t sp = GUEST_STACK_BASE + GUEST_STACK_SIZE - 0x10;
    uc_reg_write(*uc, UC_ARM64_REG_SP, &sp);

    return 0;
}

int emulate_function_a64(uc_engine *uc, uc_err err)
{
    // Hard coded function info:
    uint64_t code_start = 0x718;
    uint64_t code_len = 0x784 - 0x718;

    uint64_t start_addr = GUEST_CODE_BASE + code_start;
    uint64_t end_addr = start_addr + code_len;

    err = uc_emu_start(uc, start_addr, end_addr, 0, 0);

    return 0;
}

int close_unicorn_a64(uc_engine **uc)
{
    if (*uc)
    {
        uc_close(*uc);
        *uc = NULL;
    }
    if (code_ptr)
    {
        free(code_ptr);
        code_ptr = NULL;
    }
    return 0;
}

int main()
{
    // Initialize Unicorn Engine and emulator memory
    uc_engine *uc;
    uc_err err;

    const char *input_filename = AARCH64_INPUTS_PATH "fib";

    init_unicorn_a64(&uc, &err, input_filename);

    emulate_function_a64(uc, err);

    // Cleanup and free pointers
    close_unicorn_a64(&uc);
    return 0;
}

[PhilippTakacs]

What exactly do you measure, the execution of your binary or the emulation (uc_emu_start)?

If you measure the execution of your binary, you might mostly measure the time of your setup code and not the emulation itself. A better way to measure would be to do your setup (mmap code, allocate stack, ...), read the current time, setup unicorn (uc_open, uc_mem_map_ptr, uc_reg_write) read the current time, run uc_emu_start, read the current time. This way you can exactly see which part (setup or emulation) takes what time. To get better results do this is in a loop and calculate the mean values.

Your code copies the emulation binary two times, first with memcpy then with uc_mem_write. To speed this up you could simply use uc_mem_map_ptr with the mmaped file.

[coden-s]

Thanks for the catch where I copied the binary twice.
I’m measuring the entire execution to give a fairer comparison with QEMU which also includes the setup cost.
I’ve now also timed the different elements of the test as you suggested, and the setup time is negligible when N = 10000000 as 99.97% of the execution time comes from emulation (setup = 0.001s, emulation = 4.28s). The setup time doesn’t appear to be the cause of the degraded performance.
In my original testing, as we increased the fib size (N) from 1000 to 10000000, we see a pattern emerge where unicorn is initially faster than QEMU (at N = 1000, unicorn = 0.005s and QEMU = 0.0095s) but at larger N values, QEMU becomes meaningfully faster by comparison. This suggests that compared to unicorn, QEMU has a larger startup overhead, but also is faster at emulating.
My goal is not to beat the performance of QEMU, the degraded performance just gave me reason to believe that the translation cache (or some other mechanism) may not be functioning correctly in my example. Do you know of any way to robustly verify that the translation cache is working as intended?