GIGA: GC2145 fails when attached to the Giga Display Shiedl

[mjs513]

Retesting the Camera (GC2145) on the giga with the giga display showed that the zephyr video api changed which required updating the camera library, see PR #135 for changes. After making the changes camera visualizer sketch seems to do nothing, i.e, nothing sent to processing visualizer sketch etc,

Did notice that in debug window:

[00:00:17.789,000] <inf> usb_dc_stm32: PWR not active yet
[00:00:17.911,000] <inf> usb_cdc_acm: Device suspended
[00:00:18.136,000] <inf> usb_cdc_acm: Device configured
[00:00:23.078,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback
[00:01:17.598,000] <wrn> shell_uart: RX ring buffer full.
[00:01:17.604,000] <wrn> shell_uart: RX ring buffer full.

which does not show the usual spew when the camera is working.

However if I remove the shield and put camera directly on the giga it works.

[KurtE]

@mjs513, @iabdalkader - and all

Quick note: I have a quick and dirty stripped down version of a sketch to read the camera and output to the display
using GFX...

Most of the time it fails but I do get some frames..
It is up at: https://github.com/KurtE/Arduino_GIGA-stuff/tree/main/sketches/zephyr_giga_display_examples/Zephyr_GigaCameraDisplay_GFX

Most of the time it fails, I have had a few times when it actually reads a frames. I swap the bytes as they are in the wrong
order...

I am wondering if this might be some form of priority issue? Not sure if this version of the camera library is using DMA? Or interrupts. Will look later. Likewise for the display code. Or maybe issue with interrupts or threads priorities?...

Note sure if the thread list helps... but just in case:

uart:~$ kernel thread list
Scheduler: 253 since last call
Threads:
 0x24001110 input
        options: 0x0, priority: 14 timeout: 0
        state: pending, entry: 0x804a51d
        stack size 1024, unused 804, usage 220 / 1024 (21 %)

*0x24000d58 shell_uart
        options: 0x0, priority: 14 timeout: 0
        state: queued, entry: 0x8047aad
        stack size 32768, unused 31016, usage 1752 / 32768 (5 %)

 0x24003608 sysworkq
        options: 0x1, priority: -1 timeout: -9223372036854775808
        state: pending, entry: 0x8058945
        stack size 2048, unused 1576, usage 472 / 2048 (23 %)

 0x24000c40 usbworkq
        options: 0x0, priority: -1 timeout: 0
        state: pending, entry: 0x8058945
        stack size 1024, unused 728, usage 296 / 1024 (28 %)

 0x24003418 idle
        options: 0x1, priority: 15 timeout: 0
        state: , entry: 0x806277f
        stack size 320, unused 248, usage 72 / 320 (22 %)

 0x24003510 main
        options: 0x1, priority: 0 timeout: 17032
        state: pending, entry: 0x8058119
        stack size 32768, unused 31184, usage 1584 / 32768 (4 %)

uart:~$

I am running using @mjs513 updated version of camera...

[mjs513]

@KurtE - @iabdalkader - and all

I gave your sketch a shot and only once was I able to see more than 1 frame once!. Most of the time I will get 1 frame and then nothing:

uart:~$ sketch
[00:00:06.256,000] <inf> usb_dc_stm32: PWR not active yet
[00:00:06.378,000] <inf> usb_cdc_acm: Device suspended
height: 480, width: 800, rotated: 1
- Capabilities:
  x_resolution = 480, y_resolution = 800
, supported_pixel_formats = 25
  current_pixel_format = 16, current_orientation = 0
[00:00:06.509,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback
[00:00:06.603,000] <inf> usb_cdc_acm: Device configured
[00:02:37.817,000] <wrn> shell_uart: RX ring buffer full.
[00:02:37.823,000] <wrn> shell_uart: RX ring buffer full.
[00:02:37.829,000] <wrn> shell_uart: RX ring buffer full.
[00:02:37.835,000] <wrn> shell_uart: RX ring buffer full.

Update - now getting streaming video to display after changing pwm1 from 12 to 10Mhz.

Also now getting video to the Processing sketch but it seems to be out of synched

[KurtE]

Thanks @mjs513... I have the sketch updated, and I rotated the display and center the image and it works so far as you mentioned using 320x240 camera.

If I try setting the camera 640x480, I don't get any images... Need to see if there is issue with buffer sizing and the like.

So I then added some quick and dirty scale output code And try to output at scale 2.
320x240 output as 640x480 which fits in the 800x480 display. It looks like I get one output and then it fails...

Will look more tomorrow. (The updated sketch is up at the same place...)

[iabdalkader]

Yes, the display hogs the bus or the driver disables IRQs for too long, which causes the DCMI to overrun. This is a known issue, I've reported it before. Reducing the clock fixes it as it slows down the camera. Alternatively the DCMI could be restarted when it stops, but that makes frame drops very visible.

Anyway, here's a simplified camera+display sketch for testing.

#include "camera.h"
#include <zephyr/kernel.h>
#include <zephyr/device.h>
#include <zephyr/drivers/display.h>
#include <zephyr/multi_heap/shared_multi_heap.h>

Camera cam;
#define DISPLAY_W 320
#define DISPLAY_H 240

void fatal_error(const char *msg) {
    Serial.println(msg);
    pinMode(LED_BUILTIN, OUTPUT);
    while (1) {
        digitalWrite(LED_BUILTIN, HIGH);
        delay(100);
        digitalWrite(LED_BUILTIN, LOW);
        delay(100);
    }
}

const struct device *display_dev;
void setup(void) {
    Serial.begin(115200);

    if (!cam.begin(DISPLAY_W, DISPLAY_H, CAMERA_RGB565, true)) {
        fatal_error("Camera begin failed");
    }
    cam.setVerticalFlip(false);
    cam.setHorizontalMirror(false);

    display_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_display));
    if (!device_is_ready(display_dev)) {
        fatal_error("Device not found. Aborting sample");
    }

    
    struct display_capabilities caps;
    display_get_capabilities(display_dev, &caps);    
    display_blanking_on(display_dev);
    memset(display_get_framebuffer(display_dev), 0, caps.x_resolution * caps.y_resolution * 2);
    display_blanking_off(display_dev);
    display_set_orientation(display_dev, DISPLAY_ORIENTATION_ROTATED_90);
}

void loop() {
    FrameBuffer fb;
    static struct display_buffer_descriptor desc = {
      .buf_size = DISPLAY_W * DISPLAY_H * 2,
      .width = DISPLAY_W,
      .height = DISPLAY_H,
      .pitch = DISPLAY_W,
    };

    if (cam.grabFrame(fb, 100)) {
        display_write(display_dev, 0, 0, &desc, fb.getBuffer());
        cam.releaseFrame(fb);
    }
}

[KurtE]

Thanks @iabdalkader - I just tried this sketch and it appears to work for maybe one frame and then it is hung or simply
not reading or display more frames.

My hacked sketch which I mentioned appears to work OK at 320x240 but not bigger. If I try 640x480 nothing displays.
And I think I am seeing on the printk monitor window messages like:
[00:00:03.146,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo
If instead I try to use code to print every pixel as a 2x2 rectangle, I get one output and again camera fails to read.
Maybe need to slow it down some more

[mjs513]

@iabdalkader just to add on to what @KurtE said this is what I am seeing in the debug window when trying 640x480

[00:00:14.871,000] <err> dma_stm32_v1: FiFo error.
[00:00:14.877,000] <inf> dma_stm32_v1: tc: 0, ht: 1, te: 0, dme: 0, fe: 1
[00:00:14.884,000] <err> dma_stm32: Unexpected irq happened.
[00:00:14.890,000] <err> video_stm32_dcmi: DMA callback error with channel 0.
[00:00:14.898,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback
[00:00:14.905,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback
[00:00:14.912,000] <err> dma_stm32_v1: FiFo error.
[00:00:14.917,000] <inf> dma_stm32_v1: tc: 1, ht: 1, te: 0, dme: 0, fe: 1
[00:00:14.924,000] <err> dma_stm32: Unexpected irq happened.
[00:00:14.931,000] <err> video_stm32_dcmi: DMA callback error with channel 0.
[00:00:14.939,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback
[00:00:15.009,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback
[00:00:15.015,000] <err> usb_dc_stm32: Unable to get write lock (-16)
[00:00:15.022,000] <wrn> usb_device: Failed to write endpoint buffer 0x80
[00:00:15.030,000] <err> usb_dc_stm32: Unable to get write lock (-16)
[00:00:15.037,000] <wrn> usb_device: Failed to write endpoint buffer 0x80
[00:00:15.044,000] <err> usb_dc_stm32: Unable to get write lock (-16)

[mjs513]

@KurtE - @iabdalkader
As I just mentioned to Kurt I changed pwm1 to 9mhz and @KurtE sketch is now working with scaling factor of 2

[KurtE]

Thanks... I tried 9 and it did several frames before it hung... Or I should say until the if (cam.grabFrame(fb)) {
started failing and the BUILTIN_LED would slowly blink...

[iabdalkader]

@KurtE @mjs513 I think the FIFO was just disabled for some reason. Does this patch help?

This makes my sketch above (the only test I run) work fine at 12MHz. Lowering LTDC IRQ (currently 0) is also an option.

diff --git a/drivers/video/video_stm32_dcmi.c b/drivers/video/video_stm32_dcmi.c
index f27a06da16b..ede4ea620a2 100644
--- a/drivers/video/video_stm32_dcmi.c
+++ b/drivers/video/video_stm32_dcmi.c
@@ -155,8 +155,11 @@ static int stm32_dma_init(const struct device *dev)
 	hdma.Init.PeriphDataAlignment	= DMA_PDATAALIGN_WORD;
 	hdma.Init.MemDataAlignment	= DMA_MDATAALIGN_WORD;
 	hdma.Init.Mode			= DMA_CIRCULAR;
-	hdma.Init.Priority		= DMA_PRIORITY_HIGH;
-	hdma.Init.FIFOMode		= DMA_FIFOMODE_DISABLE;
+	hdma.Init.Priority		= DMA_PRIORITY_VERY_HIGH;
+	hdma.Init.FIFOMode		= DMA_FIFOMODE_ENABLE;
+	hdma.Init.FIFOThreshold = DMA_FIFO_THRESHOLD_FULL;
+	hdma.Init.MemBurst = DMA_MBURST_INC4;
+	hdma.Init.PeriphBurst = DMA_PBURST_SINGLE;

[KurtE]

Updated to 8mhz and it appears to work a lot longer. I do get some of the messages like:

[00:01:42.923,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo
[00:01:43.314,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo
[00:01:43.705,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo
[00:01:44.096,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo
[00:01:44.487,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo
[00:01:44.878,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo

I tried then with native 640x480... not scaled up... And it fails to show any frames and the Kernel

uart:~$
[00:00:01.344,000] <inf> usb_cdc_acm: Device suspended
[00:00:01.592,000] <inf> usb_cdc_acm: Device configured
height: 480, width: 800, rotated: 1
- Capabilities:
  x_resolution = 480, y_resolution = 800
, supported_pixel_formats = 89
  current_pixel_format = 16, cur[00:00:04.874,000] <err> dma_stm32_v1: FiFo error.
[00:00:04.882,000] <inf> dma_stm32_v1: tc: 0, ht: 1, te: 0, dme: 0, fe: 1
[00:00:04.891,000] <err> dma_stm32: Unexpected irq happened.
[00:00:04.900,000] <err> video_stm32_dcmi: DMA callback error with channel 0.
[00:00:04.910,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback
uart:~$ entation = 0
uart:~$

Edit: @iabdalkader Will try your camera patch and see if that helps... Thanks

[mjs513]

@iabdalkader - @KurtE

Sorry had to run out to get new frames (pun not intentional) for my glasses.

Anyways applied your patch and:

320x240 @12mhz - getting frames and is stable
320x240 @12 mhz with scaleing = 2 - get a couple of frames on display and then freezes.

640x480 @12 mhz - seeing same errors as I saw before.

@10MHZ w/320x240 and scaling set to 2 - several frames now and freezes still.
@8MHz w/320x240 and scaling set to 2 - no issues scaling successful

UPDATE: at 640x480 actually getting frames that out of synch with the patch applied and at 8mhx

[mjs513]

@iabdalkader - @KurtE

Think this issue is has been resolved by adjusting the pwm1 clock. However, think there are still some issues with working with the camera and display. Do you all want to continue it here or make it a discussion?

[iabdalkader]

320x240 @12mhz - getting frames and is stable

So it does fix that at least.

640x480 @12 mhz - seeing same errors as I saw before.

How do you display 640x480? The display is 480x800, do you rotate the image somehow? Can you share your sketch?

[mjs513]

How do you display 640x480? The display is 480x800, do you rotate the image somehow? Can you share your sketch?

@iabdalkader

Before getting into the graphics api (Arduino_GraphicsDisplay and Arduino_Graphics_GFX) did find the following

1. @320x240 12 Mhz works and stable as I mentioned. (seeing 47.61 fps @6MHz)
2. @320x240 with scaling works and stable at 8mhz (seeing 19.76 fps @6MHz).
3. @640x480 works and fairly stable at 6Mhz. (seeing 9.61 fps @6MHz). Still get the errors in (GIGA: GC2145 fails when attached to the Giga Display Shiedl #136 (comment) but getting descent images.
4. display_set_orientation(display_dev, DISPLAY_ORIENTATION_ROTATED_90); does not work to rotate screen. Saw that when we were working on the api.

Now for the api's. We created a display api and then moded arduino_graphics_gfx (#92 (comment)). You will need those 2 libraries to run the sketch. Note: libs work with both mbed and zephyr used ifdefs as appropriate.

Then sketch we are using to test was set up by @KurtE:

#include <elapsedMillis.h>

#include "Arduino_GigaDisplay_GFX.h"
#define GC9A01A_CYAN 0x07FF
#define GC9A01A_RED 0xf800
#define GC9A01A_BLUE 0x001F
#define GC9A01A_GREEN 0x07E0
#define GC9A01A_MAGENTA 0xF81F
#define GC9A01A_WHITE 0xffff
#define GC9A01A_BLACK 0x0000
#define GC9A01A_YELLOW 0xFFE0
#define ALIGN_PTR(p, a) ((p & (a - 1)) ? (((uintptr_t)p + a) & ~(uintptr_t)(a - 1)) : p)

#define CANVAS_ROTATION 0
uint16_t color565(uint8_t r, uint8_t g, uint8_t b) {
  return ((r & 0xF8) << 8) | ((g & 0xFC) << 3) | (b >> 3);
}

#define CAMERA_WIDTH 640
#define CAMERA_HEIGHT 480
#define SCALE 1

#include <camera.h>
 
Camera cam;

// The buffer used to capture the frame
FrameBuffer fb;

#ifdef ROTATE_CAMERA_IMAGE
uint16_t *rotate_buffer = nullptr;
#endif


// The buffer used to rotate and resize the frame
GigaDisplay_GFX display;

void blinkLED(uint32_t count = 0xFFFFFFFF) {
  pinMode(LED_BUILTIN, OUTPUT);
  while (count--) {
    digitalWrite(LED_BUILTIN, LOW);   // turn the LED on (HIGH is the voltage level)
    delay(50);                        // wait for a second
    digitalWrite(LED_BUILTIN, HIGH);  // turn the LED off by making the voltage LOW
    delay(50);                        // wait for a second
  }
}

void fatal_error(const char *msg) {
  Serial.println(msg);
  pinMode(LED_BUILTIN, OUTPUT);
  while (1) {
    digitalWrite(LED_BUILTIN, HIGH);
    delay(100);
    digitalWrite(LED_BUILTIN, LOW);
    delay(100);
  }
}



void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  while (!Serial && millis() < 5000) {}
  Serial.begin(115200);

  // Init the cam QVGA, 30FPS
  Serial.println("Before camera start");
  Serial.flush();
  //cam.debug(Serial);
  if (!cam.begin(CAMERA_WIDTH, CAMERA_HEIGHT, CAMERA_RGB565, true)) {
    fatal_error("Camera begin failed");
  }
  //Serial.println("Before setRotation");
  //Serial.flush();

  display.begin();
  elapsedMicros em;
  display.setRotation(1);
  display.fillScreen(GC9A01A_BLUE);
  Serial.println(em, DEC);
  Serial.print("Camera Width: "); Serial.print(CAMERA_WIDTH);
  Serial.print(" Height: "); Serial.println(CAMERA_HEIGHT);
  Serial.print("Screen Width: "); Serial.print(display.width());
  Serial.print(" Height: "); Serial.println(display.height());
  Serial.print("Scale: "); Serial.println(SCALE);
  Serial.println("end setup");
  Serial.flush();
}

inline uint16_t HTONS(uint16_t x) {
  return ((x >> 8) & 0x00FF) | ((x << 8) & 0xFF00);
}
//#define HTONS(x) (((x >> 8) & 0x00FF) | ((x << 8) & 0xFF00))


uint32_t display_time_sum = 0;
uint8_t display_time_count = 0;

void loop() {

  // Grab frame and write to another framebuffer
  if (cam.grabFrame(fb, 100)) {
    //Serial.println("Camera frame received");
    if (Serial.available()) {
      while (Serial.read() != -1) {}
//      cam.printRegs();
      //MemoryHexDump(Serial, fb.getBuffer(), 1024, true, "Start of Camera Buffer\n");
      Serial.println("*** Paused ***");
      while (Serial.read() == -1) {}
      while (Serial.read() != -1) {}
    }

    uint16_t *pixels = (uint16_t *)fb.getBuffer();
    elapsedMicros emDisplay;
    //for (int i = 0; i < CAMERA_WIDTH*CAMERA_HEIGHT; i++) pixels[i] = HTONS(pixels[i]);
    #if defined(SCALE) && SCALE > 1
    // Quick and dirty scale.
    int yDisplay = (display.height() - (CAMERA_HEIGHT * SCALE)) / 2;
    display.startBuffering();
    for (int yCamera = 0; yCamera < CAMERA_HEIGHT; yCamera++) {
      int xDisplay = (display.width() - (CAMERA_WIDTH * SCALE)) / 2;
      for (int xCamera = 0; xCamera < CAMERA_WIDTH; xCamera++) {
        display.fillRect(xDisplay, yDisplay, SCALE, SCALE, *pixels++);
        xDisplay += SCALE;
      }
      yDisplay += SCALE;
    }
    display.endBuffering();
    #else  
    display.drawRGBBitmap((display.width() - CAMERA_WIDTH) / 2, (display.height() - CAMERA_HEIGHT) / 2, pixels, CAMERA_WIDTH, CAMERA_HEIGHT);
    #endif

    cam.releaseFrame(fb);

    display_time_sum += emDisplay;
    display_time_count++;
    if (display_time_count == 128) {
      Serial.print("Avg display Time: ");
      Serial.print(display_time_sum / display_time_count);
      Serial.print(" fps:");
      Serial.println(128000000.0 / float(display_time_sum), 2);
      display_time_sum = 0;
      display_time_count = 0;
    }

  } else {

    digitalWrite(LED_BUILTIN, digitalRead(LED_BUILTIN)? LOW : HIGH);
  }
  delay(1);
}