diff --git a/README.rst b/README.rst index 5bc2912..2cc548c 100644 --- a/README.rst +++ b/README.rst @@ -138,18 +138,75 @@ The first way is to simply get the pre-built OpenCV library in `esp32/lib/`_ fol Fast way: --------- -The second way is by using the script in build_opencv_for_esp32.sh_. This script automatically compiles OpenCV from this repository sources, and install the needed files into the desired project. It can tweaked as needed to add and remove some parts. - -The script has 2 arguments. The first is the path to the ``toolchain-esp32.cmake`` (default is ``$HOME/esp/esp-idf/tools/cmake/toolchain-esp32.cmake``), and the second is the path where the OpenCV library is installed (default is in esp32/lib). +The second way is by using the script in build_opencv_for_esp32.sh_. This script automatically compiles OpenCV from this repository sources, and install the needed files into the desired project. It can be tweaked as needed to add and remove some parts (see `esp32/doc/build_configurations.md`_). .. _build_opencv_for_esp32.sh: esp32/scripts/build_opencv_for_esp32.sh +.. _`esp32/doc/build_configurations.md`: esp32/doc/build_configurations.md + +The script has 2 arguments. The first is the path to the ``toolchain-esp32.cmake`` (default is ``$HOME/esp/esp-idf/tools/cmake/toolchain-esp32.cmake``), and the second is the path where the OpenCV library is installed (default is in ``./esp32/lib``). + Detailed way: ------------- -The last way explains all the commands and modifications done to be able to compile and run OpenCV on the ESP32. The detailed procedure is in detailed_build_procedure.md_. +The last way explains all the commands and modifications done to be able to compile and run OpenCV on the ESP32. The detailed procedure is in `esp32/doc/detailed_build_procedure.md`_. + +.. _`esp32/doc/detailed_build_procedure.md`: esp32/doc/detailed_build_procedure.md + + + +Compiling esp-idf project using OpenCV: +======================================= + +When the OpenCV library is cross-compiled, we have in result ``*.a`` files located in ``build/lib`` folder. We now want to try to compile an example project using OpenCV on the esp32. A basic example of esp-idf project can be found in `esp32/examples/esp_opencv_basic/`_. This project simply creates an OpenCV matrix, fill it with values and prints it on the console. It's only purpose is to test the installation. + +.. _`esp32/examples/esp_opencv_basic/`: esp32/examples/esp_opencv_basic/ + +Esp-idf environment uses cmake and is separated in components. Because OpenCV libs were compiled outside this example project, we use the pre-built library functionality of esp-idf (https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/build-system.html#using-prebuilt-libraries-with-components). + +Here are the things done to add the OpenCV library to the project: + +- Copy the ``opencv/`` folder (from `esp32/lib/`_) into your project's ``main/`` component's folder. It contains the generated libraries (``libade.a``, ``libopencv_core.a``, ``libopencv_imgproc.a`` and ``libopencv_imgcodecs.a``) and some needed header files (e.g. ``cvconfig.h``, ``opencv_modules.hpp``, ``core.hpp``, ``core/`` folder, etc..). + +.. _`esp32/lib/`: esp32/lib/ + +- Link the libraries to the project by modifying the ``CMakeList.txt`` of the ``main`` project's component as below : + + .. code:: cmake + + idf_component_register( + SRC main.cpp + INCLUDE_DIRS ./opencv + ) + + add_prebuilt_library(opencv_imgcodecs "opencv/libopencv_imgcodecs.a") + add_prebuilt_library(libpng "opencv/3rdparty/liblibpng.a") + add_prebuilt_library(libzlib "opencv/3rdparty/libzlib.a") + add_prebuilt_library(opencv_imgproc "opencv/libopencv_imgproc.a") + add_prebuilt_library(opencv_core "opencv/libopencv_core.a") + add_prebuilt_library(ade "opencv/libade.a") + + set(IMGCODEC_DEP libpng libzlib) + target_link_libraries(opencv_imgcodecs INTERFACE ${IMGCODEC_DEP}) + + set(CORE_DEP libzlib) + target_link_libraries(opencv_core INTERFACE ${CORE_DEP}) + + set(OPENCV_DEP opencv_imgcodecs opencv_imgproc opencv_core) + target_link_libraries(${COMPONENT_LIB} ${OPENCV_DEP}) + + +- Finally, include the OpenCV headers needed into your source files: + + .. code:: c++ + + #include "opencv2/core.hpp" + #include "opencv2/imgproc.hpp" + #include "opencv2/imgcodecs.hpp" + + + -.. _detailed_build_procedure.md: esp32/doc/detailed_build_procedure.md Get project RAM and Flash usages @@ -167,6 +224,35 @@ At compilation time: - The file ``build/.map`` is also very useful. It indicates the memory mapping of the variables and can be used to find big variables in the application. +- The commands ``idf.py size``, ``idf.py size-files`` and ``idf.py size-components`` are very useful to see the memory segments usage. They show more precise information, and also per file usage. For instance with the `esp32/examples/esp_opencv_basic/`_ project, the size used is : + + .. _`esp32/examples/esp_opencv_basic/`: esp32/examples/esp_opencv_basic/ + + .. code:: shell + + Total sizes: + DRAM .data size: 21168 bytes + DRAM .bss size: 13280 bytes + Used static DRAM: 34448 bytes ( 146288 available, 19.1% used) + Used static IRAM: 61849 bytes ( 69223 available, 47.2% used) + Flash code: 843403 bytes + Flash rodata: 246200 bytes + Total image size:~1185900 bytes (.bin may be padded larger) + + And for the `esp32/examples/esp_opencv_tests/`_ project, the size used is: + + .. _`esp32/examples/esp_opencv_tests/`: esp32/examples/esp_opencv_tests/ + + .. code:: shell + + Total sizes: + DRAM .data size: 31812 bytes + DRAM .bss size: 14096 bytes + Used static DRAM: 45908 bytes ( 134828 available, 25.4% used) + Used static IRAM: 63741 bytes ( 67331 available, 48.6% used) + Flash code: 1373491 bytes + Flash rodata: 347440 bytes + Total image size:~1830580 bytes (.bin may be padded larger) At run time: ------------ @@ -179,76 +265,35 @@ At run time: ESP_LOGI(TAG, "heap left: %d Bytes", esp_get_free_heap_size()); -Adding images codecs support -============================ +DRAM region overflow +==================== -Things done to read/writes images in JPEG, PNG, etc.. +Depending on which part of the OpenCV library is used, some big static variables can be present and the static DRAM can be overflowed. The following errors can appear: -PNG ---- +- dram overflow + .. code:: shell -- Remove ``-DWITH_PNG=OFF`` and add ``-DBUILD_PNG=ON`` and ``-DBUILD_ZLIB=ON`` of the cmake command + .dram0.bss will not fit in region dram0_0_seg ; region 'dram0_0_seg' overflowed by N bytes - - The lib ``opencv_imgcodecs.a`` build pass - -The library is compiled in the ``3rdparty/`` folder. Copy this folder into the esp32 example project folder. - - - -JPEG ----- - -- Remove ``-DWITH_JPEG=OFF`` and add ``-DBUILD_JPEG=ON`` of the cmake command - - - Problem at compilation time. TODO - - - -Adding parallel support -======================= - -TODO - - - -Removing OpenCV unnecessary parts -================================= - -Opencv is quite big, even when compiling only the core, imgproc and imgcodec modules. Because the ESP32 has limited resources, it is a good idea to remove some parts of opencv that are in most cases not used. - - - -TODO: List the modules functionalities and what is kept or not - -Core module: ------------- - - - - -Imgproc module: ---------------- - -Colorspaces -^^^^^^^^^^^ - -Opencv supports multiple colorspaces (RGB, BGR, RGB565, RGBA, CIELAB, CIEXYZ, Luv, YUV, HSV, HLS, YCrCb, Bayer, Gray). All these colorspaces are not mandatory for an embedded system, so some are removed. - -- ``color_lab.cpp``: This file contains conversion for CIELAB and CIEXYZ (https://en.wikipedia.org/wiki/CIELAB_color_space). The conversion tables takes a lot of space in the .bss segment (~88kB) , which is already overflowing. Here are the steps done to disable this code: - - - Move ``color_lab.cpp`` to ``color_lab.cpp.bak`` - - - In ``color.hpp`` disable : - - .. code:: c++ - - // todo +- SHA256 digest overwrite + .. code::shell + + A fatal error occurred: Contents of segment at SHA256 digest offset 0xb0 are not all zero. Refusing to overwrite. + +The DRAM is the internal RAM section containing data. From the linker script ``esp-idf/components/esp32/ld/esp32.ld``, the ``dram_0_0_seg`` region has a size of ``0x2c200``, which corresponds to around ``180kB``. Due to some fixed RAM addresses used by the ESP32 ROM, there is a limit on the amount which can be statically allocated at compile time (see https://esp32.com/viewtopic.php?t=6699). To prevent this, there are some solutions: + +- If not used, disable Bluetooth and Trace Memory features from the menuconfig. Bluetooth stack uses 64kB and Trace Memory 16kB or 32kB (see https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/general-notes.html#dram-data-ram) + +- In the menuconfig, the following options can also reduce internal DRAM usage: + + - In Component Config -> ESP32-specific -> Support for external, SPI-connected RAM -> SPI RAM config, enable : + - "Try to allocate memories of WiFi and LWIP in SPIRAM firstly. If failed, allocate internal memory" + - "Allow .bss segment placed in external memory" + +- Search for big static array that could be stored in external RAM + - In ``build/.map` file is useful (looking for big variables under the `.dram0.bss` section). * `imgproc/color_lab.cpp` has variables (trilinearLUTE, InvGammaTab and LabCbrtTab) taking ~88 kB. They are used for colorspace conversions in and from CIE LAB and XYZ images. These functions were removed. + + * TODO: There are maybe other ways to save space in this segment: + + * Allow .bss segment placed in external memory + * This option of the menuconfig uses external RAM to store zero-initialized data from the lwIP, net80211, libpp and bluedroid libraries + * Additional data can be moved from the internal BSS segment to external RAM by applying the macro `EXT_RAM_ATTR` to any static declaration (which is not initialized to a non-zero value). + * TODO: See what needs to be included and if it's possible to use this esp macro in OpenCV diff --git a/esp32/examples/esp_opencv_basic/main/CMakeLists.txt b/esp32/examples/esp_opencv_basic/main/CMakeLists.txt index 8beef1f..f2ce742 100644 --- a/esp32/examples/esp_opencv_basic/main/CMakeLists.txt +++ b/esp32/examples/esp_opencv_basic/main/CMakeLists.txt @@ -1,8 +1,12 @@ -set(COMPONENT_SRCS "hello_opencv.cpp") -set(COMPONENT_ADD_INCLUDEDIRS "." "./opencv/") -register_component() - -# Be aware that the order of the librairies is important +idf_component_register( + SRCS + hello_opencv.cpp + + INCLUDE_DIRS + . + opencv/ +) + add_prebuilt_library(opencv_imgcodecs "opencv/libopencv_imgcodecs.a") add_prebuilt_library(libpng "opencv/3rdparty/liblibpng.a") add_prebuilt_library(libzlib "opencv/3rdparty/libzlib.a") @@ -10,9 +14,11 @@ add_prebuilt_library(opencv_imgproc "opencv/libopencv_imgproc.a") add_prebuilt_library(opencv_core "opencv/libopencv_core.a") add_prebuilt_library(ade "opencv/libade.a") -target_link_libraries(${COMPONENT_LIB} PRIVATE opencv_imgcodecs) -target_link_libraries(${COMPONENT_LIB} PRIVATE libpng) -target_link_libraries(${COMPONENT_LIB} PRIVATE libzlib) -target_link_libraries(${COMPONENT_LIB} PRIVATE opencv_imgproc) -target_link_libraries(${COMPONENT_LIB} PRIVATE opencv_core) -target_link_libraries(${COMPONENT_LIB} PRIVATE ade) +set(IMGCODEC_DEP libpng libzlib) +target_link_libraries(opencv_imgcodecs INTERFACE ${IMGCODEC_DEP}) + +set(CORE_DEP libzlib) +target_link_libraries(opencv_core INTERFACE ${CORE_DEP}) + +set(OPENCV_DEP opencv_imgcodecs opencv_imgproc opencv_core) +target_link_libraries(${COMPONENT_LIB} ${OPENCV_DEP}) diff --git a/esp32/examples/esp_opencv_tests/main/CMakeLists.txt b/esp32/examples/esp_opencv_tests/main/CMakeLists.txt index 7c5c82b..08f5fd9 100644 --- a/esp32/examples/esp_opencv_tests/main/CMakeLists.txt +++ b/esp32/examples/esp_opencv_tests/main/CMakeLists.txt @@ -16,12 +16,14 @@ add_prebuilt_library(opencv_imgproc "opencv/libopencv_imgproc.a") add_prebuilt_library(opencv_core "opencv/libopencv_core.a") add_prebuilt_library(ade "opencv/libade.a") -target_link_libraries(${COMPONENT_LIB} PRIVATE opencv_imgcodecs) -target_link_libraries(${COMPONENT_LIB} PRIVATE libpng) -target_link_libraries(${COMPONENT_LIB} PRIVATE libzlib) -target_link_libraries(${COMPONENT_LIB} PRIVATE opencv_imgproc) -target_link_libraries(${COMPONENT_LIB} PRIVATE opencv_core) -target_link_libraries(${COMPONENT_LIB} PRIVATE ade) +set(IMGCODEC_DEP libpng libzlib) +target_link_libraries(opencv_imgcodecs INTERFACE ${IMGCODEC_DEP}) + +set(CORE_DEP libzlib) +target_link_libraries(opencv_core INTERFACE ${CORE_DEP}) + +set(OPENCV_DEP opencv_imgcodecs opencv_imgproc opencv_core) +target_link_libraries(${COMPONENT_LIB} ${OPENCV_DEP}) # create spiffs partition (named 'storage') from the ../spiffs_image directory spiffs_create_partition_image(storage ../spiffs_images FLASH_IN_PROJECT) diff --git a/esp32/examples/esp_opencv_tests/sdkconfig b/esp32/examples/esp_opencv_tests/sdkconfig index b8f7d5b..23aa9c9 100644 --- a/esp32/examples/esp_opencv_tests/sdkconfig +++ b/esp32/examples/esp_opencv_tests/sdkconfig @@ -47,6 +47,7 @@ CONFIG_BOOTLOADER_LOG_LEVEL_INFO=y # CONFIG_BOOTLOADER_LOG_LEVEL_DEBUG is not set # CONFIG_BOOTLOADER_LOG_LEVEL_VERBOSE is not set CONFIG_BOOTLOADER_LOG_LEVEL=3 +CONFIG_BOOTLOADER_SPI_WP_PIN=7 CONFIG_BOOTLOADER_VDDSDIO_BOOST_1_9V=y # CONFIG_BOOTLOADER_FACTORY_RESET is not set # CONFIG_BOOTLOADER_APP_TEST is not set @@ -71,9 +72,9 @@ CONFIG_BOOTLOADER_RESERVE_RTC_SIZE=0 # Serial flasher config # CONFIG_ESPTOOLPY_BAUD_OTHER_VAL=115200 -# CONFIG_ESPTOOLPY_FLASHMODE_QIO is not set +CONFIG_ESPTOOLPY_FLASHMODE_QIO=y # CONFIG_ESPTOOLPY_FLASHMODE_QOUT is not set -CONFIG_ESPTOOLPY_FLASHMODE_DIO=y +# CONFIG_ESPTOOLPY_FLASHMODE_DIO is not set # CONFIG_ESPTOOLPY_FLASHMODE_DOUT is not set CONFIG_ESPTOOLPY_FLASHMODE="dio" CONFIG_ESPTOOLPY_FLASHFREQ_80M=y @@ -187,7 +188,7 @@ CONFIG_ADC_DISABLE_DAC=y # # SPI configuration # -# CONFIG_SPI_MASTER_IN_IRAM is not set +CONFIG_SPI_MASTER_IN_IRAM=y CONFIG_SPI_MASTER_ISR_IN_IRAM=y # CONFIG_SPI_SLAVE_IN_IRAM is not set CONFIG_SPI_SLAVE_ISR_IN_IRAM=y @@ -251,15 +252,14 @@ CONFIG_SPIRAM_SIZE=-1 CONFIG_SPIRAM_SPEED_80M=y CONFIG_SPIRAM=y CONFIG_SPIRAM_BOOT_INIT=y -# CONFIG_SPIRAM_IGNORE_NOTFOUND is not set # CONFIG_SPIRAM_USE_MEMMAP is not set # CONFIG_SPIRAM_USE_CAPS_ALLOC is not set CONFIG_SPIRAM_USE_MALLOC=y CONFIG_SPIRAM_MEMTEST=y CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=16384 -# CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP is not set +CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=32768 -# CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY is not set +CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY=y CONFIG_SPIRAM_CACHE_WORKAROUND=y CONFIG_SPIRAM_BANKSWITCH_ENABLE=y CONFIG_SPIRAM_BANKSWITCH_RESERVE=8 @@ -287,8 +287,6 @@ CONFIG_D2WD_PSRAM_CS_IO=10 # CONFIG_PICO_PSRAM_CS_IO=10 # end of PSRAM clock and cs IO for ESP32-PICO - -CONFIG_SPIRAM_SPIWP_SD3_PIN=7 # end of SPI RAM config # CONFIG_ESP32_TRAX is not set @@ -467,7 +465,7 @@ CONFIG_ESP32_WIFI_STATIC_TX_BUFFER_NUM=16 CONFIG_ESP32_WIFI_AMPDU_TX_ENABLED=y CONFIG_ESP32_WIFI_TX_BA_WIN=6 CONFIG_ESP32_WIFI_AMPDU_RX_ENABLED=y -CONFIG_ESP32_WIFI_RX_BA_WIN=6 +CONFIG_ESP32_WIFI_RX_BA_WIN=16 CONFIG_ESP32_WIFI_NVS_ENABLED=y CONFIG_ESP32_WIFI_TASK_PINNED_TO_CORE_0=y # CONFIG_ESP32_WIFI_TASK_PINNED_TO_CORE_1 is not set @@ -685,6 +683,7 @@ CONFIG_LWIP_TCP_QUEUE_OOSEQ=y CONFIG_LWIP_TCP_OVERSIZE_MSS=y # CONFIG_LWIP_TCP_OVERSIZE_QUARTER_MSS is not set # CONFIG_LWIP_TCP_OVERSIZE_DISABLE is not set +# CONFIG_LWIP_WND_SCALE is not set # end of TCP # @@ -1025,9 +1024,9 @@ CONFIG_LOG_BOOTLOADER_LEVEL_INFO=y CONFIG_LOG_BOOTLOADER_LEVEL=3 # CONFIG_APP_ROLLBACK_ENABLE is not set # CONFIG_FLASH_ENCRYPTION_ENABLED is not set -# CONFIG_FLASHMODE_QIO is not set +CONFIG_FLASHMODE_QIO=y # CONFIG_FLASHMODE_QOUT is not set -CONFIG_FLASHMODE_DIO=y +# CONFIG_FLASHMODE_DIO is not set # CONFIG_FLASHMODE_DOUT is not set # CONFIG_MONITOR_BAUD_9600B is not set # CONFIG_MONITOR_BAUD_57600B is not set @@ -1061,7 +1060,7 @@ CONFIG_BTDM_CONTROLLER_BR_EDR_MAX_SYNC_CONN_EFF=0 CONFIG_BTDM_CONTROLLER_PINNED_TO_CORE=0 CONFIG_ADC2_DISABLE_DAC=y CONFIG_SPIRAM_SUPPORT=y -# CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST is not set +CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST=y CONFIG_TRACEMEM_RESERVE_DRAM=0x0 # CONFIG_TWO_UNIVERSAL_MAC_ADDRESS is not set CONFIG_FOUR_UNIVERSAL_MAC_ADDRESS=y diff --git a/esp32/examples/esp_opencv_tests/sdkconfig.old b/esp32/examples/esp_opencv_tests/sdkconfig.old index 4b2488b..ed3d830 100644 --- a/esp32/examples/esp_opencv_tests/sdkconfig.old +++ b/esp32/examples/esp_opencv_tests/sdkconfig.old @@ -84,9 +84,9 @@ CONFIG_ESPTOOLPY_FLASHFREQ="80m" # CONFIG_ESPTOOLPY_FLASHSIZE_1MB is not set # CONFIG_ESPTOOLPY_FLASHSIZE_2MB is not set # CONFIG_ESPTOOLPY_FLASHSIZE_4MB is not set -CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y -# CONFIG_ESPTOOLPY_FLASHSIZE_16MB is not set -CONFIG_ESPTOOLPY_FLASHSIZE="8MB" +# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set +CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y +CONFIG_ESPTOOLPY_FLASHSIZE="16MB" CONFIG_ESPTOOLPY_FLASHSIZE_DETECT=y CONFIG_ESPTOOLPY_BEFORE_RESET=y # CONFIG_ESPTOOLPY_BEFORE_NORESET is not set