diff --git a/docs/integrations/cameras/luxonis-oak-1.mdx b/docs/integrations/cameras/luxonis-oak-1.mdx new file mode 100644 index 00000000..3452e993 --- /dev/null +++ b/docs/integrations/cameras/luxonis-oak-1.mdx @@ -0,0 +1,380 @@ +--- +title: Luxonis OAK-1 Camera Integration Guide for ROS Robots +sidebar_label: Luxonis OAK-1 +keywords: + - luxonis + - oak-1 + - rgb + - camera + - integration + - autonomy + - leo + - example +description: >- + Learn how to connect a Luxonis OAK-1 camera to your Leo Rover and integrate it + with the system using ROS. +# TODO: add an OAK-1 image (render?) +image: /img/robots/leo/integrations/oak-d/oak-d-ucm.webp +--- + +import ImageZoom from '@site/src/components/ImageZoom'; +import LinkButton from '@site/src/components/LinkButton'; +import Product from '@site/src/products/luxonis-oak-1-w-imx378.mdx'; + +# Luxonis OAK-1 Camera Integration + +In this tutorial we will go through the process of connecting a Luxonis OAK-1 +camera to your Leo Rover and integrating it with the system using ROS. + +OAK-1 is a compact AI camera for robotic vision that combines a high-resolution +12MP color sensor with on-device Neural Network inferencing and Computer Vision +capabilities provided by the built-in Myriad X VPU. Thanks to that, tasks like +object detection or tracking can run directly on the camera, without putting any +load on the rover's computer. The camera uses USB-C for both power and USB3 +connectivity. In robotics, cameras like this are commonly used for object +recognition, visual tracking and many other computer vision applications. + +{/* TODO: add an OAK-1 image (or robot with OAK-1) */} + +## What to expect? + +After completing this tutorial, you will have a Luxonis OAK-1 camera mounted on +your Leo Rover, and you will be able to access the camera's data through ROS +topics, such as: + +- `/oak/rgb/image_raw` - the RGB image stream, +- `/oak/rgb/image_raw/compressed` - the compressed RGB image stream, +- `/oak/rgb/camera_info` - the camera calibration data. + +You will also learn how to use the camera's on-device AI capabilities by running +an object detection example. The image below shows its final result - the camera +stream with the detected objects marked with bounding boxes, visualized in RViz: + + + +## Prerequisites + + + + + + + +### Referenced products + + + +## Hardware integration + +{/* TODO: add hardware integration */} + +## Software integration + +The first thing you can do is to make sure your device has the correct +permissions. To do this, you can add the following rule to the udev service: + +```xml title="/etc/udev/rules.d/luxonis.rules" +SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666" +``` + +Paste this line to `/etc/udev/rules.d/luxonis.rules` file and reload udev rules +by typing: + +```bash +sudo udevadm control --reload-rules && sudo udevadm trigger +``` + +We want the sensor functionality to be available in the ROS ecosystem, so you +should install a ROS package that provides a node for the camera. + +```bash +sudo apt install ros-${ROS_DISTRO}-depthai-ros +``` + +### Modifying the URDF model + +As we want to have the camera model visible with the rover model, we need to +create an URDF file with the OAK included. Package `depthai_descriptions` is +installed alongside `depthai-ros` and it contains OAK camera models and xacro +macros that we can easily add (list of available models +[here](https://github.com/luxonis/depthai-ros/tree/jazzy/depthai_descriptions/urdf/models)). +We need to include the correct macro to make the camera visible with the robot's +model. To do that create a new file in `/etc/ros/urdf` directory, and name it +`oak-1.urdf.xacro`. In this file, include the following content: + +```xml title="/etc/ros/urdf/oak-1.urdf.xacro" + + + + + + + +``` + +The `camera_name` property determines the names of the camera's TF frames and +topics. It has to be set to the same value as the `name` of the camera node that +we will launch later in this tutorial. The `cam_<>` properties specify the +camera's position and orientation in reference to the link set in the `parent` +property. As the `parent` property is set to "base_link", the position of the +camera is provided in reference to the origin of the Rover. + +:::warning + +In the file above, the `cam_<>` properties are set to the default values. You +will have to adjust them to your mounting solution. + +::: + +Now we have to include our camera model in `robot.urdf.xacro` file - the +description that is uploaded at boot. Add this line somewhere before the closing +`` tag. + +```xml title="/etc/ros/urdf/robot.urdf.xacro" + +``` + +### Launching camera nodes + +To get the image from camera, you need to launch the camera nodes. You can do it +by using default DepthAI launcher: + +```bash +ros2 launch depthai_ros_driver camera.launch.py +``` + +If you want to change the default parameters, you can do it while the node runs +with the usage of _RQt_: + +```bash +rqt +``` + +If you want the changes of parameters to persist across the runs, create a +`.yaml` configuration file in `/etc/ros`. This example contains configuration +for OAK-1 camera: + +```yaml title="/etc/ros/oak.yaml" +/oak: + ros__parameters: + oak.rgb.image_raw.enable_pub_plugins: + ['image_transport/raw', 'image_transport/compressed'] + camera: + i_pipeline_type: RGB + i_nn_type: none + rgb: + i_publish_topic: true + i_fps: 30.0 + i_resolution: 1080P + i_isp_num: 2 + i_isp_den: 3 + i_output_isp: true + i_width: 1280 + i_height: 720 +``` + +You can modify the parameters in the file according to your needs and then +launch: + +```bash +ros2 launch depthai_ros_driver camera.launch.py params_file:=/etc/ros/oak.yaml +``` + +:::note + +If your OAK-1 unit doesn't have an IMU, you will see this warning: + +``` +[WARN] [oak]: IMU enabled but not available! +``` + +It is harmless - the driver simply skips creating the IMU node and the camera +works normally. You can silence it by adding the `pipeline_gen` section to the +configuration file: + +```yaml +/oak: + ros__parameters: + pipeline_gen: + i_enable_imu: false +``` + +::: + +:::info + +For more information about oak configuration and usage examples you can check +out the +[depthai-ros github repository](https://github.com/luxonis/depthai-ros/tree/jazzy). + +::: + +### (Optional) Launching camera nodes on system startup + +To launch camera nodes with the created parameters file on system startup, +create a launch file in `/etc/ros`: + +```xml title="/etc/ros/oak.launch.xml" + + + + + + +``` + +:::info + +You can set the `namespace` property of the `node` tag to whatever you want, but +the `name` has to be set to the same value as the `camera_name` in urdf file. +It's required for the camera data to be placed in correct TF frame. + +::: + +Then in `/etc/ros/robot.launch.xml` add this line somewhere before the closing +`` tag: + +```xml title="/etc/ros/robot.launch.xml" + +``` + +Now your OAK-1 camera ROS node will start at launch. You can also start them now +by typing + +```bash +ros-nodes-restart +``` + +## Example usage + +One of the capabilities of Luxonis OAK-1 Camera is running object detection +directly on the camera. To test it, we will run one of the `depthai_filters` +examples. + +:::note + +If you enabled launching the camera nodes on system startup, disable them now, +as the example starts its own camera node and two nodes can't connect to the +same device. To do so, remove the `` +line from `/etc/ros/robot.launch.xml` and restart ROS2 nodes: + +```bash +ros-nodes-restart +``` + +::: + +Firstly, we need to modify the OAK parameters file to enable the neural network +on the camera and pass through the image the network runs on: + +```yaml title="/etc/ros/oak.yaml" +/oak: + ros__parameters: + oak.rgb.image_raw.enable_pub_plugins: + ['image_transport/raw', 'image_transport/compressed'] + camera: + i_pipeline_type: RGB + i_nn_type: rgb + rgb: + i_publish_topic: true + i_fps: 30.0 + i_resolution: 1080P + i_isp_num: 2 + i_isp_den: 3 + i_output_isp: true + i_width: 1280 + i_height: 720 + nn: + i_enable_passthrough: true + i_disable_resize: false +``` + +Then run the example, passing the modified parameters file: + +```bash +ros2 launch depthai_filters example_det2d_overlay.launch.py params_file:=/etc/ros/oak.yaml +``` + +Now you can switch to the PC connected to rover's access point and run RViz: + +```bash +rviz2 +``` + +In the RViz window add the visualization by selecting Image from `/overlay` +topic and clicking OK: + + + +After doing so, you should be able to see the image from the camera with the +bounding boxes and names of the detected objects, along with the confidence of +each detection shown as a percentage: + + + +The example uses the default detection network - a MobileNet-SSD model trained +on the PASCAL VOC dataset. It can detect 20 classes of objects, such as person, +car, bicycle, dog or bottle. You can find the full list of the detected classes +in the +[label map](https://github.com/luxonis/depthai-ros/blob/jazzy/depthai_filters/include/depthai_filters/detection2d_overlay.hpp) +of the overlay node. To learn more about this and other examples from the +`depthai_filters` package, check out its +[documentation](https://github.com/luxonis/depthai-ros/tree/jazzy/depthai_filters). + +## What's next? + +With the camera image available on ROS topics, you can use it in projects +involving computer vision. + +The OAK-1 can run neural network inference directly on the camera. To learn more +about the camera's capabilities, check out the +[Luxonis documentation](https://docs.luxonis.com/hardware/products/OAK-1%20W), +and for more advanced configuration of the ROS driver, see the +[DepthAI ROS documentation](https://docs.luxonis.com/software/ros/depthai-ros/). diff --git a/src/products/luxonis-oak-1-w-imx378.mdx b/src/products/luxonis-oak-1-w-imx378.mdx new file mode 100644 index 00000000..4104b1a0 --- /dev/null +++ b/src/products/luxonis-oak-1-w-imx378.mdx @@ -0,0 +1,11 @@ +import ProductPreview from '@site/src/components/ProductPreview'; + + diff --git a/static/img/robots/leo/integrations/oak-1/luxonis-oak-1-w-imx378-leo-integration-1.webp b/static/img/robots/leo/integrations/oak-1/luxonis-oak-1-w-imx378-leo-integration-1.webp new file mode 100644 index 00000000..bab429fd Binary files /dev/null and b/static/img/robots/leo/integrations/oak-1/luxonis-oak-1-w-imx378-leo-integration-1.webp differ diff --git a/static/img/robots/leo/integrations/oak-1/oak-1-detection-example-chair-monitor.webp b/static/img/robots/leo/integrations/oak-1/oak-1-detection-example-chair-monitor.webp new file mode 100644 index 00000000..04bd7f53 Binary files /dev/null and b/static/img/robots/leo/integrations/oak-1/oak-1-detection-example-chair-monitor.webp differ diff --git a/static/img/robots/leo/integrations/oak-1/rviz-oak-1-overlay-topic-selection.webp b/static/img/robots/leo/integrations/oak-1/rviz-oak-1-overlay-topic-selection.webp new file mode 100644 index 00000000..4f3b4f11 Binary files /dev/null and b/static/img/robots/leo/integrations/oak-1/rviz-oak-1-overlay-topic-selection.webp differ