Plant Identification

                    graph LR;

                    %% Define Blocks
                    A[Intel D435i Camera]
                    B[Segmentation]
                    C[Classification]
                    D[Localization]
                    E([RGB Frame])
                    F([Real-World coordinates])
                    G([WeedInfo ROS Topic])
                    H([Segmented Plant Images])
                    I([Depth Frame])
                    J([Plant Species])
                    K{Is it a weed?}
                    L([Pixel Coordinates])

                    %% Image Data Processes
                    A --> E
                    E --> B
                    B --> H
                    B --> L
                    H --> C
                    C --> J
                    J --> K
                    K --> |Yes| G
                    K --> |No| H


                    %% Depth Data Processes
                    A --> I
                    L --> D
                    I --> D
                    D --> F
                    F --> K

Node diagram of the plant identification pipeline

Segmentation

To distinguish crops from weeds, we implemented a plant segmentation pipeline using a color-based approach combined with DBSCAN clustering. Initially, HSV thresholding isolates green pixels in the image. The image is then converted to binary, with green pixels as white and everything else as black. To reduce noise, morphological operations are applied: a 3x3 opening operation and a 10x10 closing operation. Finally, DBSCAN groups the white areas into clusters, with each cluster representing a different plant.

Classification

Once different plants are identified from the segmentation pipeline, each plant is cropped from the image and processed through a pre-trained neural network from Pl@ntNet (Garcin et al., 2021). We selected the ResNet18 model, which has 3000 different plant species it can identify. To minimize false negatives, a list of expected crops is defined. If the model's top five outputs with the highest confidence include a crop from this list, the plant is identified as that crop. Any plant not labeled as a crop is then classified as a weed.

Localization

Once all weeds are identified, their coordinates are sent to the weeder arm for removal. Pixel coordinates of the weed centers, calculated during segmentation, are converted to real-world coordinates using Intel's Depth Camera D435 module with its built-in depth sensor. This achieves ≤ 11 mm accuracy in both x and y directions.

Evaluation

To validate the segmentation pipeline, precision, recall, and mean IOU values of the predicted bounding boxes were calculated and compared to the ground truth. The segmentation pipeline showed an average precision of 0.69, recall of 0.70, and IoU of 0.68. For the classification pipeline, a confusion matrix for lettuce and weeds was calaulated. The result showed an accuracy of 0.88, precision of 0.91, and recall of 0.85.