A vision-based robust grape berry counting algorithm for fast calibration-free bunch weight estimation in the field

https://doi.org/10.1016/j.compag.2020.105360Get rights and content

Highlights

  • A fast calibration-free algorithm for counting berries on grape bunches in vivo.

  • Robust to development stage from pea-sized to harvest and across varieties.

  • Average accuracy of 99% for berry number and 92% for bunch weight.

  • Applicable to users in the field for speeding up yield estimation processes.

  • Datasets available for comparison from http://www.robotics.unsw.edu.au/srv/datasets.html.

Abstract

Counting the number of berries per bunch is a key component of many yield estimation processes but is exceptionally tedious for farmers to complete. Recent work into image processing in viticulture has produced methods for berry counting, however these require some degree of manual intervention or need calibration to manual counts for different bunch architectures.

Therefore, this paper introduces a fast and robust calibration-free algorithm for berry counting for winegrapes to aid yield estimation. The algorithm was tested on 529 images collected in the field at multiple vineyards at different maturity stages and achieved an accuracy of approximately 89% per bunch. As it would mostly likely be used to obtain an average value across a block, the low bias of this method resulted in an average accuracy of 99% and was shown to be robust from pea-sized to harvest and between both red and green bunches.

Taking only 0.1 to 1 s per image to process and requiring only a smartphone and small backing board to capture, the algorithm can readily be applied to images which are captured in the field by farmers. This allowed bunch weights to be estimated to within 92% accuracy and assisted larger scale yield estimation processes to achieve accuracies of between 3% and 16%. The robustness of the method lays the foundation for fast fruit-set ratio determination and more detailed bunch architecture studies in vivo on a large scale.

Introduction

Automating yield component analysis is vital for improving yield estimation in viticulture since the current manual approaches cannot meet the requirements of fast measurement and large sampling scale to secure the accuracy of grape production forecasting. The small sample size and lack of objectivity in interpreting the state of vine development also leads to poor accuracy in yield estimation in the wine industry. State-of-the-art manual sampling immediately prior to harvest results in errors from 3 to 30% (Whitty et al., 2017) (Table 6.9), which anecdotally matches industry experience. Subsequently, wineries are forced to bear the cost of suboptimal tank space allocation, oak barrel purchases and contract adjustments as well as undertake the challenging task of managing harvest logistics within a decreasing harvest window. Hence, researchers in viticulture have been seeking solutions from image processing and computer vision to accelerate crop yield forecasting. Nuske et al. (2014) presented image processing methods which were able to generate unbiased estimates notably smaller than manual estimates.

Image processing has also been applied for grape bunch phenotyping in the context of breeding programs, and such phenotyping includes three quantitative methods of analysis (Schöler and Steinhage, 2015); the number of components, overall morphology of components and overall length of components, with components being berries, rachis internodes and other internodes etc. The number of berries remains stable after fruit set and it has vital impact on final yield for a bunch (Martin et al., 2003). Besides that, the ratio between bunch size and the number of berries per bunch is one of many factors governing the quality of the fruit at harvest. Given the number of berries per bunch and bunch weight are critical parameters for early forecasts of production we mainly focus on the yield component of berry number and its contribution to bunch yield estimation in this paper.

Currently, berry counting and bunch weighing across the grape growing season are accomplished by tedious and labour intensive manual measurements. To expedite this, two main approaches have been used: image based (2D, RGB images) berry counting and 3D point cloud sensor-based berry counting (by LiDAR or RGB-D camera).

Under the category of 3D point cloud sensor-based methods, initial work was conducted by Schöler and Steinhage (2015) who presented a fully-automated sensor-based 3D reconstruction approach to phenotyping grape bunches. The proposed approach is able to generate a comprehensive bunch structure based on the 3D point cloud by iteratively optimizing parameters which define the bunch structure. As to berry number estimation, their approach was shown to achieve 12.35% error (see Table 1 in that work (Schöler and Steinhage, 2015)). Later on, the same research group extended this work by developing software called “3D-Bunch-Tool” based on new lightweight 3D scanner (Rist et al., 2018) which can be utilized in the field. That software achieved 78.83% accuracy with R2=0.95 ((see Table 2 in that work (Rist et al., 2018)) on lab-based berry counting and the process of scanning in the lab took approximately 1 min. Field based scanning, observing only one side of a bunch meant approximately 50% of berries were observed, and these were correlated to the total number of berries from a 360° scan with an R2 value of 0.83; the actual error in terms of berry count was not presented in the paper.

The intricacy of the 3D scanning approach and cost of the sensors has meant considerable focus has been given to the imagery based solution in the field. Liu et al., 2013, Diago et al., 2015 and Ivorra et al. (2015) showed how yield component analysis could lead to more efficient forecasts using image processing. Kicherer et al. (2013) presented the Berry Analysis Tool (BAT) for counting berry number, diameter and volume, which is reliant on destemming a bunch and arranging berries on a perforated metal plate in laboratory conditions. Grossetete et al. (2012) and later Diago et al. (2015) processed RGB images to count berries using a photo of one side of a bunch, obtaining R2 values of 0.92 and 0.82 respectively between the real and detected number of berries. The work presented by Diago et al. (2015) was tested with a dataset of ten images for each of seven varieties, with R2 values varying from 0.62 to 0.95 with an average of 0.82 across the seven cultivars. Aquino et al. (2017) developed an algorithm to detect visible berries from a single bunch photo in the field, achieving F1 score = 0.89 based on their best parameter settings. As for actual berry estimation, that paper showed results of R2=0.75 and an accuracy of 84.35% between estimated berries and actual berries. This work was extended into an app (Aquino et al., 2018) which was not available online at the time of writing. However, the image processing algorithms proposed by Grossetete et al. (2012) and Aquino et al. (2017) rely on a specular reflection at a single point on each berry and are not robust following veraison since the surface of the berries may become matte and in some cases shrivelled.

Besides estimating the number of berries by processing single 2D images, 3D bunch reconstruction has also been achieved using stereo imagery (Herrero-Huerta et al., 2015). There, two 3D bunch models were built with substantial manual input and the method achieved R2=0.797 using a point model and R2=0.778 against a CAD model. This customized stereo camera arrangement also has a natural minimum range and limits applicability to ex-vivo1 analysis and maneuvering such a setup within a sprawling canopy is impractical.

Commercial mobile solutions have became the main objective which is challenging the robustness of existing image processing algorithms (Aquino et al., 2017). A common approach relies on a backing board with contrasting color to the bunch. As for actual berry counting, the most common approach is to estimate the occluded berries based on the detected number of visible berries (Aquino et al., 2017). This needs calibration and varies between cultivars and lighting conditions since most existing algorithms are sensitive to illumination changes. Cultivar dependency of this calibration has not been investigated to date given the tedious data collection procedure.

In order to relieve the burden of building calibrations for each cultivar, Liu et al. (2015) proposed a novel approach to count berries from a single image. Their method is limited to red grapes and can only deal with conical or cylindrically shaped bunches because the reconstruction procedure only follows the main branch of the bunch. A range of berry radii needs to be manual defined using their approach. The tested images were collected under laboratory conditions.

Hence with the assistance of a backing board and under the condition that the bunch can be segmented out from a high contrast backing board by mature bunch segmentation solutions (Luo et al., 2016a, Luo et al., 2016b, Perez-Zavala et al., 2018) this paper focuses in particular on the robust berry counting solutions for both red and green bunches and its contribution to bunch weight estimation.

For direct comparison with other approaches, we provide the benchmark of collected bunch images and related metrics. The datasets are published2 on the Smart Robotic Viticulture group’s website3.

In the remainder of this paper, Section 2 presents a field-robust algorithm for in-field berry counting based on a single RGB image, catering for red and green bunches with a range of berry diameters. Section 3 describes the experimental data and procedures used to validate the algorithm. Section 4 shows the accuracy of the results, as well as evaluating the robustness of the algorithm to differences in development stage, the contribution of berry counting to bunch weight estimation, and the possibility of yield estimation by image-based berry counting. Section 5 then draws conclusions and makes recommendations for future work.

Section snippets

Methodology

In general, berry counting is divided into three steps, Region of Interest (ROI) extraction, visible berry detection and actual berry count estimation, which are demonstrated in Fig. 1. We propose a novel algorithm for 3D bunch reconstruction based on a single image for fast berry counting in vineyards. According to the flowchart in Fig. 1, the proposed approach starts with sub-bunch detection then processes each sub-bunch 4

Data scope and experimental design

In total, 529 bunch images from two cultivars were tested and the details of each dataset are illustrated in Table 1 including whether in vivo or ex-vivo1 and which model of smartphone was used. All bunches were photographed in the field (whether in vivo or ex-vivo) without artificial lighting, replicating end-user usage, the only requirement being the use of a backing board to aid segmentation. The proposed 3D reconstruction algorithm and sparsity factor calculation was implemented in Matlab

Experimental results and discussion

Using the computer described above, each image took 0.1 s to be processed, without any code optimisation. Qualitatively, manual observations of the real and reconstructed bunches matched quite well, as the proposed method fits berries around the outer profile of the bunch, similar to common bunch architectures. Fig. 5 demonstrates some 3D bunch models reconstructed by the proposed method. Note the variation in the illumination conditions and bunch structure that exists among the original photos

Conclusions

This paper has presented a novel and fast algorithm which is able to count berries and estimate the 3D structure of both red and green grapes in-field from pea-sized to harvest development stages from a range of bunch architectures. Using only a single image from a smartphone and no calibration or prior information, the accuracy of the method was 89% when directly compared with the number of berries on a bunch. When averaged across 50 to 80 images, the accuracy was over 99%, showing the limited

CRediT authorship contribution statement

Scarlett Liu: Conceptualization, Methodology, Software, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization. Xiangdong Zeng: Software, Validation. Mark Whitty: Conceptualization, Writing - original draft, Writing - review & editing, Supervision, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The datasets used in this paper were obtained with the support of Wine Australia project DPI1401 Improved Yield Estimation for the Australian Wine Industry. All these datasets are freely available for download and comparison from the Smart Robotic Viticulture group’s website: http://www.robotics.unsw.edu.au/srv/datasets.html.

References (28)

  • F. Schöler et al.

    Automated 3d reconstruction of grape cluster architecture from sensor data for efficient phenotyping

    Comput. Electron. Agric.

    (2015)
  • A. Aquino et al.

    vitisFlower: Development and testing of a novel android-smartphone application for assessing the number of grapevine flowers per inflorescence using artificial vision techniques

    Sensors

    (2015)
  • K. Dahal et al.

    Assessment of hen and chickendisorder for marketable yield estimates of table grape using the berry analysis tool

    Vitis: J. Grapevine Res.

    (2018)
  • M.P. Diago et al.

    Assessment of cluster yield components by image analysis

    J. Sci. Food Agric.

    (2015)
  • Cited by (32)

    • 3D grape bunch model reconstruction from 2D images

      2023, Computers and Electronics in Agriculture
    • Automatic shrimp counting method using local images and lightweight YOLOv4

      2022, Biosystems Engineering
      Citation Excerpt :

      Counting refers to estimating the number of objects in a target area to obtain information and conduct timely control operations and has been widely used in many different fields (Liu et al., 2020; Vasconez et al., 2020; Xu et al., 2020a, 2020b).

    View all citing articles on Scopus
    View full text