About the project
GenoCLIP explores whether seeing a tree’s 3D structure during training can help a model learn from its genetics. It pairs an encoder for genetic markers with one for LiDAR scans. After training, the genetics encoder predicts branch count and mean branch angle without needing a scan.
How it works
- 1Pairs genetic markers and 3D scans from the same tree during training.
- 2Repairs the reconstructed branch structure before measuring branch counts and angles.
- 3Predicts traits from genetic data alone and compares the results with classical and deep-learning baselines.
At a glance
- →Paired genetic data with LiDAR scans for 122 peach trees, with about 131,000 genetic markers per tree.
- →Reduced mean branch-angle error from 7.4 to 6.8 compared with the genetics-only DPCFormer model; random forest reached 6.7.
- →Reduced branch-count error from 48.0 to 43.9 compared with DPCFormer; random forest had the lowest error at 40.2.
- →Corrected TreeQSM branch connections before extracting the traits used for training.
- →Used 3D scans during training; making predictions only requires genetic data.
Why this project
Measuring tree structure takes time, and our dataset had far more genetic markers than trees: roughly 131,000 markers for each of 122 trees. That makes overfitting a concern. I wanted to test whether pairing genetic data with 3D structure would help the model learn patterns related to tree shape.
The approach
I encoded the genetic markers, called SNPs, as allele counts with a separate value for missing data. For the scans, I used TreeQSM to reconstruct tree structure and corrected the branch connections before measuring traits. GenoCLIP pairs DPCFormer for the genetic data with Point Transformer v3 for the scans. A contrastive loss (InfoNCE) trains the two encoders to produce similar representations for the same tree. A regression head then predicts traits from the genetic representation. I compared this with linear models, random forest, gradient boosting, XGBoost, and DPCFormer trained on genetics alone.
Results and notes
For mean branch angle, GenoCLIP reached a mean absolute error of 6.8 and a Pearson correlation of 0.644, compared with 7.4 and 0.591 for DPCFormer. Random forest did slightly better at 6.7 and 0.68. For branch count, GenoCLIP lowered mean absolute error from 48.0 to 43.9, although its correlation was lower (0.4408 versus 0.4567). Random forest had the lowest branch-count error at 40.2, with a correlation of 0.445. The 3D training data helped reduce error relative to the deep-learning baseline, but random forest still had the lowest error on both traits in this small dataset.
Links