I've had both the Garmin 800 and the Vesper XB-8000. Here's my 2-cents' worth.
Garmin had a major software glitch in the 800 when it first came out. On our R31, I had the Garmin 800, but it would not work for more than a few minutes at a time. Garmin offered some ideas via phone, later sent a rep to the boat to inspect, tried a software upgrade, and eventually replaced it with a new unit (all of which took several months). The new unit had identical issues. So it went back to Garmin and I got the Vesper XB-8000 instead (both are easy to install, and I installed the Vesper myself). I had the Vesper on my R27 for several years before we got the R31, and I knew I would be happy with it. It has been rock solid.
It sounds like Garmin has fixed the software issues on the 800. So it should be a reliable option. If it were me, though, I'd still get the Vesper. Here are my basic reasons why:
1) Updates/upgrades: Vesper is very dedicated to it's AIS products and frequently improves them. Every improvement is available via software upgrades. XB-8000 owners have not had to purchase new hardware to get the full benefit of everything Vesper does to improve its product. The platform appears to be almost infinitely upgradable via software.
2) WiFi and App: Both units will work equally well with the Garmin chartplotter and talk to the NMEA backbone. Where the Vesper has an edge (and it is a big one for me) is that it creates its own Wifi, which then gives you AIS functionality on a supplemental computer, tablet, or smartphone as well as on your chart plotter. That has permitted me to run the Vesper app on my iPad (and my iPhone), giving me AIS control and visuals on a separate screen. That is handier for me when I am doing all my own navigation (I have to change screens on the Garmin to get details for AIS targets, a list of targets, etc). It also permits me to have a crew member monitor AIS independently (very valuable for nighttime parading during the winter Christmas Ships parade season here on the Columbia and Willamette Rivers, both of which have heavy commercial shipping traffic).
3) Other functionality: Vesper lets me fine-tune alarms, targets, etc to a higher degree than the Garmin unit did. Garmin may have addressed that, or may in the future, through software upgrades. But so far, I think Vesper has more functional and customizable user settings.
One way to assess those and other differences would be to read through the user manuals of both. The info on the Vesper website will give you a good sense of what the XB-8000 permits you to do through the Wifi and app. Then compare to the instruction manual for the Garmin.
Finally, the Vesper Cortex unit is amazing. If I were getting AIS anew, and if I could afford it, that is what I would get. I've been watching it since it was first announced about a year ago. There's nothing else like it on the market, and I think predictions are right that it will change VHF/AIS integration in much the same way that smart phones have changed how we use cell phones. Cortex's release was delayed substantially by Covid and the coast guard approval process. But it has been shipping in fits and starts since the end of August, and now seems to finally be in stock in a few places. One of those is Milltech Marine, in Seattle, which is where I got my first Vesper unit. The staff there are really helpful for anyone who wants to call and get their thoughts on the new Cortex or other models and brands (they sell both Garmin and Vesper, as well as many other brands).
And yes, I am itching to upgrade to the Cortex . . . It's probably just a matter of time for me. 😉
Finally, FWIW: I am not critical of Garmin at all. I happen to be a huge Garmin fan, and have been since I got my first handheld black and white screen marine GPS back in the mid-90s. My boat electronics, and my auto and handheld GPS units, have all been Garmin for over 20 years. But NMEA 2000 permits intrabrand functionality. And sometimes, specialty companies have benefits over companies that try to be all things to all people (and all vehicles). For me, Vesper came out ahead of Garmin on AIS.
Gini