Welcome to my colon: A tech pioneer turns to virtual reality to guide his own surgery

LA JOLLA, Calif. — Spend enough time with Larry Smarr and, chances are, he’ll invite you to step inside his colon.

Like more than a million Americans, Smarr has inflammatory bowel disease. Unlike most, he also runs a cutting-edge institute replete with reams of ultrafast computers, crack graphics programmers, a towering wall of digital screens and a pitch-black virtual reality cave — all the better to summon up a digital 3-D version of himself that he calls “Transparent Larry.” Among its features is a larger-than-life replica of his colon that includes every nook, cranny, and section of inflamed tissue.

Smarr, 69, is a physicist widely recognized for his work on creating the national network of campus supercomputers that evolved into today’s internet. Now, he runs a futuristic institute called Calit2, housed on the University of California campuses in San Diego and Irvine, that works to advance a host of fields, including medicine. For the last decade, he’s been turning technology on to himself to quantify his body’s most intimate workings, with no clear idea where the experiment might lead.

But when his health started to fail and surgery was indicated, Smarr saw the perfect opportunity to use his hoard of big data and pioneering visuals to inject a much-needed software upgrade into the operating room. He’s hoping his institute can help create the automated software that might make 3-D images in surgery routine within a few years.

His trial run — with himself as guinea pig — started a few days before his surgery last November, when Smarr called his colorectal surgeon, Dr. Sonia Ramamoorthy, to his institute’s VR cave.

“I thought, I know more about the particularities of my insides than my surgeon does,” he said. “That can’t be good.”

Ramamoorthy wasn’t surprised by the slightly odd invitation; nothing about her relationship with Smarr has been normal. “Unlike any patient experience I’ve ever had,” she said, “he came equipped with a PowerPoint.”

It turns out Smarr had detected a glaring omission in today’s modern operating suite: The 3-D computer visuals that are standard in even the most basic of video games are not routinely available to surgeons — despite their utility in detecting unexpected anatomical variations that can derail surgeries.

When they have 3-D imaging available, “surgeons love it,” said Eric Wickstrom, a professor of biochemistry and molecular biology at the medical school at Thomas Jefferson University in Philadelphia. He has worked for more than a decade to develop 3-D imaging for use in cancer surgery and published a 2013 study concluding that surgeons found it useful.

“They say, ‘Our biggest problem is cutting into an artery or vein that we did not expect,’” Wickstrom said.

But Wickstrom said very little 3-D technology has been deployed to operating rooms because it’s still too time-consuming to create the lifelike images from a stack of normal, 2-D images of the patient’s body. More research and programmers are needed, he said, to automate and speed the process.

Smarr’s 3-D model evolved over years and required much painstaking work from virtual reality expert Jürgen Schulze: He first had to go through a stack of 2-D MRI slices outlining each organ. Then he had to combine them to create a 3-D model fit for viewing on a regular computer monitor. Automating this step is a critical process in making it easier and cheaper to use such images routinely.

A surgeon visits the virtual reality cave

When she stepped inside the dark cave and popped on a pair of 3-D glasses, Ramamoorthy was amazed. Imaging of the abdominal area has lagged behind other fields because that section of the body houses so many organs of different densities and because it moves with every breath and every contraction of the intestinal muscles.

Ramamoorthy immediately saw a “no fly zone” where Smarr’s inflamed colon was sitting smack on top of his bladder. One nick from a surgical tool and urine could flood the abdominal cavity. (Luckily, they weren’t connected.) She also noticed Smarr’s large intestine rose much higher than expected and was adhered to his spleen.

The veteran surgeon knew instantly she’d have to change her surgical game plan — and knew she wanted those 3-D images with her in the OR.

“The 2-D images, they’re just not that helpful. They don’t help me think in 3-D,” Ramamoorthy said. Having a trove of 3-D images she could see in advance and even move through in virtual reality, she said, “was like a candy store for me.”

But when Ramamoorthy asked Schulze if he’d bring the 3-D images into the OR, he blanched. “I’m not good with blood,” he said.

In the end, Schulze agreed to come.

“He manned up,” said Smarr.

The day before the operation, Schulze was in the operating room, fiddling with cables, trying to see if he could connect his laptop to the surgical robot and project the images onto screens at the Jacobs Medical Center, a $943 million hospital that opened just days before Smarr’s surgery. The new OR suites, built to handle lasers, robots, and MRIs that can be used in the midst of surgery, seemed the perfect home for the radical new imagery.

And it all worked. While Ramamoorthy manned the four-armed da Vinci surgical robot anchored over Smarr’s body, others in the room had their eyes glued to an 85-inch, wall-mounted screen. It was divided into four views and showed Smarr’s virtual colon in 3-D, his MRI, a view into his abdominal cavity from above the robotic arms, and another close-in view from inside the colon, also generated by the surgical robot.

“It looked like the Starship Enterprise,” said Smarr.

Ramamoorthy likened the visuals to driving with Google Maps. Instead of having only a vague idea of where you might be, she said, “you know where you are, how far along you are and if there are any accidents ahead.”

The top-notch 3-D visuals have been the talk of surgeons at the hospital. And now that she’s had a chance to use them on one patient, Ramamoorthy keeps wishing — especially on the tougher surgeries — she could have them for others.

“I look at my teenaged sons and their Xboxes,” she said. “All of that technology is there — why is it not there for my patients?”

A couch potato discovers kale

Smarr’s obsession with learning everything he could about his body started nearly two decades ago — with a diet.

Smarr had spent two decades in Illinois, directing the National Center for Supercomputing Applications, before he was recruited to UCSD in 2000. He arrived to the seaside campus pasty, overweight, and surprised by the healthy lifestyles that surrounded him.

“I’d never eaten kale before,” he said. “I was afraid they would send me back.” Smarr immediately got a personal trainer, went on the Zone diet, and started taking nutritional supplements.

Smarr lost 30 pounds and got his 6’1” frame into shape. (He still keeps his old driver’s license in his wallet to remind himself what he used to looked like.) Not one to do anything lightly, Smarr wanted to know if he was taking the right number of fish oil capsules for optimum health. So he started taking blood tests to check and then got a little bit addicted to all the data he could collect. He wanted more and more.

“It was like a loose thread on a sweater. I just kept pulling on it,” he said.

There were many roadblocks: He had to fight with doctors who told him that he shouldn’t order his own blood tests, but that they wouldn’t order them either: “They said there’s no insurance code for that. It’s preventative.” He paid for the blood tests out of pocket — and estimates he’s spent tens of thousands of dollars by this point. “We talk about patient-centered medicine, but we’ve clearly got a long way to go,” he said.

In 2005, even though he felt fine, his blood tests showed him something was wrong. His C-reactive protein, or CRP, was five times and then 15 times the upper limit that is considered healthy. “I had no idea what CRP was. So I go to PubMed and start googling,” said Smarr, who has no biomedical training but has served as an adviser to several NIH directors. He learned CRP was a generic measure of inflammation.

It was telling him something was seriously wrong with his body but he had no idea what.

Smarr didn’t suspect his colon because his 2005 colonoscopy results looked normal. It turns out that was because his disease was atypical in many ways. It was late onset, so his intestinal tract didn’t have the battlefield of scars his physicians would expect in a man of his age, and the damage was confined to a small part of the colon.

When his CRP levels shot up again in 2008, Smarr went to his physician. Smarr, who started his career as an astrophysicist, said he was promptly sent home. “They said, ‘Do you have symptoms?’ I said, ‘I’ve got data!’ If I saw something like that in the sky, I’d be training all the telescopes on it! They said, ‘Come back when you have symptoms.’”

The crushing abdominal pain started shortly afterwards. “It was totally predictive,” Smarr sighed. “I look at medicine and say, ‘What is wrong with these people?’”

Ten days of antibiotics eased the symptoms, but never fully got his CRP numbers under control.

Unsettled, Smarr decided to take a look at genetic data he had from 23andMe. (Yet again, he was an early adopter.) He found an SNP, or genetic variation, that indicated he carried an 80 percent increased risk for Crohn’s disease, which can flare episodically.

A controversial obsession with data

In 2011, his numbers got really bad. So Smarr started collecting his stool samples for analysis. He’s had 90 tests at $375 a pop and is still collecting. (Don’t ask to look in his freezer.)

The computer scientist in Smarr is enamored with feces because it contains so much data, including hints about the health of the immune system and intestinal tract. “There are a billion bacteria in a gram of stool. Each has 5 million bases of DNA,” he said. “I look at it like a disk drive.”

By mid-2011, a stool analysis showed his levels of a protein called lactoferrin, which is linked to inflammatory bowel disease, were 125 times above the upper limit. By the end of 2011, meanwhile, his CRP had spiked to 27 times the healthy limit. A colonoscopy showed his colon wall — which was being attacked regularly by his own immune system — was much thicker than it should have been.

At Calit2, Smarr can graphically display a decade of his health data onto a digital wall made of 32 high-definition screens — cholesterol, glucose, insulin, vitamin D, CRP are among the 72 parameters he tracks. “This is me,” Smarr said.

And while he realizes that most people don’t have the same technology at hand, he feels strongly that consumers should take their health in their own hands and seek out data on their bodies.

That’s a controversial view: Many doctors believe that regular blood tests for healthy people can do more harm than good.

Tests can yield false positives, after all, which can cause anxiety and drive up medical bills as the results are checked and re-checked. And while it turned out that Smarr was right to be concerned about his CRP levels, in many cases, anomalies in test results are just that — anomalies that don’t necessarily signify real health threats. If patients rush out to treat every one of them, they’re not only going to ring up big bills, but they might also do themselves harm. Treatments inevitably have side effects.

Given those risks, the Society of General Internal Medicine doesn’t even recommend annual physicals or routine lab tests for healthy adults.

Smarr disagrees. He knows people think he’s a little bit nuts. But he doesn’t care. “If I’m not 10 to 15 years ahead,” he said. “I’m doing something wrong.”

‘I felt like a balloon about to burst’

Because he was so fascinated with the data he was collecting on his disease, Smarr did not want to be treated. “I didn’t want to mess up the beautiful time series,” he said.

He was no stranger to collecting data through pain; he once worked through the night with a raging case of pneumonia rather than give up valuable observing time at the Very Large Array Radio Telescope in New Mexico. “I am a scientist,” he said. “You do what you have to do to get the data.”

But in 2012, the bloating and rectal bleeding grew overwhelming and Smarr gave in. “I’m dedicated to getting data, but I’m not stupid,” he said. He took a month-long course of antibiotics and sterioids. His CRP fell to 1.3, which is considered relatively normal — but he wanted to see it lower still. And he noticed, oddly, that his symptoms did not seem to diminish as the CRP levels fell.

In 2016, a full body CAT scan (including a virtual colonoscopy where, said Smarr, tittering a bit, “they inflate you through your back end with air”) showed his intestinal walls were so thick in places that his lumen, or colon interior, was only a few millimeters wide. It was so narrow, indeed, that even a pediatric colonoscopy tool would not fit through. One section had an unusual kink. Stool was backing up and generating methane gas in his body.

“My wife said, ‘Oh my God, you look pregnant,” said Smarr. “I felt like a balloon about to burst.”

It was time for surgery.

Excising ‘the demon baby’

With the help of the virtual preview, Smarr’s surgery went smoothly. Ramamoorthy used the robot to cut and cauterize the inflamed colon, staple his intestines back together and reopen the lumen. Ramamoorthy then pulled out two large fistfuls of ugly red tissue — what Smarr calls “the demon baby.”

Luckily for VR expert Schulze, who sat through the entire procedure, there was not much blood. Smarr lost only about 15 milliliters, not much more than a typical blood draw.

One clear benefit of the 3-D imagery was that surgical time was shorter, which often leads to a better outcome for patients. Ramamoorthy estimates she shaved at least 30 minutes off the five-hour surgery by knowing what to expect in advance and saved another 45 minutes by docking the robot onto Smarr in a much higher location than usual to deal with his adhered spleen.

The surgical recovery was swift. With Smarr as a patient, however, it was not routine.

Just hours after waking, Smarr was back to collecting data. He desperately wanted to track his CRP numbers post-surgery to see how the procedure had affected his body. But the nurses told him CRP tests hadn’t been ordered by his physicians. Smarr refused to let them take his blood until they found a doctor who would authorize the CRP test along with the others they were running.

“They thought I was a loon,” he said. “A problem patient.” Aside from being neat data points, Smarr thinks CRP tests should be routine after all surgery for early detection of problems like sepsis. “It’s an $11 blood test after a $150,000 operation,” he noted.

He got his way and watched his CRP drop in the following days from 61 to 42 to 30 to under 1 for the first time in more than a decade. He also smuggled in his Fitbit and — to the raised eyebrows of staff — a team of colleagues who outfitted his lower abdomen with electrodes, hoping to detect the instant his colon started working again.

Now fully recovered, Smarr shows no sign of slowing down.

His goal is to create software that could automate the creation of 3-D images of organs, and he thinks imaging companies could be using such software within three to five years. He also thinks his institute can play a role in training a new generation of medical imaging VR experts.

For now, he offers digital tours of his body to physicians who ask and gives talks — sometimes alongside Ramamoorthy— about the power of 3-D visuals.

“Do you want to hold my colon?” he’ll ask as he hands out a 3-D printed replica of the organ.

He also extols Ramamoorthy’s work at every chance. “I love my new colon,” he told a crowd at a recent surgical grand rounds. “It works so great.”