3D Surgical Imaging in Hamburg, Germany: New Technology for Greater Precision

[00:00:00] So, clearly, that's not how we are online.

[00:00:09] Ladies and gentlemen, my name is Lars Kruse. I am a senior physician in orthopedics here in Altena at the Asklepios Hospital. I am very pleased that you have taken the time this evening to spend a few minutes with me.

[00:00:22] I would like to tell you something about Holo Medicine and the future of medicine, what we can do with it and what we will be able to do with it in the future.

[00:00:37] Some nice pictures. You probably already know Hamburg. Now come some even more interesting pictures. Who is this man?

[00:00:47] This man is one of the co-inventors of the first mobile phone. This is the Motorola model he is holding next to his ear, and this was in the 1970s. I don't know, those of you who are older may remember these models. They were already somewhat smaller than the first mobile phone I ever saw.

[00:01:08] Today, of course, phones look more like this. They are flat like a piece of paper, they can fold, and you can do all kinds of things with them, not just make phone calls, but probably even build an atomic bomb with them. So, they are extremely high-tech, and this is a major advancement that humanity has achieved in just a few decades.

[00:01:36] A little about my background. I came to trauma surgery and orthopedics mainly because I grew up on a farm. I had to work with my hands from a relatively young age. When I then had to decide what I wanted to do, and my father would have liked me to become a farmer, I said, "Well, I already know how to do that. I would like to do something different." And so I ended up in medicine, and surgery and orthopedics, with all the hammering and tapping involved, seemed to suit me quite well.

[00:02:11] These are the pioneers of trauma surgery in Germany. There is also a Russian among them. He discovered or developed an external fixation device for bones. This is the man down here. And this gentleman here, wearing the green jacket, was always very elegantly dressed and was interested in fast cars. He was once head of the hospital here in Hamburg's harbor and developed the intramedullary nail. He was the one who had the audacity to put a nail inside a thigh bone. When he introduced it to the market, or developed the idea, all of his colleagues criticized him. Nevertheless, he succeeded, and today it is one of the most established procedures in trauma surgery for treating bone fractures, particularly in the long bones, such as the thigh and lower leg, but also the upper arm.

[00:03:16] I have been looking into the question of how we can safely work within structures like the one shown here. This is a pelvis, and here the patient is lying on the operating table. This is an imaging device that we use to display the bony structures of the pelvis. We then want to place screws here. Safe placement means that we usually make only a small incision. We cannot simply expose the entire pelvis because there is the abdomen, with many organs, blood vessels, and nerves. Getting a screw safely into exactly the right position is sometimes not easy. The conditions with a simple imaging device are also not always ideal.

[00:03:55] Here, for example, there is an opening through which an important nerve passes. It is responsible for supplying the leg, so we absolutely must not injure it with our instruments or implants.

[00:04:07] Here you can see what can also happen. With this technique, a screw can end up completely outside the bone. This raised the question: Can we improve this somehow to prevent exactly this?

[00:04:24] Here you can see a postoperative CT scan. After the operation, this kind of computed tomography imaging is performed, and you can see that this screw should actually be completely inside the bone. Instead, it is extremely close to an important blood vessel, which can lead to dangerous bleeding.

[00:04:42] To prevent situations like this, people began considering the use of computer technology in the operating room. This is known as computer navigation. Computer navigation was initially introduced primarily for safer instrumentation and screw placement in the spine.

[00:05:00] In the spine, this is the vertebral body, and in the middle is the spinal cord, which must not be injured. The space available for placing these screws around these small structures is extremely limited. The screws therefore need to be placed very safely and precisely. Navigation is extremely helpful because image processing ultimately provides all the necessary information. It allows the surgeon to work within this seemingly virtual world and be confident that what is planned there will also be accurate in reality, meaning inside the actual patient, without causing complications.

[00:05:42] We have worked extensively on this. During my time at the university, I conducted quite a lot of research in this area. We already had about ten years of experience with it. It was all working well, but we continued trying to develop it further.

[00:05:57] Here you can see another specimen. It is not a human patient. With this combination of CT imaging and computer navigation, we must always perform a quality check. What I see on the screen, with all these colorful vectors and other visualizations, never corresponds perfectly to reality. If there are shifts or disruptions in the computer system or technology, it can happen that in reality the wires or screws being inserted are not positioned correctly at all. There may not even be bone there, but instead the bladder, uterus, or other structures that could be endangered.

[00:06:39] To prevent this, there have been further technological developments. I have always been very interested in how to make the process simple and safe enough that patients can have confidence that what we do is as accurate as possible.

[00:07:01] This is particularly important, for example, when dealing with a child's affected cervical spine. The safe zones for inserting screws, as shown here, are extremely small. Blood vessels also run through these small openings, and they are important for supplying the brain. We obviously do not want to injure them.

[00:07:30] When constructing these systems, the head is essentially secured to the cervical spine to prevent instability and to prevent patients from suffering spinal cord injury. We therefore need to be certain that the implants are positioned exactly where they belong.

[00:07:48] This is not just an idea that I came up with. Other people have also been working on it. Shortly after the collapse of the Eastern Bloc, or after the fall of the Berlin Wall, in 1992, many people were already thinking about these technologies. Since then, however, development has increased exponentially.

[00:08:09] If you now search for what "virtual reality surgery" means in medical databases, there are now thousands of results. There are more than 3,000 or even 4,000 publications and references. There is so much literature now that an individual can hardly read everything. This is not only an interesting topic in Altena, but around the world.

[00:08:38] So, what do mixed reality, augmented reality, and virtual reality actually mean? Ultimately, all of these technologies serve a purpose: their essential function is to give surgeons greater safety and a better understanding of anatomy. They also help with surgical planning in many different ways, including operating time, blood loss, the extent of surgical access, whether large or small incisions are needed, and minimally invasive approaches. All of these factors can be incorporated. Ultimately, the goal is to make surgery better and safer.

[00:09:17] To do this, you need to wear a pair of glasses. I brought one with me. They look a little fancy, but they are becoming quite comfortable. In the future, doctors will probably approach patients wearing these glasses, at least for the next few years, although they are currently relatively large.

[00:09:38] These HoloLens glasses, in this case, show me something that you cannot see. It is essentially a hologram that is also displayed on the screen. This pelvis is visible only to me because I am wearing the glasses. During an operation, however, I can see it while the glasses still function as normal glasses. I can look normally through them while the images of the bony structures are superimposed onto the patient's body. I therefore know exactly where everything is. I can do both things simultaneously. That is why it is called mixed reality. Through the glasses, I can place a virtual image onto the patient while simultaneously seeing the actual patient in reality. And when I no longer need it, I can simply lift the glasses up and see normally again.

[00:10:30] These are technologies that we need to prepare for. They can provide significant advantages.

[00:10:43] I have brought a video with me. In this video, you can see what kind of perspective this technology can provide. This is a tumor on the spine. What you can see on the sides are the ribs, and here is the spine. The tumor is shown in brown.

[00:10:59] Using the pen, or rather the controller, I can take this virtual image that exists in the room—there is even a coffee cup sitting on the table—and move it around in any space I want. I can rotate the image and, for example, look inside the spinal cord area. The bone remains visible, and with the glasses I can essentially move into this three-dimensional structure.

[00:11:22] I can precisely determine the resection areas—where and what I want to remove. I can plan how to remove the tumor completely, so that it is removed in its entirety and hopefully does not return after the operation. I can also plan the screws that are important for the procedure and determine exactly where I want to place them.

[00:11:47] As you can see, there are quite a lot of images, but they provide even more information and additional safety when performing relatively complex operations with greater precision.

[00:12:03] Here is another film, showing surgical planning and a patient. We obviously do not want to operate on this particular patient.

[00:12:36] The first thing I noticed was what was missing, and the patient also looked as though he could almost touch it.

[00:13:18] Here is another structure. This is a fractured lower leg with involvement of the joint near the foot. You can play it and see that with this cube, I can move through these structures and scroll through the joint. This is a 3D reconstruction. Here you can see the individual parts of the bone, and I can plan precisely how, in this case, where the fracture was severely damaged, I can gradually put the individual parts back together during surgery. This is incredibly important for preoperative planning, especially in complicated fractures involving a joint, to determine which step should come first and what sequence of steps should follow so that the final result is as good as possible.

[00:14:24] As you can see in the next film, this is after an operation. This was a hip fracture, where the patient had fractured the hip. The joint had also been displaced. These images can also be generated directly in the operating room. You can see the screws and plates that were inserted. Here we can see the femoral head again. Here is the femoral head, and I can see exactly where the fixation materials, screws, and plates are located. The joint itself has also been reconstructed properly, so hopefully the patient will be able to enjoy this hip joint for many years to come.

[00:15:12] We have already seen this, but it is not limited to trauma surgery. It can be used in many other medical specialties and across different disciplines. Here is an ENT specialist planning the removal of polyps or tumors in the head and facial region. Here is the liver. This technology can also be used for liver resections, meaning partial removal of the liver, or tumor treatment, to precisely plan where and in which areas to cut and remove the tumor, while leaving healthy tissue undisturbed and removing only the diseased tissue.

[00:15:53] It can also be used in medical education. Students can use simulations, much like pilots, who have to complete many hours of simulation training before they can obtain and maintain their license. They can train to become better and to perform these skills as effectively as possible, ultimately contributing to better patient outcomes.

[00:16:20] Here you can see another area. This is the liver. You can display the blood vessels and bile ducts within the liver tissue. You can precisely plan where the tumor is located and where it needs to be removed.

[00:16:35] Here we have a tumor in the head area, which neurosurgeons can also use this technology to plan. It can also be used in plastic surgery as a simulation tool to show patients what they may look like after a particular procedure. This is not necessarily surgery in the traditional sense, but there are many reconstructive procedures for patients who have suffered serious trauma or other difficult circumstances. In these cases, it can provide significant benefits by showing what can potentially be improved.

[00:17:10] This is called "Natural Rendering." With computers and artificial intelligence, we can use the enormous datasets generated by MRI or CT scans to isolate individual structures. We can remove the skin, remove the skin and fatty tissue, so that we can see the muscles and bones. We can also remove the muscles and display only the bones and blood vessels, or display only the blood vessels, which is particularly interesting for vascular medicine and vascular surgeons. This allows many different aspects of anatomy to be displayed, with each specialty able to use the technology according to its needs.

[00:17:55] Here is another film showing how I use it in the operating room. This is a tumor in the area of the sacrum, near the sacroiliac joints. I am wearing the glasses now, and the setup is arranged so that I can see everything. You cannot see what I see because I am the only one wearing the glasses. However, several people can wear them and synchronize them, so that multiple participants in the operating room can see the same thing.

[00:18:24] This is how the procedure is planned. During the operation, you can see the MRI images of the tumor. I can directly superimpose this area onto the patient. We do not want to show too much blood, but I can synchronize the images and see exactly where the resection boundaries are, how far I need to go to the right, left, up, down, forward, or backward to safely remove the tumor.

[00:18:59] These are all technologies that are still being developed. I always think back to the old large telephone and compare it with what we call a telephone or smartphone today. The development has been incredibly rapid.

[00:19:16] It also allows us to educate patients. As students, we are naturally shown these technologies, and we learn how they can be applied in everyday surgical practice. But educating patients is particularly important because now the patient can also wear the glasses while the doctor wears his glasses. Both can look at complicated structures in the same virtual space. This gives the patient a real opportunity to understand what the doctor is explaining and what condition they actually have.

[00:19:49] This is difficult because doctors study these subjects for many years and acquire a great deal of information and knowledge. Then a patient comes in, and within a very short period, the doctor has to explain what the patient has in a way that is easy to understand. That can be quite difficult. The patient is more satisfied because they understand what is happening. Patients understand their situation better, which increases their trust. Only an informed patient, who truly understands what is happening to them, can have the necessary confidence to place themselves in the hands of a doctor. That is how it should be.

[00:20:30] Here is another film in which I explain to a patient what they have, or what their brother has. This is the ankle and foot. You already know the foot. We can simply move around within the image because it is not restricted to a physical location. We can use it anywhere to explain to patients or their relatives what the next steps will be. We can move the image around and explain, step by step, to the patient and their relatives what we plan to do.

[00:21:16] An avatar—you probably know the term from movies. I don't know whether it was Walt Disney, but an avatar can represent a situation where sometimes I do not know how to do something better, or how to do it best. With this technology, it becomes possible to simply call a colleague in New York, Australia, or anywhere else. You could even call a colleague in Bremen, Hanover, or elsewhere. The colleague can put on a pair of glasses, and then both of you see the same virtual space. You can discuss the situation in that virtual space even though the colleague is not physically present. You can even invite them into the operating room in real time. If they are wearing the glasses, they can participate remotely in the operation. It could be the greatest expert imaginable. You simply invite them, they put on the glasses, and watch what you are doing during surgery. They may then be able to provide a valuable or decisive suggestion.

[00:22:27] We have prepared a film to demonstrate this. You can see the blue figure here. That is a colleague of mine from New York. This is a reconstructed upper arm head, and here is the shoulder. These are the fracture fragments. He can also interact with the shoulder and show me how he would approach it. This is my hand. I can rotate it, and he can rotate it as well. We can both interact with this virtual structure and discuss the case. We can discuss what he believes is the right approach and perhaps reach a better result together.

[00:23:13] And even that is not the end. There are now web applications, tools that make it easier to access relatively large amounts of data from an individual patient. This means data from a general practitioner, orthopedic specialist, or another healthcare provider can be transferred very quickly. These systems are often cloud-based. Of course, data protection must be maintained, but the important point is that no time is lost by sending data through the postal service. Instead, information can be shared almost instantly with people anywhere in the world, provided the patient gives consent. We can then share the information and try to find the best solution for the patient.

[00:24:08] This technology makes things easier. It makes us more flexible because we are no longer restricted by location. Ultimately, I could even give medical advice while on vacation if someone needed me. I could put on the glasses while on vacation.

[00:24:22] It will not be long before these relatively large glasses—just think back to my first pictures of the large old mobile phone compared with the smartphones we have today—become much smaller. Soon, these glasses will look like completely normal glasses, like the ones I am wearing now. Then you will be able to speak directly through the glasses. I could say to the glasses, "Please show me the patient's laboratory results," and I would then see the laboratory results virtually in the hologram. I could even explain the images to the patient while I was on vacation. This will become possible in the near future. There is no need to be afraid of it. We should be excited because it will make many things easier.

[00:25:13] As I said, this technology will not only be used in trauma surgery and traumatology. It can also be used in general surgery, plastic surgery, radiology, dermatology, and many other fields. There are countless areas of application where this new Holo Medicine, mixed reality, augmented reality, and virtual reality can make our work and lives easier.

[00:25:41] In trauma surgery, as I said, this is just one aspect. You saw this image at the very beginning. Here is another short film, with a little animation.

[00:26:02] And it is accompanied by slightly more modern music. Here you can see the patient in the appropriate position. We have been doing this for some time. It has already been about two years, so we have been working on it quite extensively. I think the possibilities are incredible, and we have already experienced many situations where it has changed how surgery and patient care can become safer and more efficient.

[00:26:38] [Applause] [Music]

[00:26:46] I actually always do this when I give a presentation. You should also take something away from it. Friedrich Nietzsche was, of course, a famous German philosopher, and he said that originality means seeing something that has not yet been given a name or cannot yet be described, even though it is right in front of everyone's eyes.

[00:27:08] In other words, there are people who are standing in a forest without even realizing they are in one because they do not know what a tree looks like. You cannot see the forest for the trees if you do not recognize the tree or give it a name.

[00:27:22] This new technology, and the technological revolution—or perhaps evolution, because technology has always developed step by step—has continued to evolve since humans first learned to walk. Humanity continues to make new inventions, and computer development, technological advancement, and now the new key term, artificial intelligence, will help us process incredibly large amounts of data more quickly. This is ultimately intended to make our lives easier and, in the end, to give us more time for the things that truly matter.

[00:28:08] That concludes my presentation. If you have any questions about this topic, I would be happy to answer them.

[00:28:34] If not, then I may say goodbye. I wish you a pleasant evening. It is summertime, so enjoy the summer and the barbecue, have something nice to drink, and I wish you good health. Have a safe and enjoyable summer, recover from the pandemic, and hopefully we will hear from and see each other again. Goodbye.