New Technology / Robotics
Technology signals, innovation themes, and applied engineering trends. Topic: Robotics. Updated briefs and structured summaries from curated sources.
HAI Seminar: Human Skill Augmentation in Robot-Assisted Surgery
Full timeline
0.0–300.0
The shortage of surgical procedures leads to significant medical errors, which are a major cause of death, prompting the need for augmented systems in robot-assisted surgery.
- There are around 230 million procedures conducted worldwide
- The world is short of at least 140 million surgical procedures
- Medical errors are the third leading cause of death in the United States
- Around a quarter of a million Americans die each year due to medical errors
- % of medical errors occur inside the operating room
- % of surgical complications from medical errors are avoidable
- The research focuses on designing systems to augment humans in robot assisted surgery
- One aspect of the research is improving motor skill learning in surgery
- Another aspect involves designing autonomous systems for repetitive tasks like suturing
- Robot assisted surgery is a sub-paradigm of minimally invasive surgery
300.0–600.0
Robots possess unique capabilities that allow them to perform multiple tasks simultaneously, enhancing surgical precision and situational awareness for surgeons.
- Robots have unique capabilities that are fundamentally different than humans
- Most systems in the market have only one robotic manipulator holding one camera for the surgeon
- Robots can have more than one pair of eyes, providing better situational awareness
- Robots can focus on multiple locations simultaneously without cognitive overload
- Typical surgical systems like the da Vinci have four robotic manipulators
- Robotic manipulators are singularly controlled by the human surgeon through a switching control system
- An example of robotic use is holding an ultrasound scan to locate a tumor inside the kidney
600.0–900.0
The automation of an extra robotic arm enhances surgical precision by allowing autonomous assistance, reducing the risk of human error during operations.
- The surgeon can accidentally move robots outside the field of view
- The goal is to automate an extra robotic arm to assist the surgeon
- The third or fourth arm can be autonomous, helping the surgeon
- Test cases for the robotic arm fall into two categories: physical interaction and non-physical interaction
- The design of the robotic arm must be force aware to handle different tissue stiffness levels
- Applying too little force may prevent proper grasping, while too much force can tear tissue
- The research focuses on an autonomous tissue retraction test
- A six degree freedom force torque sensor measures forces and torques during interaction
- Data collected includes kinematic data, vision data, and force data
- Imitation learning is used to digitize surgery and predict robotic arm movements
900.0–1200.0
The force policy demonstrates superior performance by applying significantly less force and achieving higher success rates in surgical tasks, leading to safer and more effective procedures.
- Collected 60 demonstrations over one hour using the da Vinci surgical system
- Trained two policies: a force policy and a no force policy
- The force policy is three times more successful than the no force policy on 50 rollouts
- The force policy applied 62% less force compared to the no force policy
- The force policy applied smaller values of forces, typically less than one Newton, more often
- The force policy was more gentle over time in its interaction with the tissue
- The force policy was 3.5 times more successful on unseen tissue samples compared to the no force policy
- The force policy applied 110% less force on unseen tissue samples compared to the no force policy
1200.0–1500.0
The updated teleoperation system enables two humans to control three robotic arms simultaneously, enhancing surgical precision and collaboration.
- The extrobotic arm can work with teleoperated arm surgeons surgeons for tri-lateral manipulation
- A data collection system was updated to allow two humans to control three robotic arms simultaneously
- One human controls two arms with two hands, while a second human uses the Phantom Omni device device to control the third arm
- The architecture uses the action-tracking transformer model with kinematic and vision data as inputs
- The system can predict the motion of one robotic arm autonomously while the others are teleoperated
- The project started with a two-handed task of object handover and progressed to a trial-literal task with three arms
- Demonstrations showed various configurations of autonomous and teleoperated arms working together
1500.0–1800.0
Robots can operate in parallel at multiple locations without cognitive limitations, enhancing efficiency in delicate tasks like suturing.
- Lift arm is teleoperated, right arm is autonomous
- Successful handover of objects
- Middle arm is teleoperated, two side arms are autonomous
- Goal is to pick Z3 objects and place them into a common location
- ACT architecture could be used for three handed tests
- Three handed tests can be collaborative
- Ongoing work involves collaborative analysis of automation and teleoperation
- Robots can focus on two different locations simultaneously
- Humans find it difficult to focus on two locations at the same time
- Robots do not have cognitive limitations like humans
1800.0–2100.0
The hierarchical state machine execution model enhances surgical robotics by enabling concurrent operations, resulting in nearly double the speed of traditional methods.
- We simply set the execution as a way the autonomous execution is happening using a state machine
- It allows having state machines and sub-state machines and hence the word hierarchical
- Our proposed execution model was almost twice as fast as the other two execution models
- By having multiple arms, we open up surgical robotics as one application area for multi-robot systems
- Robots can have more than one pair of eyes
- We designed an extra camera for the surgeon that can provide an auxiliary view
- The design of the camera is a cylinder inserted vertically through the same incisions
2100.0–2400.0
The dual view system in robotic surgery enhances accuracy and speed in surgical training, leading to improved surgeon skills.
- We can have a multi camera multi view system in robot assist surgery
- Conducted a user study comparing single view group and dual view group
- The goal of the surgical training test called Why Richey or Test is to move a ring from one side of the rail to the other
- The dual view group had a picture in picture view
- The dual view group was 35% more accurate in the training test
- The dual view group was 25% faster in performing the test
- The design of the pickup camera allows for adjustable inter-popularly distance
- Increasing the distance between vision sensors improves perception of the scene
2400.0–2700.0
The use of 20 millimeters and 30 millimeters in the dip-thranking test resulted in lower Hamming distance scores, indicating better performance in sequencing tasks.
- Subjects performed a dip-thranking test to sequence poles from shortest to tallest
- The required precision for the task was 2.5 millimeters
- The sequences provided by subjects were compared to a ground truth sequence using Hamming distance
- Lower Hamming distance scores indicated better performance
- millimeters was identified as the sweet spot for optimal performance
- The results suggest a shift in the design of endoscopes and cameras
- The idea of a dynamic or adaptive camera baseline was proposed
- The current camera system is teleoperated by the surgeon, with ongoing work to automate it
- Surveys indicate that the optimal forces used by surgeons vary depending on the task
2700.0–3000.0
The integration of robotics in surgery aims to enhance human capabilities, fostering trust among surgeons and patients, which is crucial for adoption.
- Did we do like error analysis before we jump into sub projects?
- Some of this was more opportunistic because I find out that, you know, when you, theres also a beauty to tell surgeons, okay, heres another cool technology
- Action shot transformers for example is an obvious case from a technical perspective
- I think now is the time to do the human augmentation
- We want to gain the surgeons trust
- The way to gain the trust of a critical mass of the surgical community is to do things like, you know, automating the camera for them
- Patients actually just because of, you know, what they hear in the media, they would actually ask if they can be operated on using Zorab
3000.0–3300.0
The development of autonomous surgery technology aims to improve patient outcomes, which is essential for gaining patient trust.
- Researchers have the responsibility to be forthcoming about the capabilities of autonomous surgery technology
- Gaining patient trust is linked to achieving great patient outcomes
- Demonstrating improved patient outcomes can help build trust in new surgical technologies
- There is a need to explore what tasks in surgery should be automated
- Current literature focuses on a limited number of tasks for automation in surgery
- Engaging with surgeons over time can help identify opportunities for automation
- The potential of a data-driven approach in robot-assisted surgery is an exciting area of research
- The Albin H. Embodiment Initiative involves collaboration between Nvidia, Johns Hopkins, and a technical university clinic
- The initiative aims to collect extensive data from various levels of surgery