good afternoon
the truth is to a an audience of
robotics researchers our speaker today
needs no introduction mls earth Taylor
did a bachelor's degree at Johns Hopkins
in PhD at stanford and then joined IBM
research where he did a long list of
things but one of them was to lead the
computer system surgery group which was
responsible one of the things for rubber
dr. and went back to johns hopkins in
1995 where he's been the director of the
engineering research center for computer
integrated surgical systems and
technology and is now going to speak to
us about the sorts of things that he's
done there he is a fellow of the I
Triple E among a number of other
societies recipient of the I Triple E
robotics pioneer award among a large
number of other awards so Russ well
thank you very very much cam I I was
just mentioning odd I have been coming
to CMU for various purposes for 30 years
and I think one of the first things that
made me think serious about moving to
academia was I was came to do a paper
with matt and Ken Goldberg and realize
that when I was coming I was happy and
this was fun no I be almost fun too but
what I what I'd like to do first is is
to give these are the usual obligatory
thank yous and acknowledgments and
disclaimers and I'm going to be talking
about work a lot of it I've been
personally involved with but but also
work of my colleagues and tried to
indicate that on the slides but this
this picture here I was working at IBM
on systems like this in fact this is a
system I developed with Ralph Hollis
these were Ralph's fine positioning
devices and it had submicron for
cision over an area about the size of
that tabletop and we used it in
manufacturing processes and this is of
course robo doc now these systems they
look really pretty different but I I
think the underlying insight that made
me begin to be interested in systems
like this is they have something in
common what they do is they couple
information to action often in
partnership with humans to fundamentally
change a process and it is really the my
belief that this interventional medical
computing can have the same kind of
impact on medicine that it's at on
manufacturing and perhaps by the end of
the talk if you don't already believe it
you might at least agree that it's
plausible and this picture here is very
similar to a picture that I first put
together for my bosses at IBM what
trying to explain why computing company
should care about something like medical
robots and if you look the notion as you
start with everything you know about the
patient and everything you know about
people in general and surgical
procedures you moosh this together into
some kind of a computational
representation and I've called that a
model it may or may not be an accurate
geometric description but a
representation of this information that
lets you do what you need to do and the
first thing is you need to diagnose the
patient's condition and plan your
intervention well once you've made the
perfect plan what you really have to do
is execute it so the idea is all of this
information comes into the operating
room or an interventional sweet where
once you've kind of fused the model to
the physical reality you can then use
appropriate technology to help you do
what you plan to do and to verify that
you've done it well we're engineers so
what is that that's a control loop tied
to a particular patients intervention
and you notice this picture here is
replicated and that really is to remind
us
that this basic process information
process actually occurs at many many
many different time scales from weeks
and months for a patient management down
to second by second in the operating
room now there was something else in
manufacturing that we did we you have
here an information control process that
helps you be consistent and because its
information controlled you actually
maybe know what you did you don't
there's no law written in heaven you
have to throw the data on the floor so
the other thing that we did in
manufacturing that was as much
motivation to me even from the beginning
was this to take this information
eventually you know what happened with
the patient if you know what you did
perhaps you can relate the two and more
effectively improve your treatment
processes so this is basically surgical
cad/cam and surgical total quality
management and my basic hypothesis is
that it is really the synergy it is
these two loops and their synergy that
has the potential to make a profound
difference in the practice of medicine
well actually that's a pretty old idea
this is Emma hotep who was generally
believed to be the father of surgery and
with Howie you didn't meet him in Egypt
I guess but and I what he taught was a
highly empirical process that weren't
computers but it involved basically all
of those steps but of course the
difference is the power of computing and
equipment interface to computing can
make that process much more effective so
just a couple examples and I'll come
back and talk about them a lot of the
work that we've done over the years has
involved placing needles in a minimally
invasive fashion very accurately on
anatomical targets that you can see in
medical images
here's another set of examples in which
typically you're doing very highly
interactive surgeries such as
microsurgery or these are the da Vinci
robot and what we're doing is we're
using this information very
interactively with the surgeon in the
operating room and you you can think of
this as surgical assistants but they're
the same basic building blocks put
together slightly differently for
different applications which leads me to
my next point if I can make this slide
come on guy hmm build the gates what are
you doing to me okay thank you whoa
I have no idea what's happening
sometimes if I care about running a pro
k so here the idea is to build this
we're fundamentally talking about
systems and there's a lot of parts there
are a bunch of things these are
buzzwords that we have in medical image
processing and modeling the computing is
ubiquitous that we're also dealing with
physical world so the interfaces between
the data world in the world and the
surgeon are important and there's a
science to systems medical systems
especially there's some very important
systems constraints so you need to think
about all of these things together and
that was the motivation that really led
us to propose and get funded this
engineering research center off or
computer integrated surgery where the
kearney carnegie mellon was one of the
charter core members of this partnership
over the years and generally i'm going
to be using examples from a very broad
set of research all focused on systems
in the balance of my talk we spent what
was a maybe by robotics Institute one
year's budget but we spent a lot of
money over over about ten years of seed
money and we're now self-sustaining just
to show these are the pictures from our
first site visit where the goal of these
engineering research centers is you
should be making contribution sort of
basic science and technical artifacts
and systems and using these different
levels to motivate and look at the
center geez between these levels and
this is the last time I I bothered to
make this picture
and we've done a lot and I can't talk
about it all so what I would like to do
for the balance of my talk is 20 where
did my my slide vanish is to go back and
use this picture as the organizing
principle for what I want to talk about
and the first piece of that is
patient-specific its specific modeling a
lot of this work is based on medical
imaging and there's a very very broad
research agenda here and that we've
worked down over the years I want to
just as an example pull out one common
thread that has been quite important for
our research and the communities over
especially about the last five six years
and that is where you have previous
information about a patient I either
like cat scans of the patient or you
usually know if it's a human you're
operating on and you know something
about how people's anatomy varies and
now you get some new information in the
form of images so what you want to do is
mush all of this information together to
make a better representation of the
patient for whatever it is you have to
do right now and often you're doing that
are actually during the intervention
itself in other cases are in planning
the intervention just a few examples
here is one this is some work I had some
involvement with but what we have here
is a patient with kidney tumor and we
want to cut the tumor out while leaving
the rest of the kidney as much as
possible undamaged and so the surgeon
needs to know where to cut so what we
took is stereo video from this is this
da Vinci surgical robot made a 3d
representation of the surface
of the kidney fuse that information
registered it with a preoperative CT
model where we knew where the tumor was
and then generate various graphical
overlays to tell the surgeon here is
where things that you can't see directly
because they're below the surface are
like the tumor the red is the cut here
line the yellow are the filling system
of the kidney another example we've done
quite a lot of work motivated mainly by
orthopedic applications where what we're
doing is we know we've got a human but
we only have a few x-ray images but we
want to get a good 3d model of the of
the person or a best guess as to what
the cat scan would have been if we can
take a cat scan right now and there
there's this has become a huge cottage
industry in medical imaging we did some
of the earlier work and there's a large
community here's one one example of how
this works basically the here is a just
sort of an edge map of what is really a
3d volumetric density model and you get
rough alignment now we're adjusting each
of the femurs to be there most likely
and then we're kind of re-registering it
all together and you can get models that
are good to about two millimeters from a
small number of images and this is
typically good enough for for the kind
of thing you need to do in planning or
navigating for orthopedic surgery one
use for planning is there's a orthopedic
procedure called an osteotomy and this
is a hip osteotomy where there's a
cartilage defect or other defect in the
help and what you're doing is you cut
the socket part of the hip joint free of
the rest of the pelvis you move it and
you bolt it together with 15 or five
hundred dollars straps of
metal and seven hundred dollar sheet
metal screws and the the point is that
one of the common cases are with young
women with hip dysplasia now with young
women you really don't want to use a lot
of radiation in their their pelvic
region if you can help it so what we're
doing is where we do take a CT just
basically of the socket because we want
that accurate and then combine it with
just a couple x-ray images and
statistical information and by mushing
this together we can get a quite a good
representation that is plenty accurate
both for planning and then for helping
us execute this this osteotomy another
example in in the operating room if you
take x-rays 100 x rays from all around
the patient you can basically do a CT
like reconstruction of the patient the
problem is to do it accurately you need
to get all of the different projection
directions and that is not always
practical in the operating room for a
lot of reasons so what we did again is
we take a prior CT or statistical
information take whatever x-rays we have
make our best guess as to the model and
then use the model to fill in any
missing x-rays and we call this hybrid
reconstruction and again on it produces
significantly better reconstructions
that are good enough to provide
effective intra-operative feedback to
the surgeon one other example here is
this is actually a screening application
if you if you have osteoporosis people
want to do a bone scan a dual energy
x-ray and then compute your fracture
risk which is basic
Lea a biomechanical beam basically it's
the strength of a funny-shaped beam
calculation you're doing it's
fundamentally a 3d calculation and but
you can't really do a CT scan two times
a year for purpose of screening so what
we do is we take a small number in this
case four of these dexa images combined
it with a statistical model and make our
best guess as to what the CT would have
been and then if you do biomechanical
analysis on our best guess CT and
compare that to the biomechanical
analysis you would do on a real CT you
get correlations better than point nine
so that's very encouraging so these are
just I'm trying to give just a quick
overview and I don't know how long I
will wind up speaking but that is some
of the flavor of the the imaging and
modeling component of what we're doing
or off sometimes it's preoperative
imaging sometimes it's intra-operative
imaging and there are a few other
examples that you'll see kind of
sprinkled through the talk planning I'm
not going to say a lot about it's
clearly pretty crucial sometimes you're
doing it on the fly a lot of an awful
lot depends on the particular
application like here are you doing
biomechanical analysis this is that
osteotomy here what we have is
statistics of where cancer is likely to
be found in the prostate
and if you want to do a biopsy to sample
the prostate to see it is their cancer
there you want to pick the biopsy sites
that minimize the chance of a false
negative and so you have to take these
statistics and register it with a real
prostate here's an example safety
barriers in a neurosurgery application I
have a slide about later execution well
this is the robotics Institute and
robots have the best movies so well I I
let me talk a little bit about some of
the some of the some of the robotics
oriented work again it there's a lot
what we're fundamentally dealing with is
coupling information to action in the
physical world and you can think of the
robot is as as the one direction but
also sensors coming back the other
direction it's that whole interface
between the data world in the world that
we're dealing with here okay this slide
is out of order well let me talk about
it what we have here is where we want to
biopsy again this is a big kidney tumor
with the robot in a CT scanner this is
actually one of the early applications
this is maybe well eight or nine years
ago and what you want to do this is CG
fluoroscopy you can see the target in
the image but what you want to do is use
the image actually to tell you where the
robot is to know in this case the
fluoroscopy you can see the needle going
in but oftentimes the the needle isn't
actually lined up with the target and so
you you need some way either to line up
the needle path so you can see it in the
image or know where at least know where
the needle is and so to do that what we
do is we put a fiducial structure
on our needle driver so that from the
pattern of even from a single image we
know exactly where that needle is and
then we can do even out of plane
biopsies or visual cervone to hit a
target but that this is really to remind
us and this does come out to dark I
apologize the we don't always need a
robot and I George stenton has pursued a
similar idea here actually I think the
first uses were actually at Hopkins and
at Tokyo and what we've got is an image
overlay device where if you have a
display in a mirror you can line things
up so the scan plane the the virtual
image of what you're seeing in the
display corresponds to the physical
location in the patient then you just
use your hand-eye coordination to do a
biopsy and so it's hard to see this is
an MRI version this is a CT version of
that idea and this is just finally
finally finally in in clinical going
into a clinical trial if here's another
example where of placing a needle on a
target under image guidance this is a
very very very simple robot for for
basically for prostate brachytherapy the
standard way that you do brachytherapy
is you plan a pattern of radioactive
seeds you're going to put into the
prostate and then they have needles that
you insert into a grid that looks like
this one at a time with the seeds in
them and then you withdraw the needle
leaving the seeds behind the problem
with that is your restricted to the grid
it's kind of tedious because you have to
line things up and and further the paths
all constrained to go straight well
that's a problem because with a patient
with a large prostate on you often can't
hit where you want to go because the
pubic bone is in the way so here this is
a very very very simple robot that
positions a needle guide and is all
registered to the ultrasound which is
giving us the image feedback in an
integrated system and we've done I
believe to clinical trials on this right
now in fact there was just a clinical
experiment yesterday on our one sting on
this yeah the needle goes into the
perineum this is the space between the
basically the anus and the base of the
penis so it goes in this this is that oh
this this is the ultrasound probe oh
this is the needle guide so this is in
the patient's rectum
ask questions lightly songs do my best
yeah that way here's the robot here's
not sure if this is a mannequin or an
actual patient they've done a lot of
actual patients but it's it's a fairly
gross picture we don't normally know
there is a lot of interest in doing both
biopsy and brachytherapy in MRI
environment oh no an MRI of course
you're you you materials become quite an
issue and so here is a joint project we
have with the Brigham to develop I need
new pictures a robot's completely air
actuated for again brachytherapy and
biopsies in MRI the thing that doesn't
really show here is the actual
integration of the control so again
there's sort of direct feedback from the
from the MRI scanner here's another
robot for it this is more from biopsy or
just injections into the prostate again
under MRI guidance so here we have an
imaging coil and a needle guide that is
kind of terrible and then originally
it's just sort of a manually actuated
remote remote mechanism for aligning the
the needle hit the target the really
cool thing to me about this is in less
than two years it went from literally
the back of a placemat sketch to a
clinical trial further the system has
been commercialized well here I'll show
the commercial in a moment here this is
some of the earlier states here this is
a prostate here are targets and then
here we r imaging after we where the
targets were after the the void is where
the needle
when and you can see it's really very
accurate the device really simple device
has been commercialized this by this
company and this is their first
prototype it's the same idea it's
sleeker better you're getting good good
targeting accuracy and they've got it
they've got an amazing imaging coil
having an imaging coil right close to
your target it gives you much better
imaging in NMR I so and now here is
there's ongoing research Lewis Wickham
and Julian you're taquita are developing
now an actuated version of of this same
thing those are all examples and I've
got 40 more I could I could give you we
we talk about needle needle placement
but they're really needle aiming robots
if you have a skinny needle one of the
problems is once you start poking it
into something it's going to bend it's
going to deflect and one of the reasons
for that is most of these needles have
bevels in them or little hooks and you
can exploit that as you advance the
needle if you twist it you can stir it
like a bicycle and so Allison Nakamura
has led the Hopkins effort in this area
cam is also with a little bit different
design has been pursuing similar work
here and I think it yours is on
neurosurgery i think is an academy
you're still doing it so so if you're
interested in it's a cool thing and if
you're interested you've got someone
local or you can come talk to folks here
one of the challenges with things like
this da Vinci robot that everyone has
seen is you have marvelous dexterity in
but but once you get smaller than 5
millimeters that just the mechanisms
become harder you know they're all these
little bearings pulleys pivots the
mechanism gets really hard especially if
you want to keep it strong so one thing
that this is work that the Nabil saman
and I did is you'd like to replace all
of that mechanism which a simple bending
metal so what we have here if you take
two pieces of metal and you kind of time
parallel I push on the one at the top
and pulling the one at the bottom I'll
get bending further those two pieces of
metal can be my structural elements so
here what we have is we have three
pieces of metal you then get bending in
two directions so we have three tubes
it's little girder and running through
each tube is wire so we have two stages
each with two degrees of bending and
then down the middle we just put an
actuator for a for a little gripper
actually what we found out much to our
dismay is that the gripper is very hard
to design to make one that works and
that actually set us back by about eight
months as we went through gripper
designs before we could begin to test
things of this sort that this has now
been licensed to a large medical
robotics company who I will not name but
most people could probably guess and
here again you can kind of see the sort
of shapes you got here's another project
that that we're doing right now and the
goal is um Ellen the motivating
application is on the next slide and let
me do this in another order what we
would like to do is get into a confined
base this is a an osteolytic lesion and
basically clean that space out from the
inside what has happened is the bone
here is deteriorated it will continue to
deteriorate unless you can clean it all
out and then fill it with cement there's
a similar problem that you this is from
where particles in the in the cup liner
of orthopedic implant you can get
similar problems in bone cancer and so
what we're after is a minimally invasive
way of cleaning that out but you still
have to exert a lot of force and have a
large family of tools so we needed a
large lumen with quite a stiff and
strong mechanism and again something
that we could mass fabricate without
mechanism and so what we have here it's
a very clever way of getting steel guide
wires through embedded in a one
millimeter wall thickness tube and so we
have an eight millimeter device with a
six millimeter lumen and it's it's
pretty strong can show so this is just
going to show some of the strength this
we for so what this is showing is that
the units don't match that you get you
know for for roughly seven and a half
pounds it's maybe five tenths or six
tenths a millimeter it so it's quite
Steph and now come on guy so and now you
can also build these things this is just
a new prototype where these are are
doing s bends and we're looking at a lot
of applications beyond orthopedics but
basically places where you need quite
high stiffness and strength in your
distal mechanism with a
large lumen yeah pardon all the shapes
are very simple this is more you're
pulling against the spring and their
guide wires that just run through the
there's one set of wires that end here
and then a parallel set that the go to
there yeah that again this is a
two-stage snake and and with and and the
key point about this is actually the
fabrication process is somewhat scalable
and there are other design and basically
we can modify the design basically by
changing the the flexure pattern well
that one no no that one is that's one
tube with with two sets of guide wires
terminated in different places there are
a lot of options I mean that that's just
kind of a very first prototype where
we're putting something together for
this osteolysis application but there
are a lot of other applications um
here's another example again this is
sort of some system work that we we did
again with this notion of human machine
partnership dexterity enhancement and
here of course this is the da vinci
surgical robot this remotely actuated
robot and what we were interested in
doing is giving the surgeon the ability
to use laparoscopic ultrasound with some
of the same dexterity as if it were open
the point about the da Vinci is the
surgeon suits looks into stereo video
controls these control handles and it
kind of feels to the surgeon like he's
manipulating his wrist and inside the
patient so what we like him also to be
able to do is manipulate ultrasound
inside the patient and um so so we
developed this they developed the tool
and we basically develop the software
and the interfaces to the tool and this
is all built on an open source software
environment that we're actually looking
for more people to collaborate with I'll
say a bit more about that at the end one
of the things this kind of is how we
might want to fuse that da Vinci project
that video da Vinci project with this
ultrasound there is a technique I'll say
more about later called the elastic rafi
where if you push on tissue with
ultrasound with an ultrasound transducer
the speckle moves with the tissue by
looking how much the speckle moves as
you push you can compute the relative
stiffness of tissue and that will often
give you better contrast if you're
trying to find anatomical structures
then plain old b-mode ultrasound you can
also often see a tumor because tumors
are often different stiffnesses then
with Alaska graphi then that you
couldn't see in standard b-mode
ultrasound so the idea is to use either
freehand palpation or or or The Da Vinci
robot to get these alas de greffi images
register those to preoperative CT then
generate an intro operative
visualization here one of the other
things that as you're doing once you
start cutting into the tumor it kind of
all flop or the kidney it all flops
around and you've lost all of that nice
preoperative data maybe you could use
like one of these new connects to kind
of track how all of that tissue is going
but in this case what you have is you
want to keep the tumor is a coherent
meatball with kind of a coating around
it of about five millimeters a healthy
tissue so so if we if so what we do is
after we've done the registration we put
a an electromagnetic marker into
tomer then the idea is once you start
dissecting the tomorrow way the tumor
stays intact and you can sell have a
visualization as to where it is so we've
done kind of this level of lab
experiment we're currently putting the
elasto graphic capability on to the da
Vinci and then the goal is to do these
things together how do you feedback
information to the surgeon for things
like forces or stiffnesses or other
information you can try to feed them
back directly haptic ly it turns out
it's really hard especially if the
master device wasn't designed and often
it's also not too popular with surgeons
especially for some of the microsurgery
applications I'll talk about later
surgeons don't really like you pushing
on their hands so here the idea is to
use in this case graphic visualization
as a sensory substitution and this is
very these examples are very largely the
work of Allison Nakamura and her student
tomo yamamoto and here this is just for
us here what she's doing or what they're
doing is they're palpating to determine
relative stiffness and then feeding it
back again just by showing a surgeon of
visualization of the relative stiffness
of the tissue
this is my personally my largest single
project and it's one that I feel really
lucky to be collaborating with cam on
and this is vet vitreoretinal
microsurgery in fact we now have
actually several existing of prior
grants with cams lab on various aspects
of this project and this is the most
difficult technical search technically
probably the most difficult surgery than
anyone does the surgeon has this is a
large instrument it's seven tenths of a
millimeter across some of them are half
a millimeter across or smaller the
surgeon stairs through a microscope this
is what the surgeon sees it's a little
brighter some of this is just this
projector but you don't really see a lot
you're staring you're holding your neck
really rigid and getting neck and back
pain and so when we put all of this
together it's technically extremely
difficult and surgeon really can't feel
the forces that is interacting with and
really it has a hard time knowing what's
actually happening between the tool and
the tissue and by the way is handshakes
so we're trying to overcome a number of
those limits and so the flagship grant
at Hopkins for this is one of these BR
PS where the goal is really to put
together a complete system this is an
earlier version of that system but the
basic components are here you take a
surgical microscope and equip it with
stereo video so this anas video display
so this is a stereo video microscope
what you don't see which is the true
heart of it is really the computer and
then their various manipulation aids and
smart tools that kind of are all
plug-ins for this basic system
a big part of what we're doing is
computer vision you want you need to
track in stereo in real time these tools
relative to the retina it's a little
hard to see but also track the motion of
the retina once you've done something
here or made a measurement as the retina
moves around you want to know how it's
changed this is just some real recent
work from rogerio Riccio who was a
postdoc with us but here he's tracking
the tool and giving when and when the
tool gets within a millimeter of the
retina you don't want to warn the
surgeon that but really because the
surgeon will get mad but what you what
you might really do is then change the
control mode of the robot if this is a
robot or turn a sensor on or off another
problem in I surgery is light toxicity
we've all gone to the eye doctor and had
our eyes dilated in them know that knows
how it it bothers us if you shine too
much light on the retina you'll sunburn
it and you can actually cause a lot of
even permanent damage it turns out that
the light that does that to you is more
the blue light than red light or
infrared light and so the idea here is
if we're using video microscopy we can
manipulate the spectrum of the input
light and then do various image
processing tricks to still give us a
full-color visualization and so in this
example this is some work of some of our
students well first what we do is we
built a RGB light source out of LEDs
where these match the colors that you
see with the might with the video camera
and then you can you can modulate the
light at a kilohertz you can change
color very very quickly and
and one of the things you can do is are
these movies going to play is um if you
make every other frame say red light and
then you use the full color frames and
then track the moat track the motion you
can then use the where you've had full
color information you can fill in and
you could you can even do this with very
significant reductions again this
projector is projecting colors a little
little off and this this movie here is
body year old there's been some progress
since but this is the kind of thing you
want to do know what are the interesting
challenges is you'd like to maybe have
the computer automatically decide when
the surgeon really needs high high
fidelity coloring when he doesn't and
have it automatically based on
understanding the task modulate no hand
tremor surgeon's hand shaken cam if you
give and seminar to this group yeah so
this is the micron device which is where
again I buy it you can actively cancel
the hand tremor so that are one of our
goals is to incorporate this sort of
free hand device into our total system
and then use various forms of endpoint
sensing video processing or sensing
built into the tool to help us control
the robot to reduce tremor the other
mechanism we have for this is what we
call the steady hand robot and here it's
just as easy for me to show here the
idea here is the surgeon and the robot
both hold the tool and the surgeon the
to the robot feels the surgeon pull on
the tool and just follows his hand and
so there's absolutely zero hand tremor
here this is a hundred micron blood
vessel that we're cannulated this is a
live video but you can see the emotion
compared to free hand and we're using
both of these platforms on for our
research probably the micron will be
quicker to get into clinical use because
it's less of a disruption of the
clinical workflow well okay now I
mentioned that the surgeon can't feel
the tools I'm going to go a little over
4 30 is that ok ok if people walk out on
me that'll be fine I'll talk to an empty
Hall so you lit the forces are effective
10 smaller then you can really feel I
mean can actually did some early work
again here with with with his devices so
what we want to do is we want to build
force sensors into the surgical tool one
of the problems is as the tool goes
through the sclera the force there is a
lot higher than the force there so the
idea is to build the sensing down inside
the eye and to do that what we've done
is we these are large forces compared to
what the thing can feel we've built
we've been using these fiber Bragg
grating oh gee it hid the sensor the
ideas this is an optical strain gauge
and as you pull on the sensor you change
the spacing of a grid which changes the
wavelength of reflected light and these
are excruciatingly Lee precise our
existing tool is about four hundred
micron
I'm stories about the author Mellie
Newton snow the other problem how do you
feed that back you can't with freehand
tool you can't pull on the tool and
surgeon doesn't like it anyhow so we've
been looking at auditory sensory
substitution so you play a little tune
that changes depending on how much force
and you can use speech and graduate
students can have fun at Johns Hopkins
now one we've just started to use these
sensors in some bunny experiments but
one of the other questions is does that
sensory substitution actually help you
and so here we made a phantom so we
could control the interaction forces and
set what we thought was the the tearaway
force on the retina as a kind of a
threshold and then use auditory sensory
substitution with a bunch of medical
students and a few attendings this is
dr. Honda to see if the feedback
actually are helps and this data is not
yet published it's not even completely
analyzed but preliminary data is pretty
encouraging alarm what that means is we
ring the bell when we figure that you
probably tore the retina and if you give
a warning before you're likely to do it
you can significantly reduce the number
of times where you would have done
damage and can you find that very
quickly it doesn't take you any longer
to do the do these peels but it provides
but you can exert lower force while
doing them actually one of the things
here that I think is most interesting is
the potential and oh and then we're
doing things like tissue force
correction one of the things I think
this most interesting here is I
the potential of using that during
training to train someone what the right
feeling is and even if you don't use it
in routine clinical use so they can view
it as training wheels one speculation is
the new generation of surgeons may come
to light the wheels another thing if if
you can use force sensing in various
ways to implement virtual fixtures and
interactive behaviors and for things
like blunt dissection you can make the
robot just want to stay in a tissue
plane and here this was something that
we found really interesting what we did
is we had the robot be very compliant in
a direction where it didn't feel like it
had a lot of force and then we're trying
to tear a membrane in this case it's on
an eggshell and this is the path of the
robot if you look it's hard to see but
just kind of naturally the thing tour in
a circular pattern and now the eye
surgeon got really excited when he saw
that because he said look that's just
the motion we try to teach our residents
to use for something called capsular
exes which is tearing sort of a membrane
on on in the eye as on the lens of the
eye as sort of the first step of of ice
of cataract surgery so this is kind of
cool again this is something that we're
just starting to play with another kind
of sensing is optical you you really
don't know much about the structure of
the retina right point beyond the tool
and there are a lot of reasons you might
want to know that so what we've done is
we've been incorporating a single
optical fiber optical coherence
tomography sensors into tools this is a
half millimeter surgical peck
and basically you can look beyond the
tool to know where the tissue layers are
and then if you scan that across a piece
of retina you can see things like here
is again I'm sorry the the projector is
hiding all the detail but this is a
piece of a pig retina and the really
neat thing to us is we need to go back
and do histology but this we think is
the internal limiting membrane of that I
which is something you just really
cannot see other ways the other thing is
the particular form of OCT we're using a
also you get the tissue color spectrum
as sort of a side effect and so you can
then process the color spectrum to tell
you this is again you can't really see
the detail this is a blood vessel and
this is sort of the optical equivalent
of b-mode ultrasound and this is ah the
blood in the in that vessel and and hear
what we've false-color did say here's
where we think oxygenated hemoglobin is
and again this can be very useful
functional information yes and then
scanning it in this case young yeah yeah
and we're using common common path for
you to main Oct if you know the
buzzwords so yeah that's what we're
doing here I this is largely the work of
Chen Kang and his students one of the
things that the problem is you want to
get this whole thing into a half
millimeter surgical tool with room for
the tool to they're also has images
bundles that are a little bit larger
really just a bit too large for most
optical retinal applications but they're
what you have is a bong
when you typically use a galvo to scan
the top end of the bundle it's pretty
cool though and then you know you can
take things and integrate them with the
robot here here is one example where
we're saying okay maintain a constant
distance from from the target surfaces
it's moving in this case this is done
with other slow robot we've also looked
at doing this with the micron and a sort
of a one degree of freedom version of
the micron that that we put together at
hopkins specifically for this keep your
distance application the other things
you can do things with if you have a
robotic devices you you take a little
scan then you then you here's a target
you come back to the target and then you
want to make a very precisely controlled
cut and so this is from a couple years
ago now and so here is the profile of
the tool and here is the target this is
something that we're very anxious to do
with the micron as soon as we get the
next generation of the micron i guess
can we can try it with current
generation too here's another of these
steady hand applications where we've
done a lot of them like that here was a
thing i did 12 years ago for using this
cooperative control to help evacuate a
hematoma here's but one of the things
you often want to do is create virtual
fixtures here this should be a video mr.
president why isn't that playing don't
have the video but basically what we're
doing here as an example is we are
machining a bone in the skull base to
get access for instance to skull base
structures for say dealing with acoustic
neuroma
now that the problem is this is very
very tedious because you you're trying
to machine often through some of the
hardest part of the body by the way if
you if you machine too far you could hit
the carotid or some other structure and
very quickly have a dead patient so the
idea is if you have the robot together
with some safety barriers that the robot
is enforcing you can you can machine
down get access and let the robot both
get rid of your hand term or support the
tool still rely on surgeon judgment but
the robot is enforcing these safety
barriers so this is another thing that
we've been very interested in doing now
in general I'm not sure I really have
time to talk much about say virtual
fixtures but there's if you wear your
robotics hat this whole question of how
do you do constrained control of a robot
either for these hand-over-hand robots
or autonomous or teleoperated robots are
very very interesting questions to us
and what we've done is a lot of work on
these virtual fixtures now if you look
at the evaluation loop again a lot of
the same technologies that we've had
often you want to run your validation I
even concurrently is real-time feedback
but you're just a couple other examples
on if you look at that prostate one of
the dirty secrets of prostate
brachytherapy is the seeds don't stay
where you thought you liked them so you
need to find out where they really want
if one is in a really bad place you I
don't need your kind of in trouble but
often if one isn't in a place where you
need one to be you can go back and put
another seed and so their various
imaging techniques this is mainly the
work of a meet Jane where we're using
x-rays to locate the seeds and a way
that is co-located Orko registered with
the ultra
images that can see the prostate I
mentioned the elastic graphi one of the
uses is if you're doing RF ablation you
stick a probe into a tumor and you
deposit energy basically to cook the
tumor from the inside out the standard
what method is a recipe you say okay a
stick in and I cooked for 10 minutes and
now I assume I've got a nice cooked meat
ball I go somewhere else and cook some
more and and and occasionally you might
have a little thermometer you meat
thermometer up maybe on the surface of
the liver sometimes in the probe you
know cook until it gets to 98 degrees C
or something what you'd really like to
do is cook until done so if you use this
elastic rafi what you can do is you can
actually see the tissue change its
mechanical property as you cook it it's
like testing meat on the grill and so
here are a couple cases clinical cases a
free hand palpated ultrasound free hand
placing needles for RF ablation and
these are what b-mode images look like
again the projector is not so good but
you can see the the lesions a lot better
in elasticity images and here are the
corresponding ct's postoperative CTS one
other example this is something not yet
published though we have a paper where
we'll find out in a few days if it was
accepted we're here the application is
injecting cement into a bone that is at
risk of fracture from osteoporosis and
the larger project the goal is to do the
planning navigate an injection device
into the right place in in in the bone
in this case something near the femoral
neck inject cement to strengthen the
bone according to what our finite
element analysis is look at the
bone that the cement that we've actually
injected and compute our notion of the
strength of the bone and then if need be
do some more so you you don't you want
what you have to assess this is a fairly
sparse set of x-rays for it for x-rays
would be typical that you might take and
so what you want to do this is ground
truth this is the cement ball that based
on what we got from a CT scan
postoperatively we want an inter
operative estimate of that the standard
way would be you'd look at the x-rays
outline the the lesion in each x-ray
intersect them from the silhouettes and
you get you get something that looks
like that what my student Blake Lucas
did is it's a very very clever method
he's able to get reconstructions that
are much more like that and so this is
something that we're really very happy
but it's a 3d level-set segmentation
done from a small set of 2d projection
images I let me just say a word about
the process loop and then I'll talk I'm
the some point I'm going to run out of
people's patience i'm already driving
you out I apologize as I said these
systems tend to create lots of data and
you'd like to be able to process that
data to correlate what you did with the
outcome and maybe that will tell you
what to do better next time you know
airplanes are safe because they have
flight data recorders this is an area of
quite a lot of interest to the computer
science part of medical robotics and let
me just talk very briefly on one of our
pilot projects are again we had some
small donor funding and now we've got
also funding from Phillips is to take
previous radiation therapy
plans where we we know the plan and the
gate elephant will take a day and use
that database to help us more
efficiently improve the planning of a
new patient and the way this planning
works today is you know here's a tumor
and things that you don't want to kill
this is the spinal cord and these are
the sella very glands a person called it
a semester sits down he spent he
specifies a really hairy optimization
problem to a computer very large
nonlinear least squares problem the
computer makes its best solution for the
settings of the machine you simulate the
settings the machine you look at the
dose that you got the decemberists then
says oh hell or something stronger comes
up with a new plan a new specifies a new
problem and you go around this loop
until you get something that the
clinician will accept the problem is you
don't know really when to stop you have
no idea of how good a job you can do and
it can take all day till you get a plan
that is acceptable so what we want to do
is we want to look at just from the
geometry of these organs the tumor and
the root and it's it's the euclidean
distance trends it's the something we
call the overlap volume histogram you
can kind of think of it as the
configuration space of the tumor
relative to these other structures we
want to get an index into a database of
all the patients we've treated before
now for quality control purposes what
you do is you say okay from my database
is there a patient who we think should
be at least as hard to plan who had a
better better result in terms of sparing
these these other structures and if
they're so then why don't you use that
that optimization problem on this
patient's geometry
turns out if you do that in a
significant fraction of cases maybe
twenty percent you can significantly
reduce the dose two critical surrounding
structures while preserving the same
goes to the tumor the other thing you
can do is if you start with sort of the
best similar patient and try to optimize
from there you can often get a plan
that's about as good as a skilled osama
triste will eventually get to but it
with much fewer tries going through that
loop so we currently have a controlled
retrospective blinded study of this
right now on our radiation therapy
patient so we we are hoping as soon as
we have that done we're going to put
this in the clinical workflow and we're
very excited by it I haven't gotten to
talk about Greg Hagar's work where what
he's doing is he's analyzing surgical
motions in trying to segment surgical
motions automatically and then you can
use this for various purposes such as
skele cess Minh like here this is the
traces of not so good surgeon and an
expert surgeon doing suturing and you
can use it for skill assessment you can
also use it to compare different
instances of the same task this is
something that I've always been really
impressed by on here what we have are
two expert surgeons working side-by-side
doing the same suturing and what happens
is if one video gets ahead of the other
the other one will pause to catch up so
this looks very impressive and it's
keeping track the thing that blew my
mind was this result here is a novice
and an expert
and so okay so the expert the computer
knows it's suturing the thing that
amazes me ok now the experts waiting
that the nut but but but at some point
this guy is going to start moving again
and what that will tell you is the
computer realizes the novice finally got
through that stage now you can clearly
use this see your golf swing that's the
Takeo isn't here but see your golf
spring you could use this for training
you can use it for skill assessment
other things maybe you can use something
like this where is to detect the phase
of the procedure so you can adjust the
help you're giving to the surgeon but
more automatically here I did tong and
this is another result from nick lepard
oi who is another of Greg Hagar's post
dogs we're here what's happening is the
robot knows what some of the motions of
this suturing test Kirby and now the
robot is doing that automatically and so
now we're back to manual operation and
now we're doing the next stitch and
again this is one of the directions were
headed I did talk myself out of time
just I'll get done I promise before five
and here is another set of results one
of the things I want to come back to is
this notion of a system vision we got a
lot of technology we can do imaging lean
do modeling we can do robots we can do
it all but the key thing is they have to
work together and I really blend further
I i believe over time you really need
all of the stuff that goes on in the
operating room to interface to the much
broader information infrastructure the
hospital in the same way that you want
your machine tools to interact with in
lithography systems to interact with the
information infrastructure that of the
factory
so I think again coming back to this
system view we have here the robots the
the imagers all of this stuff but in a
way that the robot of the future is
really going to be the operating and I
think again this comes back we need to
take a systems view for us one of the
things that we wanted was the ability to
put all of our different research I
operate interoperable in a very highly
modular fashion and so actually working
with intuitive we've put all of our
robotics and medical robotics software
into a an open source framework that we
call saw surgical assistant workstation
that we've used for a lot of the the
things we talked about earlier turns out
there's a plug in for the da Vinci so
that if you develop something say on one
of our robots we can try it out on the
da Vinci and vice versa so I just I
think I don't know if you do this kind
of thing at CMU but I think it is a very
important element of any any large
research program and we are really
looking for additional collaborators to
use and develop this environment here's
another environment based on some on
some of that c-arm x-ray stuff that
again uses these same components for a
Surgical navigation application and that
was supposed to have been hidden so
anyhow there you have it folks there's
the loop generally I think it's really
good in doing even sort of basic
research to always know why someone
would care what you're going to do as
well as what you're going to
and so this is a common model for
everything we do and this I promise is
my last slide I've ended every talk for
about 15 years with this slide and what
it comes back to is why I changed
careers I honestly believe that this
combination of technology and
information systems and humans working
together is can profoundly change
interventional medicine in just the same
way that we did with factories it can do
it by providing better capabilities
things that you can't do otherwise you
can make procedures more accurate you
can make them safer you can produce
better outcomes and that combination is
actually even if the machine is
expensive can be more cost effective
because you get it right the first time
and finally ultimately it's about the
patient what the most rewarding times
I've had haven't been getting an award
occasionally or getting a big NIH grant
or showing off to the Dean or the big
wood it's been going around with a
clinician seeing a patient wear
something some piece of our technology
has been used to help treat that patient
and
that is I think we should remember the
ultimate payoff and what we're doing
we're in business to change society to
help society and fortunately you can
have fun you can get tenure you can do
all of these other things along the way
but it's ultimately about that and this
is hard work the an area like medical
robotics you have to know something
about medicine as well as something
about robotics so there's more to learn
it takes longer and it's really hard but
it is fun and very rewarding and that's
sort of the last I wanted to say unless
you want to know about trout fishing and
I do apologize for rattling on too long
I I I realized I should have been more
selective about or more quick at some of
the earlier things I talked about I'm
happy to 10 people here answer questions
whatever none okay
okay yeah holywell community it's it's
like sailing it's all vocabulary some to
some extent that's true if you if you
learn a little medicine I mean what it
helps if you know that oughta me means
something about cutting so yeah I a lot
of it is patient one of the key lessons
is you got to listen to what the
clinician wants and understand his
application don't let the clinician tell
you the solution although if you will
often try then try to understand why he
thinks that's what he wants but I think
for us maybe the key reward has been
every is is you have to meet actually
frequently with these guys who are very
busy and you have to not make them just
impatient but a for instance for the eye
surgery work it's really cool we we have
to attending eye surgeons meet with us
every friday afternoon and they're not
shy about telling us what's wrong and
they'll try something and and you know
we'll talk about maybe another proposal
we want to do and
and try things so basically just stick
at it one of the things I kind of wish
I'd done is had the time to go to see or
two of medical school which I when I was
young I wanted to do robotics and I
thought these pre-meds people were
really not people I wanted to spend time
with but but that that's sort of the key
less and it just takes a lot of time in
very frequent communication with
surgeons these guys are very hands-on
you can describe something and they just
won't understand it if you show them
something they're very tactile people
and so if you show them something that
will put it in their hands and and maybe
get the idea and tell you what's wrong
with the thing that you just put in
their hands did I know candy do you have
the same experiences and how I had
delete but I know he did so okay yeah
what do you see is the future of
autonomy robotic surgery I mean do
surgeons really look for more autonomous
robots or just robust care and health
problems well I know what look the first
law of robotics of medical robotics is
the surgeon is in control um personally
I think this whole autonomy versus
teleoperation i'm not sure what it means
you're not going to have a robot walk
into the operating room and say ah
here's the patient and start carving i
really see it more as trading off one
for another one thing that has driven me
up off about 15 years or people who
assert that a hand over hand controlled
robot for doing something like a moan
machining procedure is safer somehow
then a robot like Robo doc where the
surgeon has verified the alignment and
now the robot is doing something with
some constrained autonomy and to cut a
shape the big danger in that kind of
robot is not the robots going to learn a
mock it's going to do exact
right thing in the wrong place so the
real safety issue is really a
registration issue i think that surgeons
are very conservative and any time you
want to give them a new piece of
technology other than a few crazy early
adopters what you've actually got to
show is somehow it makes their life
easier and convince them that it won't
do what they don't want to do I Matt you
may be older you remembered well yeah it
stands for do what I mean which there
was a guy who sort of our generation of
grad student and that was his thesis was
sort of semi automatic programming do
what I mean it would fix your bug and
here the it was more nevah don't do what
I don't mean and that's very important
so that's sort of a medic question I I
where I I think that you can have a kind
of autonomy I see actually getting in is
you get to someplace and now there's
some servo-controlled thing that would
take over and do some small motion and
then go on I mean this automatically
pulling the suture I'm not sure is such
an example if you had maybe a third hand
robot actually trying to do something
like some simple retraction and
controlling the retraction that would be
probably a more plausible scenario but
again this is all sort of speculation
it's hard to say clearly FDA it's a big
issue with them and I think you have to
be kind of careful in defining what it
manage each piece of autonomy is
providing
anything else
well again I maybe I should just close
again by thanking you folks for sticking
for a long-winded talk professors as we
get older talk longer and also again
thank many colleagues I talked about the
work again of many many colleagues and
there were many many funding agencies
both the US government and and in some
cases private companies and foundations
who helped support that work and we do
look forward to collaboration with you
all in the future