Chapter 2: Basic concepts of image analysis
In this class, we will discuss the challenges and fundamental problems of medical image analysis.
Furthermore, we will talk about the basic concepts: 3D images, intensity transformations (histogram,
window/level, filtering) and the spatial transformations (coordinate system, coordinate
transformations, image resampling and image interpolation).
Medical image analysis
What is medical imaging analysis? We acquire medical imaging data from the patient and simulate
these imaging data with each other to get more information about lesions, for example. Radiologists
will inspect these images and may diagnose. We are engineers, we are interested in all the other
things we can do with these images than only interpretation and diagnosing. Image analysis is
performing measurements on the images, for example: detection of the position of the image; or
detection of the lesion and where this lesion is located. When we have medical imaging data from
one patient, from different modalities, there is complementary information in these images that we
need. For example, a patient has a primary tumor in the lung with metastasis in the rest of the body.
We do have medical imaging data of CT-scans, PET-scan and MRI. On the PET-scan, we can show the
metastasis in the rest of the body: we see the primary lesion (in the lung) and the additional
hotspots. A big problem with PET-scans is that you don’t have enough anatomical details to tell
where the lesion is located in the body. However, when we combine the PET-scan and the CT-image,
we will have the right information for diagnosis. Now, in the hospital, we do have a PET-CT machine
which will create these overlaying images immediately.
Another combination is CT and MRI. On CT, a lesion is detected, but it is very hard to see. On the
MRI, the lesion is much more distinct. Additionally, on the MR-image, you see the different types of
soft tissue much better than on the CT-image. For radiotherapy, you need CT-data. With this
information, you know how the radiation will propagate through the brain. The CT-data will allow
you to compute the dose distribution. In conclusion, the information from a CT-scan for radiotherapy
is available to guide you. Additionally, you need to information from the MRI. Because in these
images, you can see the different tissues, because the anatomical regions are clearer than on CT-
images. For the optimal information, you match the MRI and the CT-images. We have software to
perform this (there is no MRI-CT scanner, unfortunately, because this is much more challenging). MRI
(magnetic resonance) is not compatible with the operating procedures of CT (PET and CT are
compatible!).
In order to define a lesion, we need to know what normal brain tissue and the skull look like. This is
due to segmentation. During radiotherapy, you need to tell the computer which part of the brain we
want to radiate (lesions) and which part we don’t want to radiate (healthy tissue). In order to know
this, you need to know the images, but you also need measurements! The boundaries of different
body part regions are very random. Some measurements might be used in all image analyses, for
example the volume measurement.
When a patient is treated for the lesion, a follow-up will be performed and the patient will undergo
another MRI. We need to compare those images (before treatment versus follow-up) in order to
conclude if the treatment works. The more critical the application, the more interest there is in
clinical medical image analysis!
There is hope that image analysis problems (image/background ratio, structure region segmentation,
etc.) might be fixed with Artificial Intelligence (not manually)! However, we are now waiting for
further regulations to use these tools.
Why do we perform these analyses? We want to extract clinically relevant information (the doctors
define what is clinically relevant, not the engineers or the researchers!) to help the doctors with the
Pagina 1 van 16
,diagnosis or the treatment; or we want to use this in (biomedical) research. In addition, different
researchers are more expert on these tools than only the radiologists. In research, there is more
room to interact with developers and researchers of clinical medical image analysis.
Challenges
Complex data: special nature of medical images…
Complex scenes: anatomical objects, natural variability, pathology, …
Complex applications: clinical requirements defined by medical experts…
Complex validation: lack of in vivo ground truth…
Imaging modalities
The CT-scanner is based on an attenuation of X-rays. It provides high spatial resolution (sub-mm);
dose issues; visualization of dense tissues (for example bone, masses, calcified plaques, etc.). MRI is
based on the magnetic properties of hydrogen (dipole). We see a large soft tissue contrast. The
resolution is about 1 mm, restricted by organ motion and clinical constraints on scan time. PET is
based on radio-actively labeled tracer molecules (for example FDG). There can be local concentration
of the metabolized tracer (for example glucose uptake). It has a high sensitivity, but a limited
resolution (~ 3 mm). This is mostly used for “molecular imaging”. US is acoustic wave propagation
and reflection. It has a high temporal resolution where we can image real time movement of organs
(for example in cardiology). The medical imaging technology evolves quickly, leading to new clinical
applications, for example multi-slice CT, cone beam CT, dual energy CT, MRI-DTI, high field strength
MRI, PET-CT, PET-MRI, etc.. Medical imaging and imaging analysis research are application driven!
Challenges – Complex data
The medical images provide information about the interior of the body. We cannot simply change
the quality of the images by using other light; by taking other images; or by changing the camera. We
just need to work with the images we have. This is sometimes very complex! There are different
medical imaging modalities with different data:
Projection images (2D, for example RX) might have overlying tissues
Tomographic images (3D, for example CT, MRI, US) gives us more information, but complex
topology; intrinsic limitations on resolution and contrast; and imaging artifacts (for example,
due to motion, reconstruction algorithm, etc.)
Dynamic image sequences (4D, for example US, cardiac MRI)
Most of the time, we have 3D information and much more, which makes it very complex. You cannot
say to the patient: “Stay in the scanner some time longer, because the images are not perfect”.
Additionally, you cannot say to the engineer or the doctor that the images are not perfect and that
we want new images to work with. For example, a CT-scan doesn’t take long (in a few seconds, you
can scan the thorax and abdomen and you will have a thousand slides). In MR and PET, this is not
possible! When taking a PET-scan, it will take minutes and the patient needs to breathe. You will see
this as a blurred image. That is why dynamic image sequences can be used: we will have 4D data
where we also include time. This is used, for example, by cardiac MRI: We make a snap shot while
the heart is actually moving and we try to visualize how this change over time. Be aware that this
kind of analysis includes even more data! Serial imaging gives us information about pre/post
therapy, pre/post contrast, follow-up scans. So, imaging modalities based on different physical
properties provide different kinds of information, with different resolution and contrast, requiring a
different kind of analysis.
Challenges – Complex scenes
In medical imaging analysis, we are not dealing with man-made objects, these are all anatomical
objects with a complex 3D shape (for example, the brain surface or the vascular tree). These
anatomical objects are variable over time:
Pagina 2 van 16
, Motion: periodic (for example the heart beat or breathing) or spurious (for example the
bowel movement)
Deformations of soft tissues
Pre- and post-intervention
We see a significant biological variability between subjects (there is a lot of variation). The pathology
is a deviation from normality. The image appearance is ambiguous and prior knowledge (= models) is
required. You need the information about the anatomy, but also about the different modalities (how
they work, what their outcome is, etc.). Current Artificial Intelligence (AI) algorithms do have this
information and they can tell us step by step what they are doing.
Challenges – Complex applications
For the diagnosis, for example in oncology, fusion of images of different modalities is already a
challenge. You need to think about the detection and localization which might change over time.
Additionally, the volumetry can change over time. For research (for example neurosciences) we need
the volumetry (volume) of the hippocampus, for example, for morphometry, normal biological
variation or abnormality detection. It is very complex to highlight the volume manually. We can
detect lesions (larger, but also smaller ones!) on MR-images. However, the MR-images are different,
e.g. every image is different, so not all protocols are working in the same way in the different images.
In therapy (for example radiotherapy and/or surgery), we need to know the structures which we
don’t want to radiate for target definition, therapy planning and/or follow-up. CT is a very important
modality for radiotherapy.
Challenges – Complex validation
In a real simple scenario, you cannot open the patient and search for the lesion in the brain and
check if the modalities were correct. We just don’t know! So, there is no direct access to the scene
(the patients’ interior). Due to this problem, we create phantoms to validate by simulations
(hardware or software phantoms). So, we create a simulation of the patient with a simulated lesion
and we check for the different images from different modalities and we check the results. Now, we
will create a ground truth ourselves. The software phantoms are more flexible. You need to start
from a model of a brain that you might derive from a previous image of the patient in order to build a
phantom on your computer. In this image, you can draw the lesion and we can simulate the imaging
process (so not using a physical scanner, but a scanner on the computer). The result is simulated
images with the artefact (lesion) you created yourself. You can use the new algorithm to see what
the results are. With hardware phantoms, you can create a box. However, building 10 boxes for one
patient is very hard. So, software phantoms should be the better option. Is this now realistic? What is
the clinical relevance? We can show these phantoms to our clinician and ask for their opinion!
However, you need to ask more than one clinician, because they will have other interpretations, so
also check for this variation too. There will be inter-observer variation and you need to show this in
order that you know this! So, is there consistency with the manual observer? Is there also intra-
observer agreement present? Each clinician is trained on the data from learning from their experts.
They all apply different algorithms on different data. You would not be able to tell which algorithm
will work on all of the data! You can make a challenge: You have data available from different centers
(hospitals for example) and this is your training data. You can use your algorithms on this data and
the hospitals need to send their results about this data. You don’t analyze the performance of the
data itself, you validate the outcome. In the end, you will have a leaderboard which algorithm has
scored first and second on different parameters. The benefit of this is that we have a plain field of the
algorithms. However, there is a huge drawback, because people are smart! If you want to have
citations of your paper, you might do the challenge and you might be tempered to choose the
algorithms of the center of the reviewers and the algorithm would be high in the leaderboard. So, the
protocols are very interesting! So, with learning- or training-based methods, you can rely on the data
available and start to learn with your own data if you have the right algorithm. It is very important
that you know how this data is trained and learned; and if you have in-house experts to work with
Pagina 3 van 16
