[CVPR 2019 workshop] Wenjin Wang: Camera Based Physiological Measurement

CVPR 2019

This episode is a live recording of our interview with Wenjin Wang at the CVPR 2019 conference. Wenjin Wang is a Scientist at Philips and an assistant professor at Eindhoven University of Technology. Wang's research primarily focuses on computer vision and machine learning in the medical field. He received his PhD at Eindhoven University of Technology in 2017. Wang presented a tutorial titled "Camera Based Physiological Measurement" at CVPR 2019. His presentation focused on computer vision for physiological measurements. During the interview, he shared key applications of the workshop he hosted at the conference and what it is like to work in both academia and in the industry. / Full Interview Transcripts / Wenli: We have Wenjin Wang here with us. He is a research scientist at Philips Research, and he's also an assistant professor at Eindhoven University of Technology. He's also an organizer of the tutorial “The Camera Based Physiological Measurement”. Can you tell us a little more about the tutorial that you're hosting? Wenjin Wang: The main focus is on the camera-based physiological measurements, which means that we use the camera to measure the vital signs from the human face to in the body. So this is very different from the current existing camera based technologies which only measure the outside features like how you look, like the features outside of your skin. Now we look inside of the skin, deeper into the human body. The key idea is that we use the RGB camera to measure the blood volume changes beneath the skin, so that we can derive multiple different vital signs like heart rate variability, respiration rate and blood oxygen saturation, as well as pulse transit time which is related to the blood pressure. These vital signs can be used for diagnosis, like medical care, health care, and also for some entertainment. We have collaborations with other companies in landing those technologies for gaming, like VR/AR or other computer games. This can also be used for sports and fitness exercise. For example, what I did in my previous PhD research was collaborate with a company working on the fitness equipment. We put the camera in front of a treadmill. When you are running on the treadmill, it can automatically extract your heart rates. Wenli: How do you achieve this with the RGB camera? Wenjin Wang: The principle of RGB camera is very similar to the finger oximeter. The finger oximeter or the smartwatch, like Apple Watch, can also measure your heart rate. They’re using green wavelength to measure your skin reflections, and from the skin reflections, they see the blood volume changes. The camera is similar to that, but can provide additional features. For the finger oximeter or the smartwatch, it can only use the green channel, but the RGB camera can use multiple channels, like RGB channels. Different from the content sensor on the smartwatch, the camera has the spatial redundancy, like multi-site measurement and can see your whole face, and use the characteristics that can be more robust to motion. Wenli: How accurate is the measurement that you get right now? Wenjin Wang: It depends on the use case because this technology can be landed in different scenarios, like sleep monitoring. For example, a person is sleeping in bed and we use a camera to measure his respiration rate or heart rate during sleep. The sleep is relatively easy, because there's no motion, no distortions, and the accuracy will be relatively high. In sports or fitness exercise, because you have lots of motions and body movements, then we'll have to be robust to that, and then accuracy will be lower. But we're working on that to improve the robustness. All the accuracy on the benchmarks have been published in our previous papers; if you're interested, you can also check on that. Wenli: Awesome. For the tutorial that you organized, how did you select the co-organizers that help you organize the entire thing? Wenjien Wang: The co-organizers are actually my colleagues in the university. We have been working together for a long time, we’re all familiar with each other’s work. My situation is that I'm often in the industry, at Philips Research, doing the industry-oriented, application-oriented research, but also at the same time, I am an assistant professor at Eindhoven University of Technology and also get involved in some pioneer research work in academia. The combination of those is actually a good thing because in the company, we focus on product; in the university, we focus on some fundamental research. The combination of that will give us more opportunities. Of course my colleagues are from the university, so actually we built up a very good bridge there. Wenli: As you mentioned that you spent half of your time in industry and half of your time in academic research. Do you also teach students at the university? Wenjin Wang: Not exactly teaching, but I do the supervision. For example, I have a couple of PhD students. Wenli: How do you like that? Is it like you see the long-term benefits of academia and also the shorter missions, and you can see the business side application? Wenjin Wang: Yes, exactly because myself is also in that mood. I did my PhD in combination between Phillips Research and Eindhoven University of Technology, so I get the experience on both sides. That’s actually a good thing because in the industry, it will give you exact goals like what you're doing is meaningful, it's not like the pure lab work that you did only to pursue the accuracy without any applications. If they have a very clear goal in industry, then the work will be meaningful, it will be practical and can be used in the future. That will give you a very good target or goal. And then at school, the professors will give you the supervision, give you the guidance to help you keep up with the state-of-the-art techniques in academia. Wenli: Do you notice any gap between these two? Wenjin Wang: Yes, there are still some gaps that we are improving, like communications because for the two sides, the final goal is the same, which is to push the technology forward, but on some small goals, the practical goals are a little bit different. For example, companies are more focused on the product, they do not really care whether it's really the best of this year. But it has to be low-cost, has some practical issues. And the school is more focused on the publication. Wenli: Is there any comparison on solving the real-world problems? Is the industry far ahead of academia? Or is it the other way around? Wenjin Wang: I think to understand and solve the problems, industry is doing better than academia. Because in academia, sometimes if your goal is just publications, you may improve the accuracy, for example from 98% to 98.5%, but that's not meaningful to the industry. But many researchers are still working on that, just for publications. I don't think that's a good idea. Especially for us who are doing the technologies, technologies must land in the market to serve the people, to improve people's lives. That's the final goal. If all the technologies only stay in the publications, it is not useful. Wenli: What are some of the business applications that your research has been landing in at Philips Research? Wenjin Wang: For our technology, we have many different application scenarios. For example, because what we're basically using is a camera to measure the physiological state of a person, like the vital signs. This can be used for, for example, the health monitoring at the hospital, for the ICU, or CCU (Cardiac/Coronary Care Unit), or NICU, which is Neonatal Care Units, for newborn baby. When we use the camera to monitor the babies, monitor the patients in the hospital to check the vital signs, to check the activities, these are all done in a non-contact way, so there are non-contact sensors, which can do long-term continuous monitoring without any interference. That's actually the benefit point. The second application is, for example, we can also do that at home for elder care. The grandparents are a bit older, we need to put an eye on them. For example, now I'm in the US, I'm not staying with my parents or grandparents. If I tell them, what I can do is to put a camera there to do some smart monitoring at home, and also for baby care at home. It's another application. And the third application I mentioned to you before is fitness sports. What we currently have for the treadmill are the ECG sensors, you have to touch the sensor, or you have to wear the smartwatch or the polar belt to measure your vital signs. But now we just put the camera in front of the running people to automatically extract your vital signs, like skin temperatures, and then track the heart rates, it will give you a real-time visualization of your physiological state. From that, you can adjust your training strategy or training schedule to optimize your training. What we can also do is for automotive. In automotive, the camera technology has already been used to do the driving detection, alert detection, but that's all based on appearance features like eye tracking, face tracking, facial analysis. What we can do now is to add more physiological features with that, like adding the heart rate variability that is related to the stress level of the driver. For example, the driver is falling asleep, the heart rate variability will be very different. We can also do potential heart failure prevention like cardiac arrest. We’re also planning to have this product in a vital mirror at home. It's actually a mirror, when you stand in front of the mirror, behind mirror there's a camera, it can measure your skin temperatures, and shows the vital signs immediately on the mirror. You can do a spot check at home in the morning, that's very convenient. Another application we landed and already staffed in the market is to combine the camera-based monitoring together with the MRI scanning. For example, the challenge for MRI is that if you have respiration during the MRI scanning, it will cause motion artifacts or motion blur in MRI images. Now if we have a camera, we could measure your respiration signals, then we can use the respiration signals to trigger the MRI scanning. In this way, we can eliminate those motion artifacts. There are also many other applications we can dream up, and we also have the plan to do that. Wenli: That's really exciting that you already have so many business applications. Wenjin Wang: We have both business applications and also academic publications. We have a clear goal on what we're doing. That's actually a good thing for CVPR because if you check now, it's more for the computer vision image processing or the general deep learning technologies. On our side, it's a different angle to look into the problem, now you are looking inside of your skin and your body conditions. Wenli: For the tutorial that you're hosting, what can the junior research scientists get out of it? Wenjin Wang: For example, we can open up a new window for them. As I mentioned in our tutorial, just take an example, age detection is a very popular topic in CVPR. It just uses a camera to measure how old you are. What they are doing in conventional ways is just based on training, using deep learning to train a model based on lots of appearance features, like different persons with different ages. The other way to look into this problem is to use the physiological information like the cardiovascular age. We can check your arteries, like the blood vessels. From the signals, we can derive your arterial stiffness. If the person is young, the artery is very flexible and elastic. But for older people, the artery is usually quite stiff. This arterial stiffness will be related to the cardiovascular age, since the signal waveform between young people and old people are different. So we don't look at the outside feature, we look at inside physiological features to derive how old you are. That actually gives young people, the junior researchers another window to look into this problem from a different angle. Wenli: How about the senior researchers? What can you provide to them? Wenjin Wang: For senior researchers, in our tutorial, we also met a lot of professors or managers from companies, they’re already very experienced. It can be a very good complimentary for the current work. For example, there’s a company doing VR/AR glass and they are working mostly on the augmentation/virtual part. Now they’ve seen our presentation, they may want to use the camera to measure the physiological state of the subject, and then combine that into their glass to try to measure the stress level of a person, whether this person is nervous when playing a game, and use this information as a feedback loop to control the game. It can be like these combinations. Wenli: Amazing. We were talking about medical AI and how this medical CV can be landed in the real world. I was curious what are some of the business-wise challenges that you and your team might face in the future in Europe or in China? Wenjin Wang: There are many challenges actually. For example, regarding the technology, it will still be the robustness. For example, some applications require near-infrared monitoring. Near-infrared in full darkness is actually less robust than RGB, it’s due to some physiological reasons like hemoglobin absorption sensitivity. We need to improve the accuracy and robustness in that application, for example, in full darkness, in infrared, and also in some applications that involve lots of body motions like in fitness, sports exercise, lots of these emotional challenges. If we want to be robust to that, we also need to spend time and effort on that. This is technology-wise. The other point for landing in the market is the price, not all the cameras can be used to do this. Although they're all RGB cameras, they are in different levels with cheap ones and expensive ones, so the price is also one consideration - we need low-cost camera which can still achieve similar performance or accuracy. There are some other issues like GDPR, the General Data Protection Regulation. This is a law that has been announced in Europe for a few years. They do not allow us to analyze the images of the subject, especially the physiological information, because that's very private data. We have different solutions. For example, we published a paper about using the single element monitoring. Instead of using an image, now we use a single pixel to measure your face, without identifying the face, to solve this issue. We can also solve this issue from the law side. That's the third point. There are also some other challenges that we need to solve, we are still working on that. Wenli: I know that you're based in Europe. What are some differences that you see between doing research or working in Europe and in the US? Wenjin Wang: This is my fifth time coming to the US, but every time I come here is just for the conference. I do see some differences. For example, in the US, the atmosphere is more active, more people from universities and industries are working on that. The teams are bigger, there’re many different companies and research institutes involved in this area compared to that in Europe. Here the environment is more open, there’re more communications and more chances. Wenli: Thank you so much for coming here to share with us your experience and opinions. Wenjin Wang: Thank you.