Semi-automated mid-turbinate swab sampling using a six degrees of freedom collaborative robot and cameras

ABSTRACT


INTRODUCTION
With the rise of novel variants, coronavirus disease 2019 (COVID-19) may stay for many years. Telenti et al. [1] stated that the mass deployment of vaccinations may signal the end of the COVID-19 pandemic, but it does not indicate the end of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus. The article suggests that one of the future scenarios is transitioning to a seasonal illness and being categorized as an endemic disease. Also, healthcare workers have one of the highest rates of infections and represent 3.9% of all coronavirus-related infections as surveyed on May 2020 [2]. For this reason, several articles discuss the importance and benefits of using robotic technology to alleviate the healthcare team's daily burdens [3], [4]. For example, Dheeraj et al. [5] presented a design concept and implementation of a lightweight six degrees of freedom (6-DOF) robot manipulator for handling and sorting vaccines in a cold environment. Robotic applications for COVID-19 swabbing tests for the nasopharyngeal [6] and oropharyngeal [7] areas showed that those automated applications minimized the rate of infections among healthcare workers and patients. Other related benefits include faster swab times, a less frightening patient experience, and higher quality swab tests than manual testing. Kawasaki Heavy Industries [8]  robotic system for post-sampling processes like nucleic acid extraction and reagent preparation. This will enable continuous 24-hour operation of testing centers, reduce infection risk of medical professionals, and alleviate medical professional shortages. In all countries that administer testing for COVID-19, tests are done by inserting the swab into the inner nasal area or administered orally. Wang et al. [9] compared the detection rate for SARS-CoV-2 between nasopharyngeal and oropharyngeal swab samples and it determined that nasal collection had a higher chance of detecting positive SARS-CoV-2 samples. Font et al. [10] concluded that there is a higher quality difference between the two forms of sampling. 100% of the positive samples were taken from mucosa on the nasopharynx, and 73.1% of those patients' oral samples appeared negative. Therefore, a robotic application for the nasal COVID-19 swab tests is more prominent than one that administers an oral COVID-19 test. Several robots were/are currently in development for nasal COVID-19 tests. Begum [6] and Wang et al. [11] introduced nasopharyngeal swabbing robotic arms that promote distance between healthcare workers and minimize the chance of infection. The robotic arm [6] is a self-administered 2 DOF robot for COVID testing that was reviewed to be safer, faster, and less frightening than manual swabbing. However, the technical drawback of using this robot is that the swab is inserted horizontally and activated by lifting the chin. Each patient has a different nasal passage to the nasopharyngeal, which connects to the throat. By raising the chin to start the apparatus, the inserting angle of the swab will be towards the top of the nasal concha area where sensitive tissues are located or, more extremely, the superior conchae, which serves to protect the olfactory bulb as shown in Figure 1. Wang et al. [11] designed an end effector equipped with an optoelectronic force sensor which serves as a feedback sensor once the swab hits the back wall of the nasopharyngeal area. However, the robotic arm is remotely controlled through a mobile app, and the trajectory path of the swab is dependent on the operator's experience, which may be slower and more inaccurate compared to manual swabbing. Therefore, a 6-DOF robotic arm and a visual sensor are crucial to tackling these problems. Although a limitation to using stereovision cameras is the effect of varying illumination on the generated images, the lighting condition of a COVID testing site is typically controlled. Huang et al. [12] introduced an alignment method using cameras to align a peg with a static hole which in this proposed system would be the swab and the patient's nose. As a result, a 6-DOF robotic manipulator with smart cameras can detect the tip of the nose and align the swab with the optimal nasal pathway. Brain Navi's swab robot [13] is the most advanced and developed 6-DOF robotic arm for nasal swabbing. It uses a 3D camera and AI facial recognition to detect the accurate nasopharynx position and locates the upper nasopharyngeal area. However, performing the polymerase chain reaction (PCR) test, inserting the sample into a tube, and sterilizing for the next patient takes about 5 minutes. Lockwood and Crawford [14] stated that it presented a terrifying and uncomfortable patient experience. At the same time, all the COVID-19 swabbing robots described above only perform nasopharyngeal swab tests. Although nasopharyngeal swab tests have not been ruled out, mid-turbinal swab tests are widely used and favored as the standard PCR test [15], [16].

SYSTEM
The 6-DOF robotic manipulator used in this paper is the Doosan Robot M1509 as shown in Figure 2, whose maximum workspace radius is 0.9 m with travel speeds as high as 1 (one) m/s. The visual sensor is screwed onto a 3D-printed plate, and a cotton swab is inserted into the hole with a tight interference fit. The designed end-effector plate shares the exact center with the tool flange, so the swab aligns with the central axis and passes through the tool's origin. Furthermore, the visual sensor used for the frontal vision of the patient's nose is a Pixy2 CMUcam5. The system was tested using a phantom head. A secondary visual sensor fixed to the side estimates the head angle by detecting the color pattern from a color bar whose top edge is aligned with the earlobe and the tip of the nose. The color bar and fixed Pixy2 are held up by a clamp apparatus comprising links and nuts for the desired setup.

METHODOLOGIES 3.1. Tool center position
The swab's tip determines the tool center position as shown in Figure 3, which front view is in Figure 3(a). Since the swab reached the tool flange, the tool center position (TCP) was set to 10 cm in the Zdirection as depicted in Figure 3(b). Then, it is adjusted to allow movement about the swab end.

Color code generation
It is necessary to register the color pattern on the color bar. In PixyMon software, the Pixy2 camera is trained to recognize the three colors by setting three color codes (also called CC signatures). The operator selects an area of the target color to declare the CC signatures. The embedded software will highlight the pattern and display an angle respective to the horizontal axis.  The cockpit of the robotic manipulator, shown in Figure 7, allows the operator to move the end effector manually. Button 1 of the cockpit allows movement in any direction that force is applied. Pressing button 2 fixes the TCP and only allows movement about the swab end as shown in Figure 7. Then, the teach pendant can fix the 3D coordinates of the tip of the swab and change the angle of the end effector. So, the user can input the angle registered by the visual sensor in the teach pendant. After positioning the end effector with the target angle, the operator may get the robot ready to perform the insertion. The teach pendant allows the operator to create a simple program to insert the swab in a straight line and rotate the end effector to collect a swab sample.

.1. Alignment process
In a collaborative environment between the operator and the robotic manipulator, a semi-automatic approach is proposed to collect the mid-turbinate swab sample from a patient. The operator places the tip of the swab at the entrance of the patient's nose by pressing button 1 (one) and freely moving the end effector of the 6-DOF robotic arm, which can be seen in Figure 8 via the display of the Pixy 2 camera on the end effector, the operator confirms that the yaw angle of the swab will not be approaching the septum or sidewalls of the nose. Next, the operator aligns the leading edge of the color bar from the patient's nose tip to their ear lobe. The PixyMon software detects the color bar and displays the head angle φ, shown in Figure 9. Because the head angle φ was calculated with respect to the horizontal axis of the external Pixy 2 camera, the target tool angle B can be calculated by adding φ to 90°. Note that Tool angle B on the teach pendant reads 90° when the end effector and swab are in a horizontal position. Then, the operator presses button 2 (two) from the cockpit and manipulates the end effector vertically until tool angle B as shown in Figure 10 reaches the target angle. Remind that button 2 on the cockpit is programmed to allow the operator to manipulate the end effector while keeping the TCP's endpoint or swab end fixed. As a result, the operator could align the swab's pitch angle with the patient's head angle, as shown in Figure 11. For the safety of the experiment patient, the insertion process will be administered only to the phantom head, and the alignment process is also repeated for the phantom head.

Insertion and sampling process
Finally, the operator runs the insertion program in the teach pendant to collect the mid-turbinate sample. As shown in Figure 12, the end effector travels 4 cm in the longitudinal direction of the swab, rotates the 6 th joint, retreats 4 cm, and rotates back to the initial position. Note that a target depth between 4 and 8 cm can be programmed for a better-quality sample. The maximum limit of the target depth is set to 8 cm due to the length of the swab and the distance that the swab can insert before the patient's nose touches the end effector.

SUMMARY
This paper introduces a semi-automatic approach to collecting COVID-19 nasal samples in the midturbinate region. For the experiment, the 6-DOF robotic manipulator acts as a collaborative robot or robot with an operator to collect the swab specimen. The addition of two Pixy 2 cameras, one on-end effector and one fixed for side view, assists the operator in determining the proper yaw and pitch angles of the swab. The time of alignment process resulted in between 1-2 minutes for the live patient and the phantom head patient. The alignment process varies depending on the patient's head shape, the operator's ability to handle the robotic arm, and the levelness of the fixed Pixy 2 camera. The level adjustment of the fixed camera ensures that it is parallel to the horizontal plane and is obtained using a level tool. The sampling process on the phantom head patient resulted in 1 minute, making the total experiment about 3 minutes. Although the entire experimental process is longer than the sampling process of a manual swab test, which may be administered in less than 1 minute, it proves the effectiveness and feasibility of implementing robotic applications for COVID-19 swab sampling. Since this experiment followed several manual procedures, a fully automated procedure can bring the overall sampling time closer to that of a manual swab. However, it was concerned with the importance of getting the procedural time closer to that of a manual swab test; it is also important to highlight other benefits of adopting robotic applications for COVID-19 swab sampling. Implementing a 6-DOF robotic manipulator with smart visual sensors will conserve labor, promote safety between patients and healthcare workers, minimize pain or discomfort, and permit COVID-19 swab tests to locations lacking healthcare workers.

DISCUSSION AND FUTURE WORK
Future development as an add-on to already existing features is considered to advance the algorithm and more accurate sampling. By implementing AI-based smart cameras, the alignment process can be done manually. First, train the end effector camera to detect the nasal entrance centroid, mid-nasal ridge, and swab. This allows the robotic arm to move the swab tip to the nasal entrance centroid and adjust the yaw angle to parallel the mid-nasal spine. Second, train the fixed smart camera to recognize the tip of the nose and the ear lobe of the patient. The head angle can be obtained by visualizing a line between these two points and measuring with respect to the horizontal axis. As a result, the alignment process is automated using AI smart cameras. For the safety of patients, a force-feedback sensor may be implemented in the end effector to receive forces pushing back on the swab if the swab obstructs a nasal wall. Pain tolerance varies between individuals, so data on forces that patients can hand in the nasal area has to be collected. With this data, we can create a pain threshold that will alert the robotic arm to retreat and adjust its angle. Lastly, a microendoscope with cotton swab attachments is proposed to navigate the interior of the nasal pathway. This will permit micro-adjustments of the robotic arm when entering the nose and collecting the specimen. All of the above would help the transition to a fully automated procedure.