Electronic Mirror

Sheryl Grace is no Bobby Orr. But growing up in northern Ohio—“a place where things froze,” she says—Grace, an associate professor of mechanical engineering at the Boston University College of Engineering (ENG), spent plenty of time on skates. So two years ago, when she saw her kids skating with less-than-NHL fluidity, she thought she could teach them a few things—how to get the perfect stride, how to bend the knees low for extra speed. What happened next will come as no surprise to anyone who has ever been a parent, or a kid for that matter: the kids got annoyed, and Grace got exasperated.

But Grace also left the ice that day with inspiration: an idea for an “electronic mirror,” a wearable body motion–tracking device that could help athletes get instant, objective feedback on their technique. Now, with support from a Hariri Institute Research Award, Grace is working to make the device, called a biokinematic data acquisition system. Joining Grace on the project are Richard West, a BU College of Arts & Sciences associate professor of computer science, and Cara Lewis, an associate professor at BU’s College of Health & Rehabilitation Sciences: Sargent College (SAR) and director of the SAR Human Adaptation Laboratory. Working together, they hope to create a system that could potentially not just improve athletes’ performances but also help rehabilitation patients learn to move without pain—and perhaps even assist people with disabilities to regain the use of weakened or paralyzed limbs.

“Our goal is to develop a cheap, effective suite of sensors that we can use for biokinematic data acquisition,” says Grace. Figuring out how to collect the sensor data, and then turn it into useful information, will require knowledge of not only human physiology but computer science and engineering as well.

Grace, West, and Lewis have already scoped out the basic parameters for the device. It will include five postage-stamp-sized motion trackers that a person will wear on their legs, arms, back, or other body parts. Each tracker will hold an accelerometer (a tiny sensor that detects changes in speed and turns them into electric signals), a gyroscope for measuring rotation, and a compass. By combining the data from all five trackers, the system will be able to get an instant, accurate fix on body position, especially the angles of the joints.

Today, serious athletes and rehabilitation patients typically use cameras to watch and refine their body motion. For example, to train for the Rio Olympics, the United States swim team relied on underwater cameras that recorded light from LEDs stuck to the swimmers’ wrists, shoulders, hips, knees, ankles, and toes. Lewis, who helps patients with hip pain change their walking habits to make movement more comfortable, uses a similar approach to evaluate the gait of patients in her lab. While patients walk on a treadmill, she records their movement using a network of 10 wall-mounted cameras, and then later merges the two-dimensional videos to create a 3D version. It takes at least a day of work to process the images, so patients usually have to wait a week or longer to get results. “That is a huge limitation,” says Lewis.

Another limitation: while stationary cameras work in closed environments, like a pool or a lab, they don’t translate well to the expanse of a hockey rink or to everyday settings like roads and sidewalks, where bumps and curbs can turn an ordinary walk into an obstacle course. “The proposed system can record movement out in the real world during real activities,” says Lewis. “By being able to record how someone is moving out in the real world and in real time, we could give the person feedback on that movement.”

It isn’t as simple as just strapping on a Fitbit, though. The Fitbit and other familiar fitness gadgets use just one tracker. Because the new system will use five, the researchers have to figure out how to synchronize all five trackers to the “ticking” of the exact same clock. And because the results have to be delivered in real time, they can’t just put a time stamp on each sensor and align them later. If the timing is off even a little, the joint angle calculations will come out wrong. “That’s the real-time challenge,” says West. Grace and an undergraduate mechanical engineering student, Fedir Teplyuk (ENG’18), are now trying to figure out exactly how tightly the trackers must be synchronized to get the correct joint angles.

To deliver on the real-time promise of the electronic mirror, the system will also need to be able to process data quickly. That’s where West comes in. West specializes in designing operating systems for robots, airplanes, cars, and other time-sensitive applications where a missed beat could be disastrous. Similarly, the biokinematic tracking system will need a built-in operating system that can absorb, process, and share sensor data in real time. Working with PhD student Zhuoqun (Tom) Cheng (GRS’19), he is building an operating system from the ground up that will run on specialized “Edison” computer boards from Intel and also on a sixth, central computer responsible for merging the incoming data and streaming it to a receiving computer, where users can see their results and compare them to the ideal body position.

Inspired by his work with robots and other motorized systems, West sees another application for the biokinematic data acquisition system. By hooking up the sensors to little motors called actuators, it could be possible to create assistive devices for people with disabilities, like a “virtual walking stick” that could anticipate and prevent falls. “You could potentially sense when they are about to lose their balance and trigger an actuator to help steady them,” says West. Combined with actuators, the biokinematic data acquisition system could even be applied to driverless vehicles, like “swarms” of flying drones, to help them react autonomously to obstacles and weather changes in real time, without help from an operator on the ground.

Long before those spin-offs arrive, though, Grace, West, and Lewis can see the system on store shelves alongside more traditional fitness trackers. “Christmas presents are what we were thinking!” says Grace, joking that young athletes are more likely to take constructive criticism from a computer than from a parent or coach. “They believe anything that comes out of a computer.”

Meanwhile, Grace’s children, now 11 and 13, have improved their skating through old-fashioned “analog” coaching and practice. “My kids actually skate really well now,” she says.