Confronted by criticism over its handling of player concussions, the NFL dramatically stepped up its response three years ago by installing neurotrauma specialists on the sidelines of every game and concussion spotters in booths high above the field. Their job: spot possible signs of concussions and take players out of the game if necessary.

Today, league officials point to the steps as a significant victory in their effort to protect players in one of the world’s most injury-prone sports. It’s “99.9 percent” successful, said Dr. Richard Ellenbogen, who oversees the concussion-spotting system.

In interviews with STAT, however, doctors and outside experts pointed to a series of weaknesses in the program. With limited authority, the sideline spotters cannot overrule team doctors who might disagree with their diagnoses. They are not used at NFL practices, where an estimated one-quarter of concussions were sustained last year. And some concussion specialists said they face pressure from teams to hurry injured stars back into games.

“Things happen so fast in the NFL that, by the time something happened, if we were concerned, the game was going on and they’d say looks fine to me,” said Dr. Amy Jarvis, a vascular neurologist who served on the sidelines during Jacksonville Jaguars games during the 2013 season.

Some of the doctors who have criticized the NFL program acknowledge that it has improved over time, as does the NFL Players Association, the players union that is frequently at loggerheads with the league. Ellenbogen, the co-chairman the NFL’s head, neck, and spine advisory committee, said the league can and will improve, adding that it has a “moral obligation to err on the side of safety and conservatism.”

But he insisted the neurotrauma specialists have made the game safer.

“Obviously,” he said, “the media looks at the ones that are aberrant and says, ‘How did you guys miss that one?’”

There are limited data to assess the effectiveness of the NFL concussion-spotting protocols. According to figures from the NFL and the Players Association, concussions during games and practices were down by 22.6 percent between 2012 and 2014. But concussions this year are trending higher, and threatening to erase the gains.

A constellation of spotters

If concern over missing a player’s concussion is high, it is perhaps because the costs of doing so can be steep. Research shows that players who suffer multiple concussions before their brains have healed from an initial injury are likely to experience a slower recovery, stronger symptoms, and the potential for long-term brain damage.

Concussions, though, can be devilishly difficult to identify. They rarely lead to a loss of consciousness, and they do not typically show up on CT or MRI scans.

The system developed by the NFL to spot concussions as soon as they happen involves a constellation of trainers and experts.

But the NFL’s introduction of neurotrauma specialists, known in the league as Unaffiliated Neurotrauma Specialists, or UNCs, was seen as especially valuable. They were a testament to the NFL’s commitment to independent medical expertise and safety above all else.

Independence from the teams is important, the league and players union said, to bring greater objectivity to the identification and diagnosis of concussions. Other than the neurotrauma specialists, there is only one other spotter who is not paid by teams: an athletic trainer, known as an “eye in the sky,” who is stationed in a skybox and outfitted with binoculars, video-replay technology, and an audio connection to sideline personnel and the officiating crew.

That trainer, however, does not have the same medical background as the neurotrauma specialists on the sidelines. Nor do team doctors, typically orthopedists, or other trainers, who lack medical degrees.

Limited authority on concussions

There is no single test to determine authoritatively whether a player has sustained a concussion. But the neurotrauma specialists and team doctors usually begin with a so-called Maddock’s test, in which they ask players details about the game. Who scored last? Where is the game being played?

Team doctors are encouraged to consider the neurotrauma specialist’s assessment of a player’s performance, but if a team doctor believes the player can return to play, he may do so even if the specialist disagrees.

If the team doctor believes the player needs deeper evaluation, the doctor and the specialist give the player a more detailed assessment, usually taking up to 10 minutes, in which the player is scored on a range of mental and balance-related challenges.

Again, the opinion of the team doctor prevails if the specialist disagrees with the assessment.

The NFL Players Association says it knows of no instances in which a team doctor has overruled a neurotrauma specialist and returned a player to the field too quickly. Dr. Brian Nahed, a neurosurgeon at Massachusetts General Hospital who works as a UNC at Gillette Stadium during New England Patriots games, said he has never encountered a doctor or trainer who was reluctant to diagnose a concussion.

“I’ve never had an issue,” he said. “Everybody’s really very concerned for the players’ health more than anything.”

The system, though, is not foolproof.

Spotters might miss less obvious concussions amid the chaos of 22 players, said Dr. Matthew Matava, medical director for the St. Louis Rams and former president of the NFL Physician Society.

“And that’s the fault of nobody,” he said. “It’s a very tall task that’s fraught with practical limitations.”

This season’s most public failure of the concussion-response system came on Nov. 22, late in a game between the St. Louis Rams and the Baltimore Ravens. The Rams quarterback, Case Keenum, suffered a hit to the head in clear view of most people in the stadium.

Keenum, who was prone after the play, immediately grabbed his head. A Rams lineman tried to pull him upright, but the quarterback collapsed. Seconds later, he staggered to his feet and walked haltingly toward the sideline and was met by the Rams’ director of sports medicine and performance, Reggie Scott.

Scott, who is not a physician, spoke briefly with Keenum and let the quarterback continue without bringing him to the sideline for a full concussion screening. He was later determined to have sustained a concussion in the play.

Two plays later, a Ravens linebacker sacked Keenum from the quarterback’s blind side, forcing a fumble that effectively ended the game for Keenum.

The NFL, amid a firestorm of criticism, later investigated the incident, and last week said it would fine those who, in the future, fail to follow concussion-spotting protocols.

The history of reform

In October 2009, NFL Commissioner Roger Goodell testified before the House Judiciary Committee about the long-term effects of player concussions, and was roundly criticized for evasiveness. One legislator said the NFL’s response to concerns reminded her of tobacco companies denying the health effects of cigarettes.

Within two months, the league changed its tone.

That December it said it would donate $1 million to the CTE Center, whose groundbreaking research around chronic traumatic encephalopathy, the degenerative disease resulting from repetitive brain trauma, the NFL had previously sought to discredit. The league established a rule that concussed players could not return without clearance from an independent concussion specialist.

In a nod to the seriousness of its intentions, it also accepted the resignations of doctors from the league’s concussion committee, who had become the public faces of its denials of concussion science. In their place came two prominent neurosurgeons as committee co-chairmen: Dr. H. Hunt Batjer, of Northwestern Memorial Hospital, and Ellenbogen, the chairman of neurological surgery at UW Medicine in Seattle.

“Roger Goodell called me while I was walking across the street,” Ellenbogen said. “He says, ‘I hear you’re involved in traumatic brain injury,’ and I go, ‘Yeah,’ and I’m thinking, ‘Which jerk is calling me now? Some clown, one of my friends in Boston or Washington?’ And it was Roger Goodell.”

In 2011, the Players Association drafted the first concussion-evaluation protocol to combat what it regarded as a patchwork approach to concussion management across the league.

Amid rising criticism around teams ignoring obvious concussions, the league introduced significant rules changes meant to minimize helmet-to-helmet hits and permanently installed “eye-in-the-sky” concussion spotters in the skyboxes during the 2012 season.

In early 2013, the NFL agreed to implement the UNC program and a unified concussion protocol for all teams.

Ellenbogen said he worked with neurosurgery leaders in each of the league’s 32 cities to find doctors who were not affiliated with teams, often using a hospital-centric approach. In New York, for instance, the Giants’ UNCs are from the Weill Cornell Medical Center’s neurosurgery team, while the Jets’ UNCs are largely composed of emergency-department physicians from Mount Sinai Hospital.

“At first, they said, ‘Are you crazy?’” Ellenbogen said. “No one knew what the job was, and it’s kind of grown with time. It’s become more important.”

Mixed reviews of the system

Jarvis, the vascular neurologist who worked the Jacksonville Jaguars games during the 2013 season, said the program was not exactly embraced by NFL teams.

“I can’t remember one time when we were able to say we’re concerned about that one and pull them,” she recalled. “At the end of the day, it was the team coach and head team doctor who were making those decisions.”

Obvious concussions, where players were “staggering or knocked out,” were addressed quickly. “But there was a big gray area in between, and enough players were savvy enough so that even if they were experiencing symptoms, they’d stay in.”

Jarvis, now at North Shore Medical Center in Miami, also said players were rumored to intentionally botch their baseline cognitive assessment tests during preseason, so they could pass the test more easily when actually concussed.

A former neurotrauama specialist, who spoke on condition of anonymity because of continuing relationships with the league, agreed with Jarvis’ characterization that UNCs were largely ignored in the first season. That was especially true with visiting teams, this doctor said, since the specialists were strangers to the visiting staff.

This doctor praised the NFL for investing significant authority in UNCs and other outside neurotrauma specialists. Once a player has been removed from a game, only an independent neurological specialist who is not paid by the team can determine when players can return.

However, this doctor said, teams still try to pressure the independent concussion specialists to rush players back into action. “There are some players who you’ll get a call from the trainer about every day, sometimes twice a day,” the doctor said. “They’ll tell you ‘I’m getting a lot of pressure from my superiors to clear them.’ For other players, they’re fine waiting.”

Others have raised concerns that neurotrauma consultants aren’t present at practices, even though dozens of concussions were sustained during practices last year. Jeffrey Miller, the NFL’s senior vice president of health and safety policy, said it’s not necessary to have the specialists at practices since teams are already on the lookout for signs of concussions.

“I don’t want to overstate the case,” Miller said of the UNC initiative, “but at this point we’re very pleased it’s a success. The goals of the program — putting player health first — are being met.”

Advocates for players are skeptical. They note that, although players and their coaches might be attuned to the risk of concussions, they also have reasons not to acknowledge them. Players diagnosed with concussions are pulled from games, after all, and even players who suffer concussions during practices risk missing future games.

Christopher Nowinski, executive director of the Concussion Legacy Foundation, which spearheaded early awareness about the long-term effects of concussions in football, said the concussion-spotting system is “lipstick on a pig.”

Nowinski noted a player’s career can be threatened if teams believe he is concussion-prone.

The spotting system is “primarily there to prevent embarrassment to the NFL, even though it’s caused more embarrassment because the execution has been so poor,” he said.

Other experts say that the NFL’s latest concussion statistics, though encouraging the last two seasons, may mask the larger problems.

The only way to accurately count concussions in sports is to anonymously survey players after the season, when they have nothing to lose by reporting symptoms, according to Dr. Robert Cantu, a neurotrauma surgeon and researcher with Boston University and one of the leaders in identifying CTE in football players.

Past research, he said, shows that players sustain multiple concussions for every one that is officially diagnosed. As a result of the concussion-spotting system and increased awareness, he said, the figure “might be down to three or four. There are not really good studies that tell us.”

“We know that a lot of concussions are still being missed,” Cantu said.

An earlier version of this story incorrectly characterized Dr. Amy Jarvis’s specialty.