The FMCSA’s CSA program (Compliance, Safety, Accountability), is in for some big changes. While the CSA has been a good idea in theory, it has been on the receiving end of plenty of ire over its shortcomings. In fact, interested parties have been pointing out problems with the program for some time, even before it was officially rolled out in 2010. But what is precipitating this change?
CSA was supposed to give the FMCSA a better way to identify motor carriers that needed a compliance review. The program was supposed to use inspection results and other data points to create scores that would put a spotlight on fleets who were operating in an unsafe manner. Fleets that were pinpointed would be flagged for some type of intervention, whether that be a warning letter, compliance review, or otherwise.
The problem is that CSA was plagued with problems from the very beginning. To start, brokers, insurers and attorneys were using the publicly available CSA data in ways that the Department of Transportation (DOT) did not intend or foresee. Fortunately, Congress stepped in and ordered the FMCSA to shut down public access to the data. This is where a mandate from the National Academy of Sciences (NAS) was put into place to engage with scientists and engineers from the agency to revamp the program.
The NAS Takes a Look
After completing an initial review of the program, NAS noted several problems with CSA. From highly variable assessments to no accounting for accidents where the truck driver or carrier was not at fault, NAS found plenty of problems. They also specifically pointed out several measures that conflict with state policy, are not predictive of a motor carrier’s future crash frequency, and do not reflect a carrier’s efforts to improve their safety profile.
Because of these findings, the NAS told the FMCSA last year that the model is workable, but that it is not based on a principled scientific approach. As a result, the NAS recommended a more statistical model to anchor SMS data. NAS recommended that the FMCSA take two years to study a different modeling, specifically outlining something called “Item Response Theory.”
The IRT model would be designed to use a variety of data points, from truck driver pay, truck driver turnover, and miles traveled. The new system should be weighted by more than what is right and wrong, but rather how well an individual shows a depth of understanding of the topic.
Since the NAS put this directive forward, the FMCSA developed an action plan per the FAST Act, which is what mandated the survey in the first place. As this evaluation is very data intensive, it has taken some time – and will take more time – to complete the work. Since the IRT model is quite sophisticated, the time investment isn’t frivolous. In fact, this model has been used in other industries to some success.
And while the IRT model won’t predict crashes, it should provide valuable information leading to a greater understanding of what causes crash rates across fleets. Now, with a year having gone by, the industry has a pretty good idea of what kind of changes they can expect for the program. While these are a heavy lift, they should result in a better scoring system. Let’s take a closer look at how this is shaping up.
The IRT Model is Confirmed
The IRT Model methodology offers a few key takeaways and differences that the agency will rely upon. This past July the FMCSA delivered a report to Congress regarding what its corrective action plan would entail. It specifically mentioned the IRT model.
While the FMCSA has not released a ton of details regarding what the IRT model is going to look like, we can get a sense of what may come of it from the public meetings NAS has held regarding the recommendations. One company has even licensed an IRT modeling software and developed a version that many believe could be what the new model ends up looking like.
The agency has met its goal of running a small-scale IRT model by September of this year and plans to run a full-scale model by April of 2019. Planned rollout for the new model is slated for September of 2019. Of course, there is some chance for delays, but all signs point to the agency making headway in meeting its self-stated goal with just one year to go.
With the dates set and the model confirmed, many are wondering what this new methodology will look like. We have some ideas on what motor carriers can expect.
A Focus on Safety Culture
Here at the QuickTSI blog we have talked a lot about the importance of a strong safety culture for motor carriers. It appears the FMCSA believes the same, as they have stated they are moving away from trying to predict future crashes and instead focusing on evaluating a fleet’s overall safety culture. This change came from the problem with the current program attempting to predict future crash risk.
One example of this is regarding the BASIC scores. Some of the BASICs are not positively correlated, such as the drug and alcohol basic. The logic undermining the score dictates that in some cases, a truck driver could drink and take drugs while behind the wheel. It was a very counterintuitive logic. The new IRT model would move beyond BASIC scores and focus on a score based on a motor carrier’s overall safety culture.
The planned changes to CSA will include removing violation weights, CSA points, BASIC scores, and Safety Event Groups. Rather, a motor carrier will be given a single CSA BASIC score that will focus instead of the safety culture of a motor carrier, rather than specific events or weights.
Many are predicting that even though the IRT model does not necessarily predict crash risk, motor carriers with lower safety culture scores will wind up showing more crashes happen. Without focusing specifically on crash risk, this new methodology could still do an effective job at predicting it, without a high level of variability or illogical reporting methods.
Addressing Variety in Scoring
One of the things the FMCSA was tasked with looking at was the issue of variety. A variety of violations will generally tank a motor carrier’s score. As an example, under the new model, if a fleet is doing well in four out of five sections, they will not get hit as hard as a motor carrier who is mediocre across many scoring factors.
The IRT model will focus on carriers who are deficient in many areas, as opposed to carriers who are getting a few hits in a single area. The scoring model will look at patterns instead of points. The idea of severity weights will be gone under the new program, along with violations that are weighted based on how recently they were issued. There will also no longer be any percentile rankings. The percentile rankings were a major sore spot for motor carriers, who were being hit if they were lower than others in their “peer group” even if they did not have any major violations and scored well otherwise.
The FMCSA will also take a closer look at the 66 “violation groups” that currently exist in the CSA structure. One example of this is the lighting violation group. The IRT model will look at patterns across the industry and use them to see if they point to other bad things happening within the fleet. Lighting is just one example of this.
The IRT model will also look at something called exposure. Exposure will be measured by the number of vehicles, truck drivers, inspections, and miles traveled. It will be able to adequately extract exposures so that all motor carriers are measured against the same standard. This will prevent a motor carrier from being compared to others in a “peer group” and instead focus on continuous patterns of exposure to single out specific motor carriers.
Focusing on the Data
Finally, the IRT model will isolate data quality issues. One example it pointed out was the necessity for better data collection regarding vehicle miles traveled by state and month. This will take weather into consideration. The model should also assist in collecting data related to truck driver turnover, types of cargo delivered, and overall compensation. These factors will help the agency determine a fleet’s safety culture.
Still, there are areas where the agency may not be able to fulfill all the requests put forward by the NAS. One example is regarding the VMT data. Currently, VMT data is only collected every two years through the MCS 150 form. Collecting this data monthly may be unfeasible without a major increase in manpower or technical aptitude.
These are small potatoes when compared to the changes that will come because of the NAS recommendations and move towards the IRT model. With trucking advocacy organizations in favor of these changes, get ready for the new model, as we are now under 12 months away from implementation