Aerials, Apparatus, Pumpers, Rescues, Tankers

The Dangers of Data

Issue 12 and Volume 21.

By Philip Duczyminski

Businesses have been using data to make decisions for many years now.

The data they use help them analyze their business practices and help them become more efficient and more profitable, hire better employees, and market their products more effectively. In 1994, Compstat was effectively used within the New York Police Department (NYPD). Compstat stands for COMPuter STATistics and is a management process within a performance management framework that synthesizes analysis of crime and disorder data, strategic problem solving, and a clear accountability structure.1 Compstat has been credited with reducing crime in New York as well as other jurisdictions, but would the premise behind the system work for fire departments as well?

First and foremost, this article will detail some of my personal experience with Compstat as well as provide some recommendations to make the process more suitable for fire departments. In 2009, the Novi (MI) Fire Department hired the International City Managers Association (ICMA) to perform a staffing and utilization study. In the study, ICMA recommended implementing regular Compstat meetings.2 Since that time, the department began conducting a weekly meeting between the police and fire departments with the weekly data presented to both agencies. One of the biggest things I have learned is that data usage is different for every agency. A police department can use data to find crime trends, accident trends, etc. These data can help determine where to effectively put law enforcement officers to reduce these incidents. A fire department will not necessarily reduce fires or medical incidents by placing more firefighters in a given area. However, there can be some benefits if the data are used correctly and ethically.

Why Look?

So, why would you want to look at your data? The simple answer is to improve. Did you know that 85 percent of all statistics are made up? If you do a search across the Internet, you will find thousands of statistical inconsistencies. Why would that occur? Simply put: When someone is trying to prove a point, he needs some kind of data to prove his point. Mark Twain said, “Facts are stubborn things, but statistics are pliable.”3 I know this point may cause some unrest for some, but let me ask you one more question: Have you ever seen someone try to prove his point with data and find that the opposite fact is true? Me either. Within a selection of data, there are many variables that will generally allow the person selecting the data to keep, change, alter, or eliminate data or just change the criteria to keep data out of the report that are detrimental to his cause.

I know of an emergency medical service (EMS) agency that contracts for service with a local municipality that is contractually obligated to meet a response time standard. The EMS agency must arrive to priority calls within 7 minutes 30 seconds 90 percent of the time. This looks pretty good on the surface, but if you look at the data you will find that the agency is allowed exceptions from the standard. It is allowed an exception for instances such as multiple calls at the same time, poor road conditions, etc. Once it subtracts all the calls it determines to be “exceptions,” that is its average response time. In my opinion, this shows that the data can be manipulated by many variables to ensure the agency is meeting its contractual requirements. There should never be an emergency call that is considered an exception. I am not making the argument about which service is superior. The point I am trying to make is that the numbers don’t tell the whole story, and many people will use those numbers to prove their point even if they are flawed. Residents don’t want exceptions, they want service. As George Canning said, “I can prove anything by statistics, except the truth.”4

This is why data can easily be misrepresented, deceiving, and dangerous. Does this mean you should not look at the data? No, you still need to look at your data, but you must exercise a lot of caution. “How easy it is for so many of us today to be undoubtedly full of information yet fully deprived of accurate information?” asks Criss Jami.5

Using Data

Properly used data can help an organization identify weaknesses or find places that may need improvement. The data can provide the metric to judge the effectiveness of our services to some degree. One of the common areas reported on is incident response times. Incident response time can get pretty complex, depending on what you are looking for. Is your response time the time from when the dispatch center receives the call to the point where the first arriving unit calls on scene; the time from when the apparatus began responding to the point it arrives on scene; or the time from when the call was dispatched to the time when the appropriate apparatus was on scene?

I have found that many people who want to talk about their response times want to highlight the shortest response possible. However, for a structure fire, would a supervisor vehicle with no fire suppression capabilities arriving on scene be considered the arrival time? When analyzing data, organizations should include how the data were extracted. I recently read an article on the effectiveness of compressed-air foam systems that said they were “200 to 300 percent more effective than water.”6 That sounded pretty impressive, and I wanted to see how those statistics were determined, so I looked for the references to do my own research. The problem that I found was that there were no references. I am eager to listen and learn; however, you must do more than give numbers. You must let people know how the data were extracted. Being open and honest is the only way to keep your credibility.

How Data Can Be Misleading

There are several ways that data can be misleading, and in 1954 a best-selling book titled How to Lie with Statistics was published. Many of the lessons taught are still used today. Bias sampling is the process of favoring certain outcomes over others. Therefore, the author of the data may choose a sample that would likely identify with his viewpoint. The use of small sample sizes can dramatically alter the results. If a study does not use a sufficient sample, it can cause the statistics to go crazy, many times in its favor.

Another commonly used method is to fabricate the numbers. Very seldom will anyone even challenge the data, and the misleading information will be taken at face value-for example, saying “98 percent of statistics are made up.” I am not sure if that is anywhere near true, but if I do a quick Google search, I am pretty sure I can substantiate it with a reference. Another way data are misrepresented is by simply ignoring the baseline. This happens when raw numbers are compared without adjusting for expected baseline differences and may occur by accident or on purpose. An example that occurs on a regular basis is the dollar trend over a specific time period-there is not always an adjustment for inflation.

Next we have selection bias. This occurs when the data used for a study lead to a result that is different from what you would have gotten if you had used all target information. This may occur in the event that an agency is presenting its response times. I think the majority of people could agree on what a true response time is, but it is not that simple. I personally know a fire department that reports its response time as the time from which the apparatus calls responding to the time it arrives on scene. Some may agree or disagree with that, but unless they are forthcoming with all of the information, most would not even question it.

Another method used is to simply eliminate data that do not prove the point of the author. If someone is trying to prove a point, he does not want to prove himself wrong. So, he can simply eliminate the data or change the criteria from which he will extract the data. In the fire service, most of us do not have a degree in statistics, nor have we been trained in this area. One thing is for sure: We will learn and make do with what we have. I know our fire records management software has a lot of trouble extracting good data, or there may be other factors that limit the data that can be extracted. This may lead to the author of the data having to manipulate the data manually because the system parameters or reporting software will not extract the needed data. This can lead to human error.

SIX Basic Rules of Using Data

Following are six rules for using data:

  1. Do not make data punitive. The goal of collecting and using data is to get better. If you make the process in anyway seem punitive, you will lose buy-in from your crews.
  2. Allow crews to have input into the process. The firefighters must be able to make recommendations on the data. If they are not, you will lose credibility.
  3. Be cautious where you get your numbers. You can get comparative data from many agencies around the country. The problem is that you may not know how accurate their data is. Last year I was asked for some data for one of the large organizations that does studies on fire departments around the country. I let the individual know that the requested data were impossible to extract accurately. I was told, “Don’t worry about it; it does not need to be accurate.” This struck me as funny. If my data do not need to be accurate, then who is giving them accurate data? I would not be confident that any of their data are accurate, which would impact the numbers they are supplying to agencies around the country.
  4. Do not make wholesale changes to your agencies based on data alone. Data do provide us with good tools for evaluating services. However, I believe real-life experiences and results can be more impactful than data alone.
  5. Carefully plan the data that will benefit your agency. Each agency should carefully plan what they want to study-response times, turnout times, percentage of a particular incident type, incidents by time of day, etc. I believe we should have a firm grasp and know what our agencies’ average response times are. However, I would not push firefighters to drive faster to cut response times. Looking at turnout times would be a more responsible approach. Each agency should have a goal for how fast its crews are out the doors once an incident is dispatched. These numbers should be studied to find out if crews are responding in a timely manner or to determine if there is something impeding their response. If your agency is conducting company-level inspections, is there a chance that certain occupancies are delaying responses? Data can help us identify issues and make changes.
  6. If you uncover issues, fix them. There is no use collecting and reporting on data if you are not going to do anything with them. It is negligible to know a significant problem exists and not to do anything to resolve it. The goal should be to uncover problems, determine the impact, and then plan and implement solutions.

I firmly believe that there is a real need to use data in the fire service. The law enforcement community has been using data successfully for many years now. If you look around at the majority of elected officials, they do not have a full understanding of our jobs and responsibilities, nor will they realize the impact of their decisions. A great deal of these elected officials work in the business world and understand numbers. For us to be effective in our message, we must communicate with them effectively, and we need to substantiate what we are saying with numbers that tell the other side of the story and speak their language.

Endnotes

1. University of Maryland. (2016, January). What is CompStat. Retrieved from University of Maryland: http://www.compstat.umd.edu/what_is_cs.php.

2. ICMA Consulting Services. (2009, June). Public Safety Staffing and Utilization Study. Retrieved from City of Novi: http://www.cityofnovi.org/City-Services/Public-Safety/Fire/Administration/Public-Safety-Staffing-Utilization-Study.aspx.

3. Twain, M. (2016, January 27). Quote by Mark Twain. Retrieved from Quotery: http://www.quotery.com/quotes/facts-are-stubborn-things-but-statistics-are-pliable/.

4. Canning, G. (2016). Canning Quotes. Retrieved from QuoteAddicts: http://quoteaddicts.com/topic/canning-quotes/.

5. Jami, C. (2016). Quotes About Information Technology. Retrieved from Goodreads: https://www.goodreads.com/quotes/tag/information-technology.

6. Klassen, K. (2011, August 18). Top 10 Excuses for Not Using Foam/CAFS. Retrieved from Firefighter Nation: http://www.firefighternation.com/article/firefighting-operations/top-10-excuses-not-using-foamcafs.

PHILIP DUCZYMINSKI is a 19-year veteran of the fire service and a captain and head of the training division of the Novi (MI) Fire Department. He has served with the Western Wayne County (MI) Hazmat Team and Michigan Urban Search and Rescue Task Force 1. A graduate of the School of Fire Staff and Command at Eastern Michigan University, Duczyminski is a certified Michigan fire instructor and EMS instructor coordinator and has a bachelor’s degree in fire science.