A guest post by Joseph Ingles (*)
As we move toward the fifth presidential election of the 21st Century, we see a push toward allowing voters to use electronic means to cast their ballots. The convenience and ease of use of a voting app or website would make the process easier, increase voter turnout, and make tallying the votes much easier. However, the counter argument is that allowing voters to use electronics makes it all too easy to hack the system and change the outcome. Further, breakdowns in the systems and inconsistencies across the board would call into question the legitimacy of the results.
On February 3, 2020, the Iowa Democratic Party (IDP) was the first to actually try using an app, produced by Shadow, Inc., to tally results. While other electronics methods had been used with some success, this was the first time a smartphone app was used. The results were disastrous, as tallying and reporting the vote count took almost three full days. This led many to believe that Iowa was not ready for the caucus and thus, they should not be allowed to be “First in the Nation” any longer.
This blog post explores the reasons why the app failed to work properly, including poor training, worse testing, and almost no tech support. Also, the author will offer an analysis of the ramifications not only for the IDP, but also for the State and others who might want to use such an app in the future. Finally, the author suggests best practices for those using and developing apps for such purposes.
The Iowa Caucus and Shadow, Inc.
As chairman of the Democratic Party in Iowa’s Polk County, it fell to Sean Bagniewski to train his team on The IowaRecorder, a vote tallying app that party officials had purchased from Shadow, Inc. With nearly a dozen Democratic Party candidates vying for the presidential nomination, the “First in the Nation” caucus was a big deal for Iowans, the Iowa Democratic Party (IDP) and the nation at large. Not only would the caucus help to push forward the nominating process, but it would be the first time any party primary or caucus would be using an app to tally and deliver the results (Goldmacher & Corasaniti, 2020).
Bagniewski could see that something was amiss. Even before the February 3rd caucus, he had seen issues arising with the system. Poll workers and precinct captains were struggling to log in and download the app. Even when the could download it, they could not stay connected. During preliminary tests, Shadow – the company that produced the app – failed to deliver on “app-specific training” (Goldmacher & Corasaniti, 2020).
Bagniewski decided that he would instruct his poll workers and precinct captains to call in with the results. As it turns out, he was not the only one to do so. At least a dozen chairs from various counties made that same decision. These decisions would have far reaching consequences as the results would be delayed, questioned, and all but dismissed.
In the end, it would take some three days to tally all the results and almost a week to publish that Pete Buttigieg had narrowly edged out Bernie Sanders, taking 14 delegates to Sanders’ 12. Buttigieg would also take the popular vote by fewer than 1,000 votes (Iowa Democratic Party, 2020). The faith in the system was shaken to the point that Derek Eadon, a former IDP chairman, called the mistakes a “systemwide disaster” (Goldmacher & Corasaniti, 2020).
Failure Trifecta
The IowaRecorder app failed on several levels, from the macro to the micro level. Yet the root causes of the failure can be narrowed to three; lack of transparency, failure to test, and lack of training. While each of these contributed to the failure, the lack of transparency between the DNC, IDP, and Shadow is perhaps most troubling.
Lack of Transparency
The IowaRecorder app was developed by a for-profit company called Shadow, Inc. This company has ties to several Democratic political campaigns. Additionally, the primary funding partner is a progressive nonprofit group named Acronym. While this is not necessarily a big deal, the lack of transparency became an issue after an NPR report showed that the app “had issues” and might not be ready for prime time. However, Iowa Democratic Party Chair, Troy Price, declined to give more details on the app or who wrote it and how it was funded. Price did say that security was a priority. “We want to protect the integrity of the process,” Price said. “We want to make sure we are not relaying information that could be used against us.” (Payne & Parks, 2020)
While that level of security is certainly understandable, it can also lead to a lack of trust and communication. Doug Jones, Professor of Computer Science at Iowa State, believes that the Democratic Party’s decision to keep technical detail of the app a secret made the problem worse. “Drawing the blinds on the process leaves us, in the public, in a position where we can’t even assess the competence of the people doing something on our behalf” (Scheider, 2020).
Additionally, the app and the development of it was kept secret despite the fact that Acronym, a primary investor, is a known progressive political action committee and not a software developer. While no definitive proof is available, some (Whittaker, 2020), (Rayome, 2020), (Statt, 2020) have posited that Acronym and Shadow rushed to develop the app not because of a desire to make the process better but rather, to ensure that the app was available and ready to take down Trump. However unlikely this scenario is, it fuels the fires for Trump supporters who point to “collusion” that backfired (Scheider, 2020).
The issue with the secrecy is that it made the problem worse from a coding standpoint. The app, the source code, and the process were kept secret in an attempt to keep hackers from exploiting the system. Yet, the idea of “security through obscurity” is a fallacy. Further, the issue was made worse when Iowa Democratic Party declined an offer from Department of Homeland Security official, Chad Wolf, to test the app for security flaws “We have no way of knowing if the app went through extensive testing or what levels of scrutiny there are” (Whittaker, 2020).
Failure to test
From a technical point of view, the failure to test is probably the most obvious failure on the part of Shadow and the DNC. The app was hastily built by coders who lacked the proper knowledge and technical training to build such an app. The entire app was “rushed out the door in less than two months’ time.” (Statt, 2020) Further, Shadow did not go through the proper app store review processes for either Apple or Google. Instead, they used “beta testing platforms like Apple’s TestFlight” to meet the Monday deadline (Statt, 2020).
Rayome (2020) reported that Blue Hexagon had obtained a copy of the app and tested it to see what went wrong. Rayome interviewed Irfan Asrar, head of Blue Hexagon’s cyberthreat division. Asrar said that the app had several problems with the code including links to personal websites, broken code, and incorrect instructions. “What we believe is that this is an oversight and an example of an app being rushed into production,” said Asrar. “The larger concern is that the source code for the app was so easy to obtain, which means that anyone could access it and cause damage” (Rayome, 2020).
With such a short window to roll out the app and limited testing, the chaos of the Iowa caucuses was simply too much for the app. Because the app was never fully tested, Shadow, Inc. had no way of knowing that the process would break the app. Of course, they should have known and would have known if the process had not been rushed. “It appears in this case that the app was never really tested in a manner which came close to approximating the real mess of an election,” (Jones, 2020 as cited in Schneider, 2020).
Lack of training
The caucus process is hectic at best and chaotic at worst. Each delegation must find an area at the caucus sight and convince other caucus goers to follow them. After each round, those with highest tally move on while others are left behind. This continues until all candidates have at least 15% of the total caucus goers in attendance. Iowa and other states stick with the caucus model because it “encourages more engagement and grass-roots politicking” (Epstein, 2020). While caucus workers, precinct captains, and county chairs are all fully trained in a well-established process, this year, things were different. Many of them simply had not been trained to use the new app and had no idea how it worked or even why they should be using it. “Caucus chairs, in many cases, apparently were attempting to download and install the app on their phone on caucus night.” (Scheider, 2020).
Norvell (2020) reported that the “little-to-no training” based on surveys and interviews done with Democratic Party chairs. Rather, precinct captains and volunteers were told to “test using it on their own.” Additionally, the app was not downloaded on the app stores like other apps are. Rather, volunteers had to apply to download it. After multiple security prompts, they were able to download the app. In addition, the pin number had to be changed every day to promote a more secure environment. “I think that we wanted to make sure it was secure and no one was hacking it so we added just enough security to make it hard for people to navigate,” (Niles, 2020 as quoted by Norvell, 2020). As a result, some were locked out of the app completely because they forgot their pin.
As a result of the issues faced, many simply abandoned the app and reported results manually. When this happened, the phone lines became jammed and wait times of over an hour results in many precincts. Even when the precinct captains could get through, they found that the information was not being transmitted properly. “A lot of people had difficulty downloading the app and were not happy when they did the test on it. So, we came to a consensus not to use it.” (Foley & Cassidy, 2020). Johnson County precinct chair, Jonathan Green, is an IT manager by day. He reported that he was able to test the app, but kept getting error messages when reporting results. “It was just a wreck. The system was not prepared to handle the app not working and everyone was overwhelmed” (Green, 2020 as quoted by Foley & Cassidy, 2020).
Analysis
While the Democratic Iowa Caucus certainly had its issues, the results were ultimately pushed out to the public and the delegates were added to the total. The work around, as it were, was to go back to the old-fashioned method of counting by hand and calling in the results. While time consuming, it was effective. Additionally, Nevada caucus officials chose to scrap a similar app developed by Shadow to prevent similar results, there. (Scheider, 2020).
However, the clear loser in Iowa was the Iowa herself. Several of the previous contests were questioned. In 2016, Republican party officials initially called the state for Romney, but then reversed themselves to give the state to Trump. That same year, Sanders and Hillary Clinton were in a virtual tie for three days before Clinton was declared the winner. As a result, many have called for Iowa to give up its status and give the “first in the nation” to someone else. This could mean that Iowans are no longer able to hold the sway of politicians and thus will quickly be forgotten (Goldmacher & Corasaniti, 2020).
Further, it is not realistic to simply continue with the status quo. As Millennials and Gen Z begin to engage in the political process, they will demand and find solutions to these issues. With the lines at polling places growing each year and the wait to vote increasing, a solution must be found. That solution can be a mobile platform, if it can be tested and secured.
CEL Testing
One solution for testing apps is to automate the process. That is, instead of just putting testing as a node on the network diagram, the process for testing should be automated and continuous. One approach to this process is called the Continues, Evolutionary, and Large-Scale (CEL) Mobile Testing process (Linares-Vasquez, Moran, & Poshyvanyk, 2017). The CEL process works best when the app needs diverse testing with continuous delivery but has time and budget constraints. This is exactly the environment where Shadow found itself when rolling out the IowaRecorder app.
Continuous. The model introduced here follows the continuous integration and deliver (CI/CD) model first introduced by Booch (1991). Under CI/CD, apps are continuously tested, updated and redeployed so as to have multiple interations of the app. Any change to the source code would necessitate and automatic testing of the of the current version of the app. Interfaces and reports are updated in real time and automatically so coders can concentrate on fixing, not finding, problems.
Evolutionary. Here, we allow the entire process to evolve together, not independently. That way, changes made in one section can be integrated across the app platform simultaneously. This, in turn, would trigger another round of automatic testing so that entire process is allowed to evolve together.
Large-scale. “The engine should enable execution of test cases that simulate real conditions in-the-wild” (Linares-Vasquez, Moran, & Poshyvanyk, 2017, p. 405). To reduce costs, the testing could be done in a virtual environment as much as possible. The real-world roll out can be done in the cloud and with on prem devices to give the best possible chance for both success and failure. Finally, initiating large-scale testing will push the system to its limits before it breaks in the real world so that developer know where to fix issues.
Issues with CEL. The main issues with CEL revolve around the Continusou Integration and Deployment protocols. Irani (2018) points out that a common mistake is to automate the wrong processes first. This could easily happen to any app and would have made the problems with the IowaReporter app worse. There, if developers had automated the data gathering process first instead of the data reporting, the system would have responded much better.
Another issue with CEL is a lack of coordination between integration and delivery. Integration feeds delivery. Without that pipeline in place, the system breaks down (Irani, 2018). This is more about the human factor than the actual data pipeline. By making sure that the integration comes first and the delivery second, the entire process works seamlessly. “One example is Netflix. Companies like Netflix can complete integration, testing, and delivery in a matter of two to three hours. They established a system that passes code from hand to hand without indecision and discussion” (Irani, 2018). This lack of coordination between integration and delivery was evident at Shadow. In fact, the programmers were inept and thus did not create a solid architecture to begin with (Statt, 2020). As a result, the CEL testing framework probably would not work with the IowaRecorder app.
Black-box Testing
Black-box testing is a method of testing that looks at the functionality of an app without seeing the code or internal working. So, the tester need not fully understand how the app is supposed to do what it does. Rather, they only need to understand what it should do and the steps to make it work. This allows for end-user testing and unit testing at a much faster rate. It also creates an environment where the programmer does not know who the testers will be, so they must write code that is simpler and more streamlined (Gao, Tsao, & Wu, 2003).
Random testing and scenario-based testing are ideal for mobile app development as these techniques allow for the maximum number of test scenarios in a shorter amount of time. When faced with time constraints such as Shadow was, these techniques make quick work of the testing process, give instant and useable feedback, and help developers find problems and solutions quicker (Gao, Bai, Tsai, & Uehara, 2014).
Issues with Black-Box Testing. The main issue with Black-Box testing as it applies to IowaRecorder is that the test cases are system specific so the specs have to be well-documented and understood. As a result, static methods of testing must be applied before you can black-box the app (Satoh, 2004). Thus, the Black-Box testing probably would not work with app because the specs were not well documented. Put another way, even if you do not know what makes the vehicle move and you only know how to drive it, someone has to know how it works and have that info written down if you have any hope of fixing the vehicle when it breaks down.
Mobile Test Automation Frameworks
As companies such as Apple and Google realized that they could monetize app development, they also realized the need for making sure the apps actually worked on their mobile devices. As a result, the software developer kits (SDKs) from each of these companies have white box tools built in to provide easy testing as developer work.
For the Android/Google Play store, the SDK has a tool called JPF-Android that makes the testing process much faster and easier. By building on the Java Patherfinder (JPF) platform and adapting it to fit the Android platform (also written in Java), the SDK helps to make the transition from desktop/laptop to mobile much less painful for developers (Mahmood, et al., 2012).
For the Apple App Store, the programming language is Swift. Build on the Objective-C language, Swift makes the process of compiling and writing code much easier. For Swift, Apple has gone beyond the SDK to an Integrated Developer Environment (IDE). The IDE uses the Xcode graphical user interface to make the process easier. The IDE also tests and debugs as you go, so that you have a virtaul black-box environment to use (Sharma & Sharma, 2018).
Of course, for either of these to work, Shadow would have to open up their code and develop the app so that it is available on the app store. By choosing not to do this, Shadow made its app vulnerable and ultimately failed to deliver.
Conclusion
Shadow, Inc. failed to deliver on its promise and ultimately took some $63,000 for a product it did not deliver (IECDB, 2020). Also, IDP Chair Troy Price has said he will fall on his sword and step down over the debacle (Pfannenstiel, 2020). However, mobile apps are not going away. The mobile app industry is growing by orders of magnitude. As more and more customer and clients demand apps for almost every task, developers must find a way to offer apps that not only tally results, but also let citizen vote in primary and general elections. Surely there will be those who refuse and must be accommodated. Burying our collective heads in the sand and ignoring the rising tide is not going to work. This should not be the end of the line for mobile apps in our election process. By utilizing the tools available, streamlining the training process, and retooling to fully utilize the process, app developers can ensure that their apps work.
As for Sean Bagniewski, the Polk County Democratic Party Chair, he says folks still had a great time at the caucuses, even if the party had difficulties delivering the results. “The first timers, the people in the room and the volunteers all had a really good night. The reporting stumbled, but if we have tens of thousands of new voters in Iowa, that’s a success for us” (Bagniewski, 2020 quoted by Jacobsen, 2020).
References
- Booch, G. (1991). Object Oriented Design With Applications. San Francisco: Benjamin/Cummings.
- Epstein, K. (2020, February 3). How do the Iowa caucuses work, and how are they different in 2020? Retrieved from TheWashingtonPost.com
- Foley, R., & Cassidy, C. (2020, February 4). Avalanche of issues takes out Iowa plan for high-tech caucus. Retrieved from APNews.com
- Gao, J., Bai, X., Tsai, W., & Uehara, T. (2014). Mobile Application Testing: A Tutorial. Computer, 46-55.
- Gao, J., Tsao, H., & Wu, Y. (2003). Testing and Quality Assurance for Component-based Software. Shanghai: Artech House.
- Goldmacher, S., & Corasaniti, N. (2020, February 4). ‘A Systemwide Disaster’: How the Iowa Caucuses Melted Down. Retrieved from The New York Times
- IECDB. (2020). Schedule B: Expeditures for Iowa Democratic Party. Des Moines: Iowa Ethics and Campaign Disclosure Board.
- Iowa Democratic Party. (2020, February 27). IDP Caucus 2020 Results. Retrieved from Iowa Democratic Party: https://results.thecaucuses.org/
- Irani, Z. (2018, March 28). Five common pitfalls of CI/CD and how to avoid them. Retrieved from InfoWorld
- Jacobsen, J. (2020, February 4). Polk County Democratic leader on Iowa caucus results chaos: ‘Big, big issue’. Retrieved from WeAreIowa.com
- Linares-Vasquez, M., Moran, K., & Poshyvanyk, D. (2017). Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing. 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp. 399-410). Shanghai, China: IEEE.
- Mahmood, R., Esfahani, N., Kacem, T., Mirzaei, N., Malek, S., & Stavrou, A. (2012). A whitebox approach for automated security testing of Android applications on the cloud. 2012 7th International Workshop on Automation of Software Test (AST) (pp. 22-28). Zurich: IEEE.
- Norvell, K. (2020, February 4). What really went wrong on Iowa caucus night? County Democratic chairs reveal the problems. Retrieved from DesMoinesRegister.com
- Payne, K., & Parks, M. (2020, January 14). Despite Election Security Fears, Iowa Caucuses Will Use New Smartphone App. Retrieved from NPR.org
- Pfannenstiel, B. (2020, February 13). Iowa Democratic Party Chairman Troy Price to resign after caucus chaos. Retrieved from DesMoinesRegister.com
- Rayome, A. (2020, February 6). The app that broke the Iowa caucus: An inside look. Retrieved from CNet.com
- Satoh, I. (2004). Software Testing for Wireless Mobile Computing. IEEE WirelessComm, 141-145.
- Scheider, A. (2020, February 4). What We Know About The App That Delayed Iowa’s Caucus Results. Retrieved from NPR.org
- Sharma, P., & Sharma, R. (2018). Design And Development of a Multi Featured iOS Mobile Application using Swift 3. IEEE Journal of Research, 1-8.
- Statt, N. (2020, February 5). Motherboard just published the terrible app that caused chaos at the Iowa caucuses. Retrieved from TheVerge.com
- Whittaker, Z. (2020, February 4). Iowa’s caucus app was a disaster waiting to happen. Retrieved from TechCrunch.com
(*) This blog post initially appeared as an essay for one of my courses at Florida Atlantic University. It is being shared here as a guest blog post with the author’s consent.
All views and opinions expressed in this post are those of the authors and do not necessarily reflect my opinion or the official policy or position of any agency, organization, or company.