Ready to explore the dark side of data science? How did a flawed study by a 19th-century statistician set a discriminatory precedent in the insurance industry? How are algorithms shaping your financial and employment opportunities, and at what cost? Can we ensure a future where data serves us, and not the other way around? If you've ever wondered about the power and pitfalls of data-driven decision-making, this is a must-read. Dive in to explore how data science impacts everything from your insurance premiums to your job prospects, and how it's perpetuating systemic inequalities.

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

On a sweltering August day in 1946, Cleveland Indians' player-manager Lou Boudreau faced a formidable challenge: Ted Williams. Williams had just decimated Boudreau's team with three home runs and eight RBIs in the first game of a doubleheader. Desperate, Boudreau decided to shift his entire defense in the second game, positioning his players exactly where Williams typically hit the ball. This unorthodox strategy paid off, turning many of Williams's hits into outs.

Fast forward to today, and baseball teams use detailed data to make such decisions. Managers know where every player has hit every ball, against various pitchers, in different situations. This data-driven approach to player positioning is part of a broader trend in baseball stat analysis, epitomized by Michael Lewis's "Moneyball." Teams now scrutinize every variable they can quantify to maximize their chances of winning.

But baseball's data models are transparent and fair. Everyone has access to the stats and understands how they're interpreted. The models are continuously updated with new data, and their assumptions are clear. In contrast, many other areas of life are now controlled by opaque and potentially harmful models, known as Weapons of Math Destruction (WMDs).

These models, often used without transparency, can lead to unfair discrimination. For example, recidivism models used in courts predict a convict's likelihood of reoffending based on factors like their neighborhood or family background, which can unfairly lengthen sentences for minorities. These models create destructive feedback loops, reinforcing the conditions they aim to mitigate.

Another everyday example of modeling is a mother's internal model for cooking meals for her family. This dynamic model adapts based on her family's preferences and daily appetites. If formalized, it could be detailed and systematic, but it would still reflect her personal biases and priorities.

Models in our lives, whether predicting baseball outcomes or judicial sentences, reflect the judgments and biases of their creators. They simplify the complex world into manageable predictions, but they must be transparent and fair to avoid becoming WMDs. As we increasingly rely on these models, it's crucial to scrutinize their fairness, transparency, and impact on people's lives.

Can Baseball's Data Revolution Teach Us to Tame 'Weapons of Math Destruction'?

Imagine having a simple morning routine: feeding $2 into the coffee machine before catching the train from Joliet to Chicago. One day, the machine gives back four quarters instead of two. This anomaly repeats a few times over the next month. If this were a tiny glitch in the financial markets, a quant at a hedge fund would spot it, crunch years of data, and train an algorithm to predict this pattern, making millions until the opportunity vanished.

This is the world Cathy, the author, entered at D.E. Shaw, a hedge fund known for its mathematical prowess. The shift from academia to the fast-paced hedge fund life was exhilarating—teaching at Barnard and researching algebraic number theory felt slow in comparison for her. Plus, the pay was three times higher. But what started as an intellectual treasure hunt quickly turned into a front-row seat to financial chaos for her.

At Shaw, quants ruled supreme, with traders being mere functionaries called "executioners." Cathy's team, the "futures group," was tasked with predicting market movements. The environment was secretive, with restricted communication even among colleagues, not unlike a networked cell structure. This isolation led to tension, especially during panics over holidays when strange issues arose, like a $50 million yen position needing urgent attention.

As markets collapsed in 2008, Cathy saw the human impact behind the numbers starkly. The mortgage-backed securities, once boring and reliable, turned out to be toxic, built on flawed mathematical models. These models, designed to obscure risk, were a weapon of financial destruction (WMD). The crisis revealed the ugly truth: the wealth they generated came from people's retirement funds and mortgages, not just abstract numbers.

The collapse exposed the real victims—homeowners losing their homes, Americans losing jobs, and credit card defaults soaring. The financial system's reliance on risky, opaque models was clear to Cathy, yet little changed. Banks and hedge funds continued their practices, barely hindered by new regulations.

Disillusioned, Cathy left Shaw in 2009, determined to fix financial WMDs. She joined RiskMetrics Group, analyzing risk for banks and hedge funds. Yet, she soon realized banks viewed risk reports as mere formalities, focusing more on appearances than actual risk management.

In 2011, Cathy transitioned to the burgeoning field of data science, working at a startup predicting user behavior on travel websites. The parallels with finance were striking for her—both industries valued wealth and success over ethical considerations. The algorithms optimized profits but often at the expense of real people.

As Cathy delved deeper into the Occupy movement and blogged about these issues, her disillusionment grew. She saw the misuse of mathematics was rampant, from finance to Big Data, perpetuating inequality and creating a dystopian future. Alarmed, Cathy quit her job to investigate and challenge the ethical implications of these algorithms, advocating for a more responsible and humane use of data and math.

From Hedge Fund Havoc to Ethical Algorithm Advocate: Cathy's Journey Beyond Wall Street

Imagine sitting down to dinner with friends in San Francisco or Portland and realizing that sharing plates is impossible because everyone is on a different diet. Now, envision that one of these diets, like the Paleo diet, became the national standard. The agricultural economy would be upended, demand for certain foods would skyrocket, and diversity in farming would shrink. This analogy helps us understand the impact of scale—how a single formula, when adopted widely, can distort entire systems. 

This is precisely what happened with the U.S. News & World Report college rankings. In 1983, U.S. News decided to rank 1,800 colleges to guide students and parents in making informed decisions. Initially, the rankings were based on opinion surveys from university presidents, but this led to complaints about fairness. Over time, the magazine incorporated data like SAT scores, graduation rates, and alumni donations into a complex algorithm. 

The rankings quickly became the gold standard, and colleges began to game the system. Baylor University paid for students to retake the SAT to boost scores, and some schools even falsified data. This model, while it pushed colleges to improve in measurable ways, also led to unintended consequences. Colleges focused on metrics that mattered for rankings rather than genuine educational quality.

The rankings created a feedback loop, where lower-ranked schools struggled to attract top students and faculty, leading to further declines. Colleges spent enormous resources to climb the ranks, often at the expense of affordability and accessibility. This has contributed to skyrocketing tuition costs, leaving students burdened with debt.

Even well-intentioned efforts to create alternative rankings, like those proposed by President Obama, face the same pitfalls. Any new model could be gamed, leading to different but equally harmful consequences. For instance, focusing on graduation rates might lead colleges to lower academic standards.

Ultimately, the U.S. News rankings have tied the education system into knots, creating a rat race that benefits the privileged and leaves many students behind. The solution may lie in transparency, allowing students to access the data they need to make informed decisions without being swayed by a single, rigid ranking system.

Are College Rankings by U.S. News & World Report Distorting the Future of Higher Education?

During Cathy's time as a data scientist at Intent Media, a venture capitalist visited to discuss the promising future of targeted advertising. He painted a picture where data-driven ads would seamlessly integrate into our lives, making them so relevant that we'd welcome them. This vision, however, had a dark side, particularly evident in the predatory practices of for-profit colleges like the University of Phoenix and Corinthian College.

These institutions targeted vulnerable individuals with sophisticated marketing strategies, exploiting their hopes and fears. They used extensive data to find "pain points" and promised a better future through education, often leading to massive debt with little to no job prospects. For example, Corinthian College, which charged exorbitant fees for degrees of dubious value, was eventually shut down for fraudulent practices but not before leaving students with $3.5 billion in debt.

The predatory model extends beyond education. Payday loan companies, using similar tactics, charge exorbitant interest rates, trapping people in cycles of debt. These operations are supported by data brokers who sell personal information, often leading to theft and further financial strain.

The rise of machine learning and natural language processing has only intensified these practices. Algorithms now predict and exploit our behaviors and vulnerabilities at an unprecedented scale. For example, misleading ads that falsely promised benefits like financial aid under President Obama's name were used to lure people into for-profit colleges.

This kind of targeted predation perpetuates social inequalities, driving the wealth gap wider. While some benefit enormously—like Gregory W. Cappelli, CEO of Apollo Education Group, who earned $25.1 million in 2011—many more are left in financial ruin.

Regulatory measures are beginning to address these issues, but the sophistication and scale of these predatory practices make them difficult to combat. The need for comprehensive data privacy laws is more pressing than ever to protect vulnerable populations from exploitation.

Targeted Ads and Predatory Practices: The Dark Side of Data-Driven Marketing

In the green hills fifty miles west of Philadelphia lies Reading, Pennsylvania, a city that once thrived on railroads, steel, coal, and textiles. However, as these industries declined, so did Reading's fortunes. 

By 2011, it held the highest poverty rate in the U.S. at 41.3%. The 2008 recession hit Reading hard, slashing tax revenues and forcing significant cuts to the police force. Faced with persistent crime and a shrinking department, Police Chief William Heim turned to technology, investing in PredPol, a Big Data start-up from California. This crime prediction software used historical data to forecast where crimes might occur, allowing officers to patrol high-risk areas more effectively. The results were promising: burglaries dropped by 23% in a year.

PredPol and similar programs like CompStat in New York City and HunchLab in Philadelphia have become popular tools in budget-strapped police departments across the country. These programs focus on "hot-spot" policing, similar to defensive models in baseball that predict where a ball will land based on a player's hitting history. 

However, the inclusion of nuisance crimes like vagrancy and minor drug offenses in these models can create a feedback loop, drawing more police to impoverished neighborhoods and leading to more arrests for minor offenses. This cycle disproportionately affects black and Hispanic communities, as geography often serves as a proxy for race in segregated cities.

The "broken-windows" theory, popularized in the 1980s by George Kelling and James Q. Wilson, posited that addressing minor crimes could prevent more serious ones by maintaining neighborhood order. This led to zero-tolerance policies in the 1990s, which some credited with reducing violent crime, although others pointed to factors like the legalization of abortion and economic growth. Regardless, these policies resulted in the incarceration of millions of young minority men for minor offenses.

Stop-and-frisk policies, like those implemented in New York City under Mayor Bloomberg, relied on police discretion to stop and search individuals deemed suspicious. While touted as effective in reducing crime, the practice overwhelmingly targeted young African American and Latino men, leading to accusations of racial profiling and harassment. A federal judge eventually ruled the policy unconstitutional, citing violations of the Fourth and Fourteenth Amendments.

Predictive policing models like the one used in Chicago, which identified individuals most likely to commit violent crimes, further illustrate the challenges of fairness and efficacy in law enforcement. One young man, Robert McDaniel, found himself on such a list despite never having been charged with a violent crime. This kind of preemptive policing, based on social networks and other data, raises questions about the balance between preventing crime and respecting individual rights.

In Camden, New Jersey, a different approach has shown promise. After becoming the murder capital of the U.S. in 2011, Camden's police department was restructured with a focus not only on reducing crime but also on building community trust. This empathetic approach, emphasizing relationship-building over arrests, could serve as a model for other cities.

As we navigate the complexities of modern policing, the challenge remains to balance efficiency with fairness, ensuring that Big Data and predictive models serve to protect all communities equally without perpetuating existing inequalities.

Can Big Data and Predictive Policing Models Truly Balance Crime Prevention with Individual Rights?

A few years ago, Kyle Behm took a break from Vanderbilt University due to bipolar disorder. After recovering, he applied for a part-time job at Kroger, recommended by a friend. 

Despite his qualifications, Kyle was rejected because he "failed" a personality test by Kronos, a workforce management company. This test, similar to the "Five Factor Model" used in hospitals, assesses traits like extraversion and neuroticism. Kyle's father, Roland, discovered that many companies used these tests and decided to challenge their legality, arguing they might violate the Americans with Disabilities Act (ADA).

These personality tests are part of a $500 million industry, with their use increasing significantly. Despite claims of their effectiveness, research by Frank Schmidt from the University of Iowa shows these tests are poor predictors of job performance. Instead of finding the best candidates, these tests often exclude many, including those with mental health issues, perpetuating unfair barriers.

This workplace bias extends to the digital age, where automatic systems filter job applications. Studies from the University of Chicago and MIT found that resumes with typically white names received 50% more callbacks than those with African American-sounding names. Such biases are embedded in hiring algorithms, which now sift through resumes to find matches, often disadvantaging those without insider knowledge on optimizing their applications.

In the tech industry, companies like Gild analyze "social data" from platforms like GitHub to identify potential hires, showing a shift towards evaluating social capital and community engagement. However, these methods can still overlook significant offline contributions, perpetuating gender and cultural biases.

Ultimately, these automated hiring systems, much like outdated pseudosciences, can entrench existing prejudices under the guise of scientific objectivity. The challenge is to ensure these tools are used to promote fairness and opportunity rather than exclusion.

Digital Gatekeepers: How Modern Hiring Algorithms Perpetuate Bias and Exclusion

In today's corporate America, the term “clopening” has emerged, describing the grueling schedule where an employee closes a store late at night and returns early the next morning to open it. This practice is particularly prevalent among low-wage workers at giants like Starbucks, McDonald’s, and Walmart, leading to sleep deprivation and disrupted lives, especially for those with childcare responsibilities.

The root of such erratic scheduling lies in the rise of Big Data. In the past, family-owned businesses adjusted schedules based on simple observations. Today, sophisticated software analyzes vast amounts of data—from weather to social media trends—to optimize staffing. For instance, a rainy day might increase café traffic, necessitating more employees. However, this relentless pursuit of efficiency often leaves workers with unpredictable and exhausting schedules.

A poignant example is Jannette Navarro, a single mother working at Starbucks. The unpredictability of her shifts, including clopenings, made it impossible for her to manage childcare or continue her education. Her story, featured in a 2014 New York Times article, prompted corporations to promise better scheduling practices. Yet, a follow-up revealed that companies like Starbucks struggled to implement these changes, as their culture and managerial incentives were deeply rooted in efficiency metrics.

The scheduling software is an extension of the “Just in Time” economy, where efficiency is paramount. However, this often comes at the expense of workers' well-being. The algorithms keep staff on minimal hours, avoiding benefits like health insurance, and make it hard to hold a second job, trapping workers in a cycle of poverty and instability.

Moreover, such systems contribute to a chaotic upbringing for many children, as their parents juggle erratic work hours. Studies have shown that this instability negatively impacts children's cognitive and behavioral development.

The story of Tim Clifford, a seasoned New York City teacher, illustrates another dimension of how data-driven models can fail workers. Clifford received wildly fluctuating evaluations from a value-added model, which affected his career and morale, despite no changes in his teaching methods. This flawed system, stemming from statistical errors and misinterpretations, underscores the broader issue of how such models can unjustly impact individuals’ lives.

As society grapples with these challenges, the need for countervailing forces becomes clear. Regulators, unions, and vigilant press coverage are essential to expose and mitigate the adverse effects of these data-driven practices. While the drive for efficiency in capitalism is natural, it must be balanced with justice and fairness for the workforce.

Is Big Data's Quest for Efficiency Turning America's Workforce into Modern-Day Serfs?

In the "good old days," local bankers held the keys to financial opportunities. If you wanted a loan, you’d dress up and hope the banker, well aware of your background and possibly your social circle, deemed you trustworthy. This often locked out minorities and women, who had to seek more open-minded bankers.

In the 1950s, mathematician Earl Isaac and engineer Bill Fair revolutionized lending with their FICO score, focusing purely on financial data like debt load and bill payment records. This color-blind system opened doors to millions and improved accuracy in predicting loan defaults.

Fast forward to today, and algorithms have taken over, for better and worse. While traditional FICO scores are regulated and offer transparency and feedback loops, new "e-scores" derived from zip codes and online behaviors are arbitrary and often discriminatory. Companies like Neustar quickly rank potential customers for profitability, while credit card giants like Capital One analyze browsing patterns to tailor their offers, often disadvantaging those from poorer neighborhoods.

These e-scores can create vicious cycles. For instance, someone from a low-income area might get a lower score and worse credit terms, perpetuating financial struggles. This unregulated data economy also extends to employment, where nearly half of employers check credit reports, potentially trapping people in poverty.

Errors in these automated systems can be devastating. Background checks might confuse individuals with similar names, leading to unfair rejections. While some errors get corrected by diligent humans, the reliance on automated systems often leaves the affected individuals with little recourse.

The rise of data-driven decision-making has also led to troubling practices. Companies now use vast amounts of data, including minor details like spelling on loan applications, to judge creditworthiness, inadvertently penalizing less educated applicants. Peer-to-peer lending platforms initially aimed to democratize finance but have increasingly attracted institutional investors, further embedding secretive and potentially discriminatory e-scores into the financial system.

In essence, while algorithms promised fairness by focusing on relevant data, many modern systems have regressed, relying on proxies that perpetuate past injustices. The challenge remains to balance efficiency with fairness, ensuring that technology serves to uplift rather than entrench inequality.

Algorithmic Bias: How Modern E-Scores Reinforce Financial Inequality

In the late 19th century, Frederick Hoffman, a statistician at Prudential Life Insurance, published a report claiming that African Americans were uninsurable due to their high mortality rates. 

Hoffman's analysis was flawed, confusing correlation with causation and failing to consider the impact of poverty and systemic racism. This report reinforced discriminatory practices like redlining, where insurers and bankers refused to invest in certain neighborhoods, a practice that still exists in subtler forms today.

Fast forward to the modern era, and insurance has evolved with data science. Insurers now use vast amounts of personal data to calculate individual risks, often leading to unfair pricing. For instance, a 2015 Consumer Reports study revealed that auto insurers prioritize credit scores over actual driving records, disproportionately impacting poor and minority communities.

The insurance industry's shift towards individualized risk assessment, while seemingly fair, often perpetuates inequalities. Drivers in poorer neighborhoods or those with irregular commutes face higher premiums. Surveillance technology, like telemetric units in cars, tracks driving behavior to offer discounts, but it often penalizes those who can't afford to avoid risky areas.

In the workplace, wellness programs encouraged by the Affordable Care Act have led to increased surveillance of employees' health. Companies like Michelin and CVS impose penalties on workers who don't meet certain health metrics, often based on flawed statistics like BMI. These programs, while well-intentioned, often fail to improve health outcomes and instead serve as a means for companies to reduce their insurance costs by penalizing employees.

As data becomes more integral to insurance and employment, privacy is increasingly a luxury. Insurers and employers use data to create behavioral tribes, sorting individuals into groups for targeted pricing and monitoring. This data-driven approach risks perpetuating existing inequalities and creating new forms of discrimination, as people are judged and priced based on opaque algorithms.

In a world where data dictates so much of our lives, it's crucial to ensure transparency and fairness in how this information is used. Without oversight, these practices could lead to a future where the vulnerable are further marginalized, and privacy is reserved for the wealthy.

Is Data-Driven Insurance Reinforcing Inequality and Eroding Privacy?

Imagine launching a campaign against weapons of mass destruction (WMDs) and posting a petition on Facebook. Who sees it? That’s up to Facebook’s mysterious algorithm, which decides based on users' behaviors and interests. This algorithmic power isn’t just about social media—Facebook has a significant role in shaping our political landscape.

During the 2010 and 2012 elections, Facebook’s “voter megaphone” experiment encouraged users to vote by showing "I voted" updates from friends. Studies revealed that peer pressure to vote was more influential than civic duty. Facebook’s reach and influence meant their campaign increased voter turnout by 340,000. This kind of sway could swing entire states, even national elections.

Facebook’s ability to influence extends beyond voting. In 2012, researchers altered news feeds to see if it affected users' moods. It did. Positive or negative posts led to corresponding changes in users' emotions, proving that Facebook could subtly manipulate feelings on a massive scale.

Not only Facebook but also companies like Google, Apple, and Amazon have vast amounts of data on us. They could influence our political behavior by tweaking algorithms. It’s a hidden, immense power.

This potential for manipulation isn’t just theoretical. The 2012 Obama campaign used sophisticated data analytics to microtarget voters. By profiling voters based on consumer data, they tailored messages to specific groups, increasing engagement and donations.

This targeting can be problematic. For instance, at a private fundraiser, Mitt Romney's candid remarks about 47% of Americans being "takers" were recorded by a caterer and went viral, damaging his campaign. This incident underscores the importance of secrecy and precise targeting in politics.

Microtargeting has evolved, with campaigns using detailed voter data to craft personalized messages, often invisible to the public. This creates an imbalance where only select voters receive tailored information, potentially influencing their votes without broader scrutiny.

Researchers have shown search engine biases could shift political preferences significantly. In a study, skewed search results changed voter preferences by 20%. Given that 73% of Americans trust search engines, such manipulation could have a profound impact.

Despite the power of microtargeting, most political campaigns still spend heavily on television ads, which reach a broader audience. However, even TV is moving towards personalized advertising, increasing the potential for targeted political messages.

This targeted approach can deepen divisions, as people are fed information that reinforces their beliefs, often without knowing what others are seeing. This secrecy and manipulation can undermine democracy, making some voters disproportionately influential while ignoring others.

While microtargeting could be used for positive purposes, like identifying and helping those in need, its current use often aims to manipulate and divide. Changing the objective from exploitation to assistance could transform these powerful tools into forces for good.

Algorithmic Influence: How Facebook and Tech Giants Shape Democracy

While the evolution of data-driven decision-making has the potential to optimize outcomes, it also carries significant risks of bias and unfairness. The transformation of baseball strategies through data analytics parallels the broader trend where detailed data informs decisions. However, unlike the transparent and continuously updated models in baseball, many algorithms in other sectors are opaque and reinforce existing inequalities, becoming Weapons of Math Destruction (WMDs).

The financial markets and predictive policing illustrate how flawed models can have devastating real-world impacts, from economic collapses to systemic biases in law enforcement. These models often create feedback loops that entrench the problems they aim to solve, disproportionately affecting marginalized communities.

In employment, the rise of personality testing and algorithmic hiring highlights how data can perpetuate biases and exclude qualified candidates, especially those with mental health issues or from disadvantaged backgrounds. Similarly, erratic work schedules driven by efficiency metrics in retail and fast-food industries underscore the human toll of data-driven optimization.

Credit scoring and insurance practices reveal another layer of complexity, where modern algorithms, while initially promising fairness, often revert to discriminatory practices by using proxies that disadvantage the poor and minorities. These systems can trap individuals in cycles of poverty and instability, demonstrating the need for transparency and fairness.

Furthermore, the political realm is not immune to the influence of big data. Social media platforms and search engines have the power to subtly manipulate public opinion and voter behavior, raising concerns about the integrity of democratic processes. The ability to microtarget voters with tailored messages can deepen societal divisions and undermine the principle of equal influence in elections.

Overall, this highlights the dual-edged nature of data and algorithms. While they offer opportunities for efficiency and personalization, their potential to reinforce inequalities and manipulate behaviors necessitates vigilant oversight and a commitment to ethical use. Balancing efficiency with fairness is crucial to ensure that these powerful tools uplift rather than entrench existing disparities.

Home

Trending Books

News & Mags

Relax

Curated

Collections

Key Ideas & Notes

Insights

Highlights & Bookmarks

Listening History