Meet Mappy, a Software System That Automatically Maps Old-School Nintendo Games
Many of the classic Nintendo Entertainment System games are marvels of level design. The introductory moments of Super Mario Bros., for example, famously teach players to search for magic mushrooms by making it difficult to avoid the first one they encounter. In the decades since their initial release, those titles have been extensively explored, their every secret unveiled by avid enthusiasts.
Nevertheless, it has remained difficult to make clear, simple maps of the games and their worlds. As Joseph Osborn, a Ph.D. student in computer science at the University of California–Santa Cruz told me, the conventional method has remained largely unchanged for years.
“Traditionally, game maps are made by taking a lot of screenshots and pasting them together,” Osborn said. “Even in early video game strategy guides or things like that, you’d have somebody with a capture card or a camera set up in front of a TV, and they would have to take pictures multiple times in the level and stitch it together by hand.”
In collaboration with his colleagues Adam Summerville and Michael Mateas, Osborn set out to find an alternative to that labor-intensive and error-prone process. Their answer comes in the form of a software system called Mappy that they describe in a paper available now on the scholarly preprint service arXiv.
In essence, Mappy (not to be confused with the game of the same name) autonomously generates maps of a game’s levels (or, in some cases, its whole world), suturing together long scrolling screens and figuring out how distinct rooms connect to one another. “At a high level, what we do is look at what’s visible on screen at any given time. We record what the player could be seeing, and automatically stitch it together into a large map,” Osborn says. The process allows them to generate images that display the total makeup of a room, even as it changes in response to the player’s actions or other on-screen events.
Mappy is not a fully autonomous system, in that it doesn’t figure out how to actually navigate the game on its own. Osborn and his collaborators first have to feed it information from a playthrough, including data about which buttons were being pushed at any given time. But, as Osborn explains, “During Mappy’s execution, it’s automatically pressing the buttons” based on that information. It even has rules in place to make sure that what it’s seeing is actually part of the playable world and not, for example, a separate cut scene.
Though the process is made easier by the relatively simplicity of the NES’s graphics hardware, Mappy’s work is still difficult. Osborn and his co-authors write, “Mapping out a minute of play (3600 frames) takes between five and six minutes, mainly due to the expensive scrolling detection and game object tracking.” Some of those challenges would likely be amplified if they attempted to apply their method to a more complex console such as the Super Nintendo Entertainment System, which employs graphical layering to generate effects such as fog drifting by in the foreground. Nevertheless, they argue that similar techniques should still be feasible on other game systems.
Osborn imagines an array of possible applications for Mappy. For one, he suggests, it might help those who are experimenting with procedurally generated game levels, giving them a consistent dataset from which to train machine learning algorithms that could produce playable worlds of their own. He also proposes that it might empower a kind of search engine that lets you look up a specific section of a game and then leap directly to that moment, letting you play through your memories.
Whether or not Mappy gets us there, it already stands as a charming and impressive accomplishment. As it demonstrates, modern computer science still has a great deal to teach us about the cartography of our digital past.
Future Tense Newsletter: How Doxxing, Data, and DNA Are Disrupting Our Future
Greetings, Future Tensers,
Technology is changing the way we protest, and that includes how the government reacts to it. On Saturday, the Department of Justice requested a warrant to force protest group #DisruptJ20 to turn over data that would reveal essentially anyone who has visited the website of the group responsible for many protests against Donald Trump’s inauguration. Jacob Brogan writes about how DreamHost, the company that hosts DisruptJ20’s site, is resisting the effort and what this says about Trump’s scary record of collecting data on those in opposition to him.
In the U.K., lawmakers are considering a law that would ban the identification of individuals from anonymized data. That may sound great, but cracking anonymous data can be essential to important security research, explains Nick Thieme. And data security isn’t the only threat to our technological well-being. April Glaser spoke with researchers at University of Washington about how they were able to encode malware into DNA and what potential dangers it could lead to for hospitals and research centers.
- Still a bad idea: If everyone knows blackface is a bad idea, why are companies still creating apps to allow people to change their skin color? April Glaser begs developers to “knock if off.”
- Book smart: Hardcover textbooks are costing underfunded school districts millions of dollars. E-books could provide a cost-saving answer for both students and teachers, writes Lindsey Tepe.
- Ready, set, sew: If you think video games and crafts are on opposite sides of the recreational spectrum, this new game is ready to prove you wrong. Grace Ballenger reports on how new computer interfaces, such as looms, can make gaming more widely accessible.
- Generation ¯\_(ツ)_/¯: Lisa Guernsey explains why parents shouldn’t be panicked by a recent viral story about how smartphones are changing the lives of teens.
- E-market eclipse: Excited for the eclipse? Buy your glasses online? Amazon is offering refunds for some eclipse-viewing products—though it isn’t clear whether they’re all faulty.
For Future Tense
The Department of Justice Demands Records on Every Visit to Anti-Trump Protest Site DisruptJ20
If you’ve visited the website DisruptJ20, which helped organize protests during the inauguration of Donald Trump, the Department of Justice is interested in learning more about you.
On Saturday, a judge in the Superior Court of the District of Columbia approved a search warrant that would require DreamHost, DisruptJ20’s provider, to turn over a wide range of information about the site and its visitors. In addition to information about the site’s creators, the DOJ demands “logs showing connections related to the website, and any other transactional information, including records of session times and duration.” In short, the government is looking for records of everyone who even visited the site, which is to say it's effectively compiling info on those who showed even a modicum of interest in protesting the administration.
DreamHost is resisting the effort. In a blog post, the company acknowledges that it has “no insight into the affidavit for the search warrant (those records are sealed).” Nevertheless, DreamHost also writes that its general counsel “has taken issue with this particular search warrant for being a highly untargeted demand that chills free association and the right of free speech afforded by the Constitution.” As it goes on to explain, turning over the requested records would mean providing 1.3 million visitor IP addresses along with “contact information, email content, and photos of thousands of people.”
As ZDNet notes, “Several purported members of [DisruptJ20] were arrested for alleged violent conduct during the protests.” It links to a Washington Post article from January that claims, “Police said in court filings that the damage caused by the group was in excess of $100,000.”
In advance of the inauguration itself, however, the organizers claimed on their site laid out sweeping, but still legal, goals. “We’re planning a series of massive direct actions that will shut down the Inauguration ceremonies and any related celebrations–the Inaugural parade, the Inaugural balls, you name it,” they claimed.
DreamHost’s blog post stresses that those who came to the site in search of information about such activities had every right to do so, just as they had every right to protest the inauguration. As such, it’s not clear why the DOJ would need their IP addresses and other related data. “That information could be used to identify any individuals who used this site to exercise and express political speech protected under the Constitution’s First Amendment,” the post reads. “This is, in our opinion, a strong example of investigatory overreach and a clear abuse of government authority.”
This is not, of course, the first time that the Trump administration has sought sweeping information about citizens. In late June, the DOJ demanded massive amounts of voter registration data, information that many states refused to provide. While that may have been part of an ongoing effort to purge voter rolls, this new warrant is troubling in part because it suggests the Trump administration is also actively gathering records about its opponents.
We’ll likely know more after a hearing about the request, currently scheduled for Friday, Aug. 18 in Washington.
A Proposed Anti-Doxxing Law in the U.K. Could Make Personal Data Less Secure
Personal medical information from 1 in every 4 Americans has been stolen. On average, there were three data breaches in the U.S. every day during 2016. People outed by trails of left-behind data have taken their own lives. “Better” outcomes of doxxing include relentless abuse and death threats.
Against this backdrop comes the United Kingdom’s wise move toward new data protection laws. As part of this process, a proposed law would ban “intentionally or recklessly re-identifying individuals from anonymised or pseudonymised data.” Digital Minister Matt Hancock told the Guardian the law, if implemented, will “give people more control over their data, [and] require more consent for its use.” This shift recognizes that the threat of doxxing can chill your comfortable internet browsing. But, counterintuitively, it also makes your data less secure.
Whether your data can be truly anonymous is up for debate. But “anonymous data” usually refers to information that cannot be associated with the person who generated it. This kind of data is vital to scientific research. When I conducted cancer research, I used free, open-access, genetic data from real people with real diseases. Having the same, constantly updated, genetic data freely available to all scientists creates a baseline, which can help verify results, allow researchers to avoid echo chambers, and aid reproducibility.
The issue is that “anonymous” data can often be de-anonymized. In 2006, Netflix released the data of 500,000 customers in the interest of crowdsourcing improvements for their prediction algorithm. As was the standard at the time, they removed all personally identifiable information from the data, releasing only customers’ movie ratings, thinking this would keep their customers’ identities hidden. They were wrong. Arvind Narayanan and Vitali Shmatikov, researchers from the University of Texas at Austin, compared movie ratings from the Netflix dataset with publicly available IMDB data, and because movie ratings are very personal—I can’t imagine anyone other than me 5-starring The Man With the Iron Fists, Hunter x Hunter, and My Cousin Vinny—they were able to match names of IMDB users with Netflix accounts.
This research demonstrates two things: 1) Anonymous data often isn’t, and 2) it can be critically important for researchers to—as the U.K. might put it—“intentionally … re-identify individuals from anonymised or pseudonymised data.” Researchers need the ability to break privacy systems. When the options are a good guy picking your lock to convince you it’s broken, or a bad guy picking your lock to steal your passport, the choice is clear. The analysis from the University of Texas at Austin led to lawsuits against Netflix. More importantly, it warned the entire data industry that the privacy methods Netflix was using at the time were insufficient and should no longer be used.
The U.K.’s Information Commissioner’s Office anonymization code of practice considers data “anonymized” if it is not “reasonably likely” to be traced back to an individual. “Reasonably likely” isn’t well defined, but if only one research team in the world can de-anonymize the data, the data probably falls under their definition of anonymous. Under this definition, and the newly proposed laws, the Texas researchers would have committed a crime punishable by an apparently unlimited fine. Shmatikov, who is now a professor at Cornell University, views the U.K.’s proposed law as perfectly wrong. He told me that the kind of research that will keep people safe is “exactly the kind of activity that [the U.K.] is trying to penalize.” He later said he would not have conducted his research if it were penalized by this sort of law.
To be clear, I do not unequivocally support all research that seeks to break security systems. The leadership from the unnamed company that sold the FBI the software used break into the San Bernardino shooter’s iPhone should be tarred, feathered, and marched down a busy street by a matronly nun ringing a bell. It didn’t require the FBI to release the details of that security flaw, so the exploit might be sitting in your pocket right now, waiting to be abused. A white-hat hacker always releases his or her secrets.
But laws are not the cure for doxxing. The ongoing research of white-hat researchers like Shmatikov will not stop people from invading each other’s privacy, but it will keep all of our data safer.
And the new laws do offer protection for research in other related areas. For instance, citizens may be barred from having their data removed from university datasets if the institution can argue the data is “essential to the research.” It would be extremely easy to create a similar research exception in the deanonymization statutes. Now, Shmatikov doesn’t see this as a perfect solution. In his eyes, good security research can be done by people outside academia. He also believes the harm from deanonymization occurs when personal information is shared across the internet, not when someone is deanonymized.
Still, the ICO has two options. It could add an exception to the deanonymization laws for researchers who sequester the data they deanonymize, whether the researchers are academic or not. Or, it could penalize the sharing of deanonymized data, rather than its creation. (Remember, the U.K. doesn’t have the First Amendment.) Both paths would disincentivize internet hordes from making the private public, without handcuffing researchers from improving the technology that will actually help.
New Fabric Interfaces Weave Together Textiles and Computers in Unexpected Ways
Imagine sitting in front of a “Choose Your Own Adventure” tale. It’s just like the book format that you might remember from childhood, but instead of being on a page, it unfolds on the screen. You read through several paragraphs of text, and are presented with four colorful options to shift the direction of the story. Here’s the magical part: You select your preferred option by passing a certain color thread through a loom. That’s right—this game is loom-controlled.
When most people think of interacting with computers, they think of traditional interfaces, like a computer mouse, keyboard, or cellphone. However, as computer capabilities evolve, so do the variety of interfaces. One intriguing and nontraditional idea is to use fabric-based interfaces for fun or to achieve a goal. Because all of us are familiar with fabric as a material, the systems also aim to be more inclusive.
One version of a cloth-based computer interface is a video game system called Loominary, which uses a tabletop loom as an interface to weave a scarf.
WannaCry Hero Pleads Not Guilty to Malware Charges
On Monday, cybersecurity researcher Marcus Hutchins, better known by the nom-de-keyboard MalwareTech, pleaded not guilty to creating and distributing malware, Motherboard reports.
As April Glaser has previously explained in Slate, Hutchins rose to international prominence after he helped stop the WannaCry ransomware attack earlier this year. Accordingly, it came as a shock to many when he was arrested in August for his alleged contributions to a banking Trojan called Kronos, a piece of banking malware seemingly unrelated to WannaCry.
According to Motherboard, “[T]he prosecution said that Hutchins had admitted ‘that he was the author of the code that became the Kronos malware’ when he spoke to FBI agents” in an earlier hearing. Kronos, which first appeared in mid-2014 and reportedly sold for $7,000, primarily targeted banks in the United Kingdom and other countries, leading some, Motherboard writes, to ask why “a British researcher being indicted in the United States for a malware that apparently had no American victims.”
Even if Hutchins did contribute to the Kronos code, as prosecutors allege, it’s still not clear what, if any, evidence they have that he helped market it. The Guardian cites Jake Williams, a cyberscurity researcher who suggests it’s unlikely that Hutchins would have done so, since he refused payment for a legitimate project they worked on together around the time Kronos was active. “I have a hard time picturing him refusing money for work from me but at the same time taking money for illegal activities,” Williams tells the Guardian. More recently, as the paper also notes, Hutchins donated reward money that he received for helping shut down WannaCry.
For now, at least, Hutchins is out on bail and will, Motherboard reports, “be allowed full internet access so he can continue to work as a security researcher.” His trial is scheduled for October.
Amazon Is Refunding Some (Possibly Defective) Eclipse-Viewing Products
If you recently ordered eclipse glasses from Amazon, you may have received a disquieting email over the weekend. The message indicates that the company is offering refunds for many eclipse glasses and other solar-viewing products on the grounds that they may not meet safety standards. And while Amazon is crediting those customers for the full amount of their purchase, it’s not requesting that they return the defective products.
Space.com notes that this recall has been weeks in the making in the making: “Counterfeit solar eclipse glasses were first reported in late July when the nonprofit American Astronomical Society warned that some glasses falsely claimed to meet the international standard, known as ISO 12312-2, for safe solar viewing.” The AAS subsequently published a set of updated guidelines for safe viewing of the event. If you run a search for eclipse glasses on Amazon today, the site pops up a warning notice before results populate, noting that it has “implemented safety requirements for all filters for solar viewing offered on the site.”
Click through, and you’ll find that the message is directed at vendors rather than the site’s customers. It requires that they provide a range of information, including documentation about the manufacturers of the glasses and test results from an accredited laboratory.
Some vendors affected by the recall are reportedly upset. According to the Portland-based NBC affiliate KGW, “Two separate Amazon solar eclipse glasses sellers called KGW Saturday following Amazon’s recall. Both said their products were verified as safe and manufactured by companies approved by NASA. But their glasses are still under a recall.” One Amazon customer showed Slate a refund notice for a product on the AAS’s list of reputable vendors.
In some cases, though, things may be complicated. As KGW notes, AgneaAstro, another of the affected vendors, was selling “eclipse glasses come from two NASA-approved sellers: Thousand Oaks Optical in Arizona and Baader Planetarium in Germany.” But as the American Astronomical Society reports, one line of the Baader Planetarium’s shades is eclipse-ready while the other is not.
Ultimately, this recall speaks to the complex balancing act that Amazon attempts every day. Even as thousands of vendors operate through its site with minimal supervision, the company remains the face of almost every product that passes through its interface. In this case, it’s likely taking a sizable financial hit as it works to sustain that unwieldy system.
The AAS, for its own part, recommends that you buy your eclipse glasses in person, since even some of its reputable vendors “use different suppliers for their websites than they do for their stores.” Maybe brick-and-mortar retail really will survive after all!
Hackers Can Now Store Malware on DNA
Your DNA is packed with a lot of information, but typically it comes pre-loaded. It’s your DNA, after all. But increasingly, scientists and researchers are learning ways to load all kinds of additional data onto DNA made in labs—including computer viruses.
New research that came out Thursday from the University of Washington demonstrates how hackers were able to encode malware into a short strand of DNA. It was then used to infect and take control of a computer connected to a DNA sequencing machine. And as if that alone isn’t alarming enough, the malware-infused DNA was programmed to launch the virus on its own when the DNA is analyzed. So instead of malware being spread through an email attachment, their research showed that “DNA can be a method of compromising the computer,” said Peter Ney, one of the Ph.D. students who worked on the research team at UW.
As you can probably tell, this hack was very complicated to carry out and, of course, it requires your computer to be connected to a genetic sequencing machine, which you presumably don’t have in your home office.
But the threat here isn’t (currently) to personal computers—it’s to research facilities and DNA sequencing labs. Hypothetically, a malicious actor could send DNA with malware in it to a lab for sequencing. When it’s run through the sequencing system, the malware would be unleashed onto the corresponding computer and take control over that system, where it can read future DNA sequences or even alter genetic data. This isn’t a hack that poses a near danger since it was so complicated to carry out, the researchers say. Still, gene sequencing’s price is dropping rapidly. In 2001, it cost about $100 million to sequence a genome. Now it costs about $1,000, and it will only get cheaper, particularly as the quest for precision medicine continues. The U.S. has a goal of sequencing the DNA of 1 million Americans to learn how to individually tailor medical treatment. So this hack could be a sign of larger problems down the road.
Malware isn’t the only thing that can be coded into DNA. Earlier this year, scientists at Harvard discovered a way to upload a GIF into a DNA sample. The Universal Declaration of Human Rights and 100 works of classic literature have been converted into digital data uploaded onto synthetic DNA. And DNA, it turns out, may be a viable alternative to the ways data is stored now, as companies typically keep digital records on servers in massive, power-hungry data centers. Hard drives and flash drives have relatively short shelf lives and can crash over the course of a few years, but DNA, on the other hand, could potentially store information for thousands of years.
Theoretically, it may even be possible to store files on our own DNA, allowing humans to carry malware or secret files. But it wouldn’t necessarily be easy to extract—not yet anyway.
“Once the DNA is inside you, it would be very difficult to pick that DNA out and amplify that specific piece,” says Yaniv Erlich, a professor at Columbia who previously has shown it’s possible to upload a computer virus or an Amazon gift card on DNA, though he didn’t actually carry out a hack, as the researchers at UW did.
“You might be able to store data on your skin or microbiome,” Erlich said. “If we take bacteria from your own skin, it could carry a file. We can also embed DNA with data into food.” Elrich has also worked on research aimed at making miniature DNA sequencers that are much smaller and more affordable.
If a piece of DNA inside you carries digital files, it doesn’t mean that DNA is going to reproduce all over your body. To do that, the file carrying DNA would probably have to be put into an embryo, and right now most of that kind of research is focused on curing genetic disease.
But the more researchers find ways to store information on DNA and or even hack into sequencer systems to alter results, the more this emerging field of research will need to think about security, too. Anything, it seems, can be hacked—even DNA.
Correction, August 10, 2017: This article originally misspelled Peter Ney’s last name.
Hey, Tech Companies: Knock It Off With the Apps That Let People Change Their Skin Color
For a brief moment on Wednesday, FaceApp—the app that went viral in April for taking a photo of someone’s face and making them look older, younger, more masculine, or more feminine—had a new feature that let users see how they would look if they were a different race. That means white users could make their faces black, and black users could whiten their skin. There was also an Asian filter as well as one to make people look Indian.
Within hours, FaceApp had pulled down the new feature. That’s good. But really: Who thought it was a good idea? Did they actually think that this would go well?
You may be thinking to yourself, This sounds awfully familiar. Haven’t we been through this nonsense before? Yes. Yes, we have. The new race categories were an update to the app. Back in the spring, when everyone first ran to download and play with FaceApp, it had a feature called Spark that lightened users’ skin tone. Spark was removed as an option on FaceApp after users took to social media calling the company out for whitewashing.
At the time, FaceApp called what happened “an unfortunate side effect of the underlying neural network caused by the training set bias, not intended behavior”—a rather ridiculous excuse.
me and my three ethnically diverse half brothers unequivocally condemn the new faceapp filters pic.twitter.com/uMNfIrb73f— Alex Nichols (@Lowenaffchen) August 9, 2017
But somehow, for some mysterious reason, Wireless Lab, the company that developed FaceApp, apparently decided to lean in to its race filters, adding even more options than just skin whitening. But adding a blackface filter, an Asian face filter, and an Indian face filter didn’t level things out. At all.
Perhaps the Russia-based company wasn’t aware of the history of racial parodying on our side of the pond. But considering the outrage that ensued after its whitewashing filter, one would think FaceApp had learned that making an app to change people’s race will never, ever, ever, ever be OK. Almost every country in the world continues to grapple with the racist aftermath of hundreds of years of colonial history. There’s too much very real and very recent history of people trying to pass or change their race in order to survive or live a more privileged life.
To be fair, FaceApp isn’t the only photo filter social media app that’s toyed with race. Last year on April 20, the national holiday for marijuana enthusiasts in the U.S., Snapchat offered users a filter that made them look black with dreadlocks. It was obviously a nod to Bob Marley but it was nevertheless a blackface filter, and the company got significant blowback, too. Not that Snap learned its lesson. Later that year, in August, Snapchat released another alarming filter, an “anime” option that made people’s face look more yellow, their eyes look more narrow, and teeth bigger. And yet again, the company was chastised for making a racist filter.
The amazing thing is that this keeps happening—that these features make it to the market at all. Please, tech companies: Knock it off with the race-based face filters. There’s nothing cool about making someone look like a different race. Stick to flower halos, glasses, hats, and animal-ear filters. So many people who use the internet are plenty racist enough already. There’s no need to help.
Future Tense Newsletter: How Facebook Is Coming to Terms With Its Political Influence
Greetings, Future Tensers,
Nine months since the U.S. presidential election, Facebook is still in the early stages of coming to terms with its political influence. As part of its ambitions to create a platform that goes beyond connecting friends and family, it has unveiled new features aimed at connecting government representatives to the citizens they’re elected to serve. But Faine Greenwood questions whether we can trust a social media network’s attempts to better our representative government. She writes, “Not only is it unclear whether these tools will actually foster meaningful engagement, it’s also questionable whether private companies like Facebook—even if they’re well-meaning—should be trusted to keep democracies’ best interests at heart.”
Relatedly, Will Oremus took stock of Facebook’s most recent attempts to combat fake news—a major problem during the presidential campaign, of course. He points out that the company no longer uses the term “fake news” in press releases. Facebook has also launched a “Related Articles” feature to the news feed as a way to give users “easier access to additional perspectives and information, including articles by third-party fact checkers.”
While Facebook wants to bring some tech to government, government was thinking about tech this past week. The Senate is considering (surprisingly) sensible security legislation for internet of things devices. In more concerning news, Immigrations and Customs Enforcement is investigating the potential to use predictive policing technology—which critics say depends on biased data—in its new “extreme vetting” program.
Other things we read this week while trying to figure out why our stash of bitcoin doubled last week:
- Coding boot camps: Although coding boot camps were supposed to be the next big thing in higher education, the recent closures of two high-profile camps reveals the industry is struggling to find a sustainable business model.
- Wacky neural networks: Jacob Brogan talks with Janelle Shane about how (and why) she teaches neural networks to name craft beers and come up with knock-knock jokes.
- Paper technology: Rachel Adler chronicles how improvements made to the printing press and the manufacturing of paper contributed to major societal changes in the 19th century and heralded the beginnings of mass media.
- Sex-trafficking bill: Members of Congress have introduced a bill that would make it easier to punish online service providers when criminals use their services for sex trafficking. But Mike Godwin says that the bill, as currently written, could threaten internet freedom and innovation.
- Religious apps: Aneesa Bodiat explores the benefits and pitfalls of using new technology to satisfy spiritual and religious needs.
For Future Tense