Improving Photo Search: A Step Across the Semantic Gap

Posted by Chuck Rosenberg, Image Search Team

Last month at Google I/O, we showed a major upgrade to the photos experience: you can now easily search your own photos without having to manually label each and every one of them. This is powered by computer vision and machine learning technology, which uses the visual content of an image to generate searchable tags for photos combined with other sources like text tags and EXIF metadata to enable search across thousands of concepts like a flower, food, car, jet ski, or turtle.

For many years Google has offered Image Search over web images; however, searching across photos represents a difficult new challenge. In Image Search there are many pieces of information which can be used for ranking images, for example text from the web or the image filename. However, in the case of photos, there is typically little or no information beyond the pixels in the images themselves. This makes it harder for a computer to identify and categorize what is in a photo. There are some things a computer can do well, like recognize rigid objects and handwritten digits. For other classes of objects, this is a daunting task, because the average toddler is better at understanding what is in a photo than the world’s most powerful computers running state of the art algorithms.

This past October the state of the art seemed to move things a bit closer to toddler performance. A system which used deep learning and convolutional neural networks easily beat out more traditional approaches in the ImageNet computer vision competition designed to test image understanding. The winning team was from Professor Geoffrey Hinton’s group at the University of Toronto.

We built and trained models similar to those from the winning team using software infrastructure for training large-scale neural networks developed at Google in a group started by Jeff Dean and Andrew Ng. When we evaluated these models, we were impressed; on our test set we saw double the average precision when compared to other approaches we had tried. We knew we had found what we needed to make photo searching easier for people using Google. We acquired the rights to the technology and went full speed ahead adapting it to run at large scale on Google’s computers. We took cutting edge research straight out of an academic research lab and launched it, in just a little over six months. You can try it out at photos.google.com.

Why the success now? What is new? Some things are unchanged: we still use convolutional neural networks -- originally developed in the late 1990s by Professor Yann LeCun in the context of software for reading handwritten letters and digits. What is different is that both computers and algorithms have improved significantly. First, bigger and faster computers have made it feasible to train larger neural networks with much larger data. Ten years ago, running neural networks of this complexity would have been a momentous task even on a single image -- now we are able to run them on billions of images. Second, new training techniques have made it possible to train the large deep neural networks necessary for successful image recognition.

We feel it would be interesting to the research community to discuss some of the unique aspects of the system we built and some qualitative observations we had while testing the system.

The first is our label and training set and how it compares to that used in the ImageNet Large Scale Visual Recognition competition. Since we were working on search across photos, we needed an appropriate label set. We came up with a set of about 2000 visual classes based on the most popular labels on Google+ Photos and which also seemed to have a visual component, that a human could recognize visually. In contrast, the ImageNet competition has 1000 classes. As in ImageNet, the classes were not text strings, but are entities, in our case we use Freebase entities which form the basis of the Knowledge Graph used in Google search. An entity is a way to uniquely identify something in a language-independent way. In English when we encounter the word “jaguar”, it is hard to determine if it represents the animal or the car manufacturer. Entities assign a unique ID to each, removing that ambiguity, in this case “/m/0449p” for the former and “/m/012x34” for the latter. In order to train better classifiers we used more training images per class than ImageNet, 5000 versus 1000. Since we wanted to provide only high precision labels, we also refined the classes from our initial set of 2000 to the most precise 1100 classes for our launch.

During our development process we had many more qualitative observations we felt are worth mentioning:

1) Generalization performance. Even though there was a significant difference in visual appearance between the training and test sets, the network appeared to generalize quite well. To train the system, we used images mined from the web which did not match the typical appearance of personal photos. Images on the web are often used to illustrate a single concept and are carefully composed, so an image of a flower might only be a close up of a single flower. But personal photos are unstaged and impromptu, a photo of a flower might contain many other things in it and may not be very carefully composed. So our training set image distribution was not necessarily a good match for the distribution of images we wanted to run the system on, as the examples below illustrate. However, we found that our system trained on web images was able to generalize and perform well on photos.

A typical photo of a flower found on the web.

A typical photo of a flower found in an impromptu photo.

2) Handling of classes with multi-modal appearance. The network seemed to be able to handle classes with multimodal appearance quite well, for example the “car” class contains both exterior and interior views of the car. This was surprising because the final layer is effectively a linear classifier which creates a single dividing plane in a high dimensional space. Since it is a single plane, this type of classifier is often not very good at representing multiple very different concepts.

3) Handling abstract and generic visual concepts. The system was able to do reasonably well on classes that one would think are somewhat abstract and generic. These include "dance", "kiss", and "meal", to name a few. This was interesting because for each of these classes it did not seem that there would be any simple visual clues in the image that would make it easy to recognize this class. It would be difficult to describe them in terms of simple basic visual features like color, texture, and shape.

Photos recognized as containing a meal.

4) Reasonable errors. Unlike other systems we experimented with, the errors which we observed often seemed quite reasonable to people. The mistakes were the type that a person might make - confusing things that look similar. Some people have already noticed this, for example, mistaking a goat for a dog or a millipede for a snake. This is in contrast to other systems which often make errors which seem nonsensical to people, like mistaking a tree for a dog.

Photo of a banana slug mistaken for a snake.

Photo of a donkey mistaken for a dog.

5) Handling very specific visual classes. Some of the classes we have are very specific, like specific types of flowers, for example “hibiscus” or “dhalia”. We were surprised that the system could do well on those. To recognize specific subclasses very fine detail is often needed to differentiate between the classes. So it was surprising that a system that could do well on a full image concept like “sunsets” could also do well on very specific classes.

Photo recognized as containing a hibiscus flower.

Photo recognized as containing a dahlia flower.

Photo recognized as containing a polar bear.

Photo recognized as containing a grizzly bear.

The resulting computer vision system worked well enough to launch to people as a useful tool to help improve personal photo search, which was a big step forward. So, is computer vision solved? Not by a long shot. Have we gotten computers to see the world as well as people do? The answer is not yet, there’s still a lot of work to do, but we’re closer.

Art, Copy & Code: sending kisses around the world

Thanks to modern technology you can connect with your loved ones by sending a quick note, a photo of your cat, even a smile :) around the world in seconds. But one of humanity’s most iconic forms of communication—the kiss—has been left out in the cold. Now, though, you can send a kiss to anyone, anywhere in the world, through Burberry Kisses, a new campaign from Burberry and Google. And not just any kiss, but your kiss.

To get started, simply visit kisses.burberry.com and pucker up in front of your webcam (this works best on Chrome). Using unique kiss-detection technology, the site will detect the outline of your actual lips, which you can choose to dress up with a Burberry lipstick color. If you’re using your touch screen mobile or tablet, you can actually kiss your screen (you might want to wipe it off first) and your lip outline will be taken from there. After that, write a short message and send it to someone from your Google+ friends list or via email. Then sit back and see the envelope with your message fly from your city to the receiver’s destination across a 3D landscape. The receiver gets an email, from which they can see the same journey, read your message and hopefully respond with a kiss of their own.



For an example, see this message I sent to my mom this morning. All the kisses being sent around the world can be seen in a real-time interactive map, capturing the story of the world’s love. You don’t have to kiss and tell: all kisses are private unless you choose to share.

Burberry Kisses is the latest campaign in our Art, Copy & Code project, an ongoing series of brand partnerships to re-imagine how brands tell stories in a connected world. With this project, we’ve tried to create a beautiful experience that comes to life across all screens, and helps connect you to the people who are important to you, wherever they are. For more details on the campaign, see our agency blog or visit our website.

Posted by Aman Govil, Art, Copy & Code Project Lead

Asking the U.S. government to allow Google to publish more national security request data

This morning we sent the following letter to the offices of the Attorney General and the Federal Bureau of Investigation. Read the full text below. -Ed.

Dear Attorney General Holder and Director Mueller

Google has worked tremendously hard over the past fifteen years to earn our users’ trust. For example, we offer encryption across our services; we have hired some of the best security engineers in the world; and we have consistently pushed back on overly broad government requests for our users’ data.

We have always made clear that we comply with valid legal requests. And last week, the Director of National Intelligence acknowledged that service providers have received Foreign Intelligence Surveillance Act (FISA) requests.

Assertions in the press that our compliance with these requests gives the U.S. government unfettered access to our users’ data are simply untrue. However, government nondisclosure obligations regarding the number of FISA national security requests that Google receives, as well as the number of accounts covered by those requests, fuel that speculation.

We therefore ask you to help make it possible for Google to publish in our Transparency Report aggregate numbers of national security requests, including FISA disclosures—in terms of both the number we receive and their scope. Google’s numbers would clearly show that our compliance with these requests falls far short of the claims being made. Google has nothing to hide.

Google appreciates that you authorized the recent disclosure of general numbers for national security letters. There have been no adverse consequences arising from their publication, and in fact more companies are receiving your approval to do so as a result of Google’s initiative. Transparency here will likewise serve the public interest without harming national security.

We will be making this letter public and await your response.

David Drummond
Chief Legal Officer

90 ideas to change the world: Announcing Google Science Fair Finalists

Many great scientists developed their curiosity for science at an early age and in January we called on the brightest young minds from around the world to send us their ideas to change the world. Our 2013 Google Science Fair attracted an exciting and diverse range of entries, with thousands of submissions from more than 120 countries.

After a busy few months for the judges, we’re ready to reveal our 90 regional finalists for the 2013 Google Science Fair. It was no easy task selecting these projects, but in the end their creativity, scientific merit and global relevance shined through. This year’s finalists projects range from using banana peels to produce bioplastics to research into a green treatment for contaminated water. Other projects include a study on the effects of video gaming on the cognitive function of the brain and the evaluation of wireless transmission of electricity.

The 90 Regional Finalists come from all over the world.

For the second year, we’ll also be recognizing the Scientific American Science in Action Award. This award honors a project that makes a practical difference by addressing an environmental, health or resources challenge. From the 90 finalists’ projects, 15 were nominated for this year’s award.

On June 27 we’ll announce the 15 global finalists and the winner of the Science in Action Award. These young scientists will then be flown to Google’s California headquarters for the last round of judging and a celebratory event on September 23.

Thank you to everyone who submitted a project—we really appreciate all your hard work. Congratulations to our 90 regional finalists!

Posted by Sam Peter, Google Science Fair team

How green is the Internet?

More than ever, people are using the Internet to shop, read, listen to music and learn. And businesses rely on Internet-based tools to operate and deliver their services efficiently. The Internet has created all kinds of new opportunities for society and the economy—but what does it mean for the environment?

We’ve been working to answer that question and enlisted the help of Lawrence Berkeley National Laboratory (Berkeley Lab) to gather more data. Their study (PDF), released today, shows that migrating all U.S. office workers to the cloud could save up to 87 percent of IT energy use—about 23 billion kilowatt-hours of electricity annually, or enough to power the city of Los Angeles for a year. The savings are associated with shifting people in the workforce to Internet-based applications like email, word processing and customer relationship software.

These results indicate that the Internet offers huge potential for energy savings. We’re especially excited that Berkeley Lab has made its model publicly available so other researchers and experts can plug in their own assumptions and help refine and improve the results.

Of course, understanding the impact of shifting office applications to the cloud is only part of the story, which is why last week we hosted a summit called “How Green is the Internet?” to explore these questions in greater detail. At the summit, experts presented data on how the growth of Internet infrastructure, including devices like phones and tablets, can impact the environment. We also saw great excitement about the potential for entirely new Internet-enabled tools in areas like transportation, e-commerce and digital content to deliver huge energy and carbon savings. We’ve posted the videos from those sessions and invite you to take a look.



One of our goals in hosting the summit and supporting the Berkeley Lab study was to identify and encourage new research on this topic. We’ll continue to work to answer some of these questions, and we hope others will too.

Posted by Michael Terrell, Senior Policy Counsel, Energy & Sustainability

2013 Google PhD Fellowships: 5 Years of Supporting the Future of Computer Science

Posted by Michael Rennaker, Google University Relations

We are extremely excited to announce the 2013 Global Google PhD Fellows. From all around the globe, these 39 PhD students represent the fifth class in the program’s history, a select group recognized by Google researchers and their institutions as some of the most promising young academics in the world. As we welcome the newest class of PhD Fellows, we take a look back at the program’s roots and hear from two past recipients.

In 2009, Google launched its PhD Fellowship Program, created to recognize and support outstanding graduate students pursuing work in computer science, related disciplines or promising research areas. In its inaugural year, 13 United States PhD students were awarded fellowships, drawn from an extremely competitive pool of applicants. The global program now covers Europe, China, India and Australia and continues to draw some of the best young researchers, reflecting Google’s commitment to building strong relations with the academic community.

Among those first recipients of the fellowship award are 2009 PhD Fellow Roxana Geambasu, Visiting Professor in the Computer Science Department at Columbia University, and 2010 European Doctoral Fellow Roland Angst, Visiting Assistant Professor at Stanford University and affiliated with the Max Planck Center for Visual Computing and Communication. As early recipients of the award, Roxana and Roland reflect on the impact that the Google Fellowship program had on their careers.

For Roxana, the fellowship provided the tools and connections that helped lay the foundation for her academic career. She believes industrial fellowship programs are very important, as they give students an opportunity to interact more closely with industry.

“Beyond the financial support, I think that the fellowship impacted my career in many important ways. First, the Google fellowships are regarded as highly competitive, so receiving the award was probably a big plus on my resume when I was interviewing for faculty positions.”

“Second, the award yielded a mentor within Google, Brad Chen, with whom I've kept in touch ever since, as well as opportunities to visit the campus, deliver talks and meet Google engineers. Brad and I continue to meet at conferences and discuss my work, his work and (of late) the work of my students; it’s through that relationship I’m exposed to new people from Google and gain valuable advice about faculty award opportunities.”

Roland Angst credits the award with the ability to lighten his teaching load and instead focus on his research, which ultimately prepared him for his future academic career. Like Roxanna, Roland states that the fellowship also gave him the opportunity to establish connections with people working in related topics in industry.

“In my view, programs such as the Google Fellowship Awards represent an important and integral link between industry and universities. Firstly, such programs increase the awareness in the academic world for relevant problems in industry. Secondly, these programs allow the IT industry to express their gratitude to the educational services provided by the universities on which the IT industry heavily relies on."

We welcome the latest recipients of the Global Google PhD Fellowships for 2013 with great excitement and high expectations. Recognized for their incredible innovation, creativity and leadership, we are very happy to support these excellent PhD students and offer our sincere congratulations.

Google Maps and Waze, outsmarting traffic together

We’ve all been there: stuck in traffic, frustrated that you chose the wrong route on the drive to work. But imagine if you could see real-time traffic updates from friends and fellow travelers ahead of you, calling out “fender bender...totally stuck in left lane!” and showing faster routes that others are taking.

To help you outsmart traffic, today we’re excited to announce we’ve closed the acquisition of Waze. This fast-growing community of traffic-obsessed drivers is working together to find the best routes from home to work, every day.

The Waze product development team will remain in Israel and operate separately for now. We’re excited about the prospect of enhancing Google Maps with some of the traffic update features provided by Waze and enhancing Waze with Google’s search capabilities.

We’ll also work closely with the vibrant Waze community, who are the DNA of this app, to ensure they have what’s needed to grow and prosper.

The Waze community and its dedicated team have created a great source of timely road corrections and updates. We welcome them to Google and look forward to working with them in our ongoing effort to make a comprehensive, accurate and useful map of the world.

Posted by Brian McClendon, Vice President, Geo

Building A Visual Planetary Time Machine

Posted by Randy Sargent, Google/Carnegie Mellon University; Matt Hancher and Eric Nguyen, Google; and Illah Nourbakhsh, Carnegie Mellon University

When a societal or scientific issue is highly contested, visual evidence can cut to the core of the debate in a way that words alone cannot — communicating complicated ideas that can be understood by experts and non-experts alike. After all, it took the invention of the optical telescope to overturn the idea that the heavens revolved around the earth.

Last month, Google announced a zoomable and explorable time-lapse view of our planet. This time-lapse Earth enables you explore the last 29 years of our planet’s history — from the global scale to the local scale, all across the planet. We hope this new visual dataset will ground debates, encourage discovery, and shift perspectives about some of today’s pressing global issues.

This project is a collaboration between Google’s Earth Engine team, Carnegie Mellon University’s CREATE Lab, and TIME Magazine — using nearly a petabyte of historical record from USGS’s and NASA’s Landsat satellites. And in this post, we’d like to give a little insight into the process required to build this time-lapse view of our planet.

Previews of the phenomena visible in these time-lapses.

First we'll describe Google’s Earth Engine system for deriving the time-series imagery. Second, we'll tell you more about CMU’s open-source “Time Machine” software for creating and streaming large, explorable time-series imagery.

Annual Composites: Distilling a Massive Dataset

Google Earth Engine brings together the world's scientific satellite imagery — over a petabyte of multispectral imagery recording over 40 years of history — and makes it available online with tools that scientists, independent researchers, and nations can use to mine this massive warehouse of data to detect changes, map trends and quantify differences on the Earth's surface using Google’s computational infrastructure. Today, the platform is used to monitor the Amazon and estimate forest carbon in Tanzania, among hundreds of other partners developing new uses for the technology.

Using Earth Engine, we first built annual global mosaics at a resolution of 30 meters per pixel for each year from 1984 through 2012. We started with a total of 2,068,467 scenes from the Landsat 4, 5, and 7 satellites, comprising 909 terabytes of data. The Earth’s atmosphere is a constantly-shifting sea of clouds, so in order to assemble a seamless cloud-free view of each year we analyzed all the images available at each location and used a simple cloud model to separate out the clouds from the ground. To help correct for atmospheric and seasonal effects, we used an additional 20TB of data from the MODIS MCD43A4 product to build a cloud-free low-resolution model of the Earth over time. We combined all this to produce a statistical estimate of the color of each pixel for every year for which data was available. Producing the final 29 global mosaics took a bit less than a day and consumed approximately 260,000 core-hours of CPU.

Some areas of the planet are almost perpetually cloudy, obscuring satellite views. In addition, before the more capable Landsat 7 began operating in 1999, coverage in some areas of the world was sparse, particularly in Asia, for various operational and technological reasons. We wrestled with how best to visualize areas with missing or cloud-obscured images from each year. In the end, after much experimentation, we chose to simply interpolate between valid image years. Other techniques, such as greying out invalid data, created distractingly large artifacts, visually drowning out the valid information. However, the downside with the approach we have taken is that it can be difficult to tell which data is original and which is interpolated. We are exploring the possibility of including a view that allows drilling down into the non-interpolated, original mosaics.

"Time Machine": An HTML5 Time-Series Exploration Tool

Once we had produced the final global images, we adapted the Carnegie Mellon CREATE Lab’s open-source “Time Machine” software, which enables authoring, streaming, and exploring very-high-resolution videos. Time Machine videos take advantage of the power of HTML5 and modern web browsers: they are streamed as multiresolution, overlapping video tiles and displayed in a web page by manipulating the HTML5 <video> tag, in much the same way that Google Maps first demonstrated using the HTML <img> tag.

Examples of zoomable timelapses with hundreds of millions or billions of pixels per frame include documenting plant growth, bee colony collapse, and very-large-scale simulations of the universe. Time-lapse Earth, however, sets a new record for giant videos: each frame of the video is a global Mercator-projected map with a resolution of 30 meters per pixel at the equator, for a total of 1.78 trillion pixels per frame. That’s about a million times larger than a standard HD video stream. In order to scale to such large videos, we needed to integrate Time Machine’s data production pipeline into Earth Engine and the rest of Google’s infrastructure. Encoding the final video tiles consumed approximately 1.4 million core-hours of CPU in Google’s data centers over the course of about a day. For CMU's researchers, this would have been impossible without Google's resources.

Combining all three phases of product generation:

• Total processing time: 3 days
• Total CPU usage: 1.8 million core-hours
• Peak CPU usage: 66,000 simultaneous cores

Destination locations of top 1500 share links, weighted by number of visits.

Time-lapse Earth is powerful because it helps us to access and construct the story of our planet. That story will become richer with each release, as we continue to improve fidelity and add data. The story-teller is everyone — scientists and citizens alike provide the real value by interacting, exploring, layering their knowledge upon the globe, and sharing their insights so that we can all better understand our world.

We are especially proud of the collaboration that made time-lapse Earth possible, and believe it to be an exemplar of how industry, academia, government, and the press can benefit from working together deeply over a period of years. By drawing on the strengths of each member of the collaborative community, Google strives to integrate the world's technical expertise and knowledge in order to tackle innovative and groundbreaking projects. In doing so, it is our goal to deliver an impactful service, one that can put a focus on the dramatic effect we are having on our planet.

Remembering WEIZAC: the beginning of computing in Israel

Israel is now one of the world’s tech powerhouses, second only to Silicon Valley as a hub for startups, but it wasn’t always this way. Today, in honour of the 84th birthday of Professor Aviezri Fraenkel, we’re delighted to share a short film sharing his story of working on the WEIZAC, Israel’s first computer.


Short film produced with support from Google as part of our ongoing computing heritage series

The impetus to build a computer in Israel came from Professor Chaim Pekeris, an MIT-trained geophysicist and mathematician, who made it a condition of accepting a job at the then-fledgling Weizmann Institute. An illustrious committee was set up to consider Pekeris’s request and initially opinion was divided. In particular, Albert Einstein was skeptical a computer in Israel would receive sufficient use and queried whether the skilled resources to build it were available. It took much convincing by another committee member, mathematician and computing luminary John Von Neumann, before the project got the go-ahead.

Construction of the WEIZAC (“Weizmann Automatic Calculator”) got underway in late 1953 under the leadership of Professor Pekeris and Jerry Estrin. A protege of Von Neumann, Estrin arrived in Israel armed with design drawings based on the computer at the Institute of Advanced Study in Princeton. After advertising for recruits, a small team of engineers and technicians was assembled, among them Aviezri Fraenkel.

It took the team a lot of ingenuity to source the necessary materials. Some were imported, but others were clever adaptations, such as the thin copper strips that came from a small local bicycle-part shop! Despite such hurdles, progress was steady, and the major components were in place by the time Estrin returned to the U.S. 15 months later.

The WEIZAC performed its first calculation in October 1955 and was soon much in demand by Israeli scientists. It remained operational until the end of 1963—50 years ago this year. Nowadays it resides in the Weizmann Institute’s Ziskind Building as a fitting memorial to where computing in Israel began.

I have fond memories of passing by the WEIZAC every day when I studied at the Weizmann Institute, where I also had the privilege to attend a class by Professor Fraenkel. With the release of this short film, I’m delighted to be learning more from him about such an important chapter in Israel’s tech history.

Posted by Yossi Matias, Senior Engineering Director, Head of Israel R&D Centre

What the ...?

Dear Google users—

You may be aware of press reports alleging that Internet companies have joined a secret U.S. government program called PRISM to give the National Security Agency direct access to our servers. As Google’s CEO and Chief Legal Officer, we wanted you to have the facts.

First, we have not joined any program that would give the U.S. government—or any other government—direct access to our servers. Indeed, the U.S. government does not have direct access or a “back door” to the information stored in our data centers. We had not heard of a program called PRISM until yesterday.

Second, we provide user data to governments only in accordance with the law. Our legal team reviews each and every request, and frequently pushes back when requests are overly broad or don’t follow the correct process. Press reports that suggest that Google is providing open-ended access to our users’ data are false, period. Until this week’s reports, we had never heard of the broad type of order that Verizon received—an order that appears to have required them to hand over millions of users’ call records. We were very surprised to learn that such broad orders exist. Any suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.

Finally, this episode confirms what we have long believed—there needs to be a more transparent approach. Google has worked hard, within the confines of the current laws, to be open about the data requests we receive. We post this information on our Transparency Report whenever possible. We were the first company to do this. And, of course, we understand that the U.S. and other governments need to take action to protect their citizens’ safety—including sometimes by using surveillance. But the level of secrecy around the current legal procedures undermines the freedoms we all cherish.

Posted by Larry Page, CEO and David Drummond, Chief Legal Officer