Blog Post

Search engines, big data and network effects

Search engines are intermediaries in a two-way market between users and advertisers. Their huge stocks of data about users and their preferences can help search engines offer better services to all parties. But does this make market entry difficult for new players? And can we see network effects emerging in the search engine market?

By: and Date: November 22, 2016 Topic: Digital economy and innovation

Search engines respond to queries by providing relevant and valuable information about the topics that their users are looking for. They also act as intermediaries which match consumers with providers of services or sellers of products. To monetise their work, search engines collect and process data from users and sell advertising slots to companies. By analysing the users’ data, they can improve the quality of the search engine algorithm and provide more relevant organic search results, but they can also design personalised advertising strategies for companies’ products and services. Companies are thus more successful in placing their products, and consumers receive recommendations tailored to their interests.

Advertising slots are allocated on a competitive basis to the companies that are willing to pay the highest amount to get some prominence in the platform when the user is searching for relevant terms. Slots are auctioned and the winners have to pay a fee to the search engine whenever a potential consumer clicks on the displayed ad (pay-per-click strategy). Search engines base their revenues almost exclusively on selling advertising slots (see for example some relevant statistics in Liem and Petropoulos, 2016).

So, in this two-sided market environment, advertisers assign a higher valuation to participating in the search engine platform when they expect to have a higher probability of matching with as many consumers as possible. More consumers can increase the value for companies to be advertised in the search engine in two main ways. First, companies obviously see that they can be connected with more potential buyers as the pool of relevant consumers increases. Second, search engines with more users can analyse the additional data and improve their matching algorithms. They can then provide more personalised services to advertisers which can in turn have higher chances of success in attracting consumers. Given the competitive allocation of advertising slots, more consumers, therefore, implies higher revenues for the search engine as advertisers are willing to bid more for prominence.

Consumers, on the other side of the market, value information that leads them directly and quickly to what they are looking for, to minimise their search costs (for example, consumption of time). Hence, they mostly prefer to only visit the search engine of the highest quality (according to their standards and criteria) even if they have the option to use different search engines simultaneously.

In betwen these two sides, search engines compete in quality and efficiency of search (for example, accuracy of results, page load speed, real time relevance) in order to attract more consumers who will in turn bring more advertising revenues which can be used for further improving their quality to attract even more consumers, and so on. So, it is of interest for the search engines to build up a large stock of users’ information that can be analysed in order to provide good quality services which attract more consumers and advertisers.

How important is the scale of data collection for the improvement of services?

In general, the answer depends on the complexity of the algorithm in place, the number and the noise in observations. But in the context of data related to human behaviour and preferences, the amount of data and the quality of predictive power seem to be positively related.

When data are drawn from human actions, noise rates are often high. Social scientists have long argued that one way to circumvent the poor predictive validity of attitudes and traits is to aggregate data across occasions, situations, and forms of actions. This provides an early suggestion that more (and more varied) data might indeed be useful when modelling human behaviour data. The implication for predictive analytics based on data drawn from human behaviours is that by gathering more data from individuals (aggregated by the modelling), one could indeed hope for better predictions.

Fortuny, Martens and Provost (2013) demonstrate that when predictive models are built from sparse, fine-grained data—such as data on low-level human behaviour—economies of scale are important and we continue to see marginal increases in predictive performance even to very large scale. However, the curve between predictive power and data size do seem to show some diminishing returns to scale, albeit without any ceiling that limits the increasing pattern.

Their empirical results are based on data drawn from nine different predictive modelling applications, from book reviews to banking transactions. The study provides a clear illustration that larger data indeed can be more valuable assets for predictive analytics similar to those used in search engines.

Big data and quality driven competition

Since the quality of search engines increases with data gathering and analysis, search platforms with a larger stock of information than their competitors can have a competitive advantage. They can perform better and attract more users and consequently more advertisers. This suggests that the amount of data processed is linked with market power in the search engine market. Incumbents with a large stock of information and experience in data analytics can protect their market position against new entrants and firms that are far behind in data accumulation. Moving to the frontier of the search market and establishing a large base of users requires new entrants to offer very attractive prices for the advertising slots and at the same time to bear the costs of developing a good quality dynamic algorithm that can attract consumers. This is an investment that can only pay off in the long run. Bing, for example, in its effort to increase its installed base of users incurred losses for many years before starting to be profitable.

Note that in many markets product differentiation among competitors is multi-dimensional. Consumers’ preferences are affected differently by each of the multiple dimensions. If I need to subscribe to a mobile operator my choice depends not only on the quality of the network but also on whether I want to combine mobile with TV and home internet, whether I wish to call often in another country or how much mobile internet I intent to use. The plurality of options and preferences lead to multi-dimensional market competition where there is room for entry and growth of firms that specialise in specific dimensions.

The search engine market is by construction centralised and product differentiation has only on dimension, the quality of service. (Some could argue that privacy is another dimension, but this is a secondary choice parameter, especially since it is difficult for the users to rank competitors with respect to the privacy protection they provide). That means that the search engine of higher quality than its rivals can increase its market power relatively fast or even dominate the market, especially under the implementation of efficient methods of machine learning and data analytics.

Moreover, in many cases, companies that own a popular search engine can use the collected data for their operation in other data-driven markets to improve the quality of their services and consequently their market position.  For example, since 30% of mobile searches are about location, queries in the search engine about finding a particular location can improve the quality of navigation services through the additional data collected by the search engine. So, the search engine can help the “mother” company to enter to new markets and improve its market position through data accumulation and analysis.

Are there network effects in the search engine market?

Direct network effects in an online platform exist when an increase in the number of users or in usage of the platform directly increases the users’ value from accessing the platform. Facebook is a typical example of a platform where the value of using the network highly depends on the number of friends that also use the network. In search engine platforms, on the other hand, such direct effects are absent as each user derives individual benefits from its use without any direct effects to other individual users.

Indirect network effects exist when the increase in the number of users or usage of the network spawn increases in the value of a complementary product or network, which can in turn increase the value of the original. The relevant question is if there are indirect network effects in the search engine market which increase the value of participation for users.

Hal Varian, the chief economist at Google, argued at  a recent Bruegel event on big data and market competition that it is the accumulated experience in the market that helps search engines to learn how to improve the quality of their services (learning by doing effect) just as it happens in normal businesses. He claims there are no indirect network effects involved on the users’ side:  “The higher the number of customers a business has, the higher the revenue of the business, revenue which can be reinvested in the maintenance and improvement of the business so as to attract more users.”

While experience clearly matters, the impact of learning by doing effect is greater than in normal businesses due to the importance of data in improving the search engine algorithm. The addition of new users makes the search engine more competent to provide high quality services through the processing of the additional data collected by the newcomers. That generates extra value for the advertisers and further increases search engine’s revenue as well as its incentives to improve quality and attract even more consumers.

However, I would argue that indirect network effects are also present. We should not ignore that search engines are two-sided markets in which both sides are inter-dependent. What happens on one side has an impact on the other side and affects the value of accessing using the search engine for both sides of the market. The addition of new users increases the value of using the search engine for the existing ones not only because of the increase in quality (due to the learning by doing effect) but also because more users attract more advertisers. That leads to more matching opportunities for the existing users through the sponsored links of the search engine and therefore their value from accessing increases.

Network effects may also extend to organic search results given that the ranking of firms depends to some extent to firms’ choices such as their search engine optimisation strategies. While the algorithm of the search engine is secret, published guidelines provide an overview of how firms can increase their chances for high rankings in the organic search results. The arrival of new users increases the incentives for firms to get a better ranking position in order to attract more users by investing in search engine optimisation and the quality of their website. So, they provide better web services to their visitors (existing and new) and help them to reach what they are looking for in a more timely and efficient fashion.

When we assess market competition in the search engine market we need to take into account the importance of big data in defining market power. Especially when users do not pay any price to access search platforms and the latter compete in other key dimensions such as the (data-driven) quality of service.

Republishing and referencing

Bruegel considers itself a public good and takes no institutional standpoint. Anyone is free to republish and/or quote this post without prior consent. Please provide a full reference, clearly stating Bruegel and the relevant author as the source, and include a prominent hyperlink to the original post.

Read about event More on this topic

Past Event

Past Event

An alternative mobile operating environment?

Walking the wire: we discuss risks and benefits involved for the EU should it embark on developing a new smartphone operating system.

Speakers: Hosuk Lee-Makiyama, J. Scott Marcus, Renato Nazzini, Peter Stuckmann and Andreas Zimmer Topic: Digital economy and innovation Date: April 29, 2020
Read article More on this topic More by this author



Backstage at BAM19: How can Europe's economy thrive in the digital age?

Backstage at the Bruegel Annual Meetings, Giuseppe Porcaro talks with session chair Reinhilde Veugelers on Europe's economy in the digital age.

By: The Sound of Economics Topic: Digital economy and innovation Date: September 4, 2019
Read article Download PDF More on this topic


Digitalisation and European welfare states

EU policymakers must find answers to pressing questions: if technology has a negative impact on labour income, how will the welfare state be funded? How can workers’ welfare rights be adequately secured? A team of Bruegel scholars, with the support of the Mastercard Center for Inclusive Growth, has taken on these questions.

By: Georgios Petropoulos, J. Scott Marcus, Nicolas Moës and Enrico Bergamini Topic: Digital economy and innovation Date: July 9, 2019
Read article More on this topic More by this author



Deep Focus: Developing Europe's digital single market

Bruegel senior fellow J. Scott Marcus joins Sean Gibson for this episode of Deep Focus on the 'The Sound of Economics', elaborating on a Bruegel study for the European Parliament into the progress made with the Commission's Digital Single Market Strategy since 2015.

By: The Sound of Economics Topic: Digital economy and innovation Date: February 12, 2019
Read article Download PDF More on this topic

External Publication

Contribution to Growth: The European Digital Single Market

Numerous legislative measures have been initiated or enacted in support of the overall achievement of a Digital Single Market (DSM). This in-depth analysis provides a brief stock-taking of what has been achieved in economic terms, of what remains to be done, and of candidate initiatives for the next legislative term.

By: J. Scott Marcus, Georgios Petropoulos and Timothy Yeung Topic: Digital economy and innovation Date: February 12, 2019
Read article More on this topic More by this author



Backstage: Brexit consequences for EU’s ICT policy

Bruegel senior fellow Scott Marcus welcomes former European Regulators Group chairman Kip Meek to explore the consequences of Brexit for ICT policy-making in Europe.

By: The Sound of Economics Topic: Macroeconomic policy Date: September 25, 2018
Read article

Parliamentary Testimony

European Parliament

The role of independent expertise in legislative process

Testimony before the European Parliament Committee on the Internal Market and Consumer Protection (IMCO).

By: Zsolt Darvas, J. Scott Marcus and Bruegel Topic: European Parliament, Macroeconomic policy, Testimonies Date: July 18, 2018
Read article

Parliamentary Testimony

European Parliament

The potential impact of Brexit on ICT policy

Testimony before the European Parliament's Committee on Industry, Research and Energy (ITRE).

By: J. Scott Marcus and Bruegel Topic: Digital economy and innovation, European Parliament, Testimonies Date: June 27, 2018
Read article Download PDF More on this topic

Working Paper

How big is China’s digital economy?

The rise of influential Chinese digital giants, including Baidu, Alibaba, Tencent and Xiaomi has shown the world that China is a global leader in digital innovation and it is not surprising that China has started to influence the global digital market. But is China exploiting its full potential in this area? To answer this question, the authors assess how big China’s digital economy is relative to the rest of its economy, and how China performs compared to the rest of the world.

By: Alicia García-Herrero and Jianwei Xu Topic: Global economy and trade Date: May 17, 2018
Read article More on this topic More by this author

Blog Post

How e-commerce reshapes markets and firms’ strategies

The development of e-commerce has affected both demand and supply fundamentals of markets, changing the way competition works. In the effort to develop a frictionless and welfare maximizing digital single market across the EU, it is necessary to carefully review the disruptive forces on e-commerce on markets and firms’ strategies.

By: Georgios Petropoulos Topic: Digital economy and innovation Date: May 7, 2018
Read article More on this topic

Blog Post

Can roaming be saved after Brexit?

The referendum where UK voters chose to exit the European Union has many unanticipated consequences. One that is gaining visibility in the UK just now is the impact of Brexit on mobile roaming arrangements. How might the UK maintain roaming arrangements with the EU in the event of a hard Brexit?

By: J. Scott Marcus and Robert G. Clarke Topic: Digital economy and innovation Date: September 21, 2017
Read article More on this topic

External Publication

Economic Implications of Further Harmonisation of Electronic Communications Regulation in the EU

One of the ways in which the European Commission has sought over the years to strengthen the European single market is by means of increased harmonisation of the regulation of electronic communications. To the extent that the European Union functions as a confederation of somewhat autonomous member states, however, there are both practical and political limits to the degree of harmonisation that is realistically desirable or achievable.

By: J. Scott Marcus and Christian Wernick Topic: Digital economy and innovation Date: August 11, 2017
Load more posts