Strategies for expanding IXPs and other Internet/cloud infrastructure

6 November 2012 - A Workshop on Critical Internet Resources in Baku, Azerbaijan

Also available in:
Full Session Transcript






14:30 CET









     >> KAREN ROSE:  Our part at the Internet Society those previous workshops help shape our interconnection and traffic exchange programmes which we conduct in a number of regions around the world.  So this panel today represents an evolution of those past IGF discussions.  And our goal is to help put IXPs in the larger context of promoting broader growth in the Internet economy, including Internet infrastructure growth, investment and deployment and maximizing lessons learned around the globe.  As such for this panel we are going to be aiming to move the discussion around these issues from beyond the technical so we can broadly cover IXPs and a range of issues related to promoting and developing the Internet economy.  A number of areas that we aim to discuss on the panel today includes providing an overview of progress of deploying IXPs and discussing the challenges and gaps that remain, real world experiences in deploying IXPs such as in Africa and those lessons learned and Asia, technical, economic and policy strategies for promoting the deployment of Internet infrastructure and establishing a sustainable market and thinking ahead and sharing thoughts about solutions for overcoming barriers. 

     We are lucky today to have world leading experts in these areas.  And by way of introduction we have Michael Kende who is a partner at Analysys Mason, a global consultancy company.  Bill Woodcock is a research director at Packet Clearing House, a non?profit organisation dedicated to understanding and supporting Internet traffic exchange technology policy and economics.  Fiona Asonga, chief executive officer of test box, the Telecommunications Service Providers of Kenya which is the managing body of KIXP Nairobi and IXP in Mombasa.  We also have Paul Wilson Director General of APNIC.  And Martin Levy, director of IPv6 strategy at Hurricane Electric, a global network provider that participates in a significant number of IXPs around the globe. 

     Let me just go over the format very quickly.  In order to provide maximum opportunity for interaction each of the panelists are going to provide a short opening statement, around eight minutes and then we will jump directly in to discussion. 

     So to kick things off I would like to go to Michael Kende.  Michael, you have done a lot of work looking at the economic benefits of Internet Exchange Points and factors that influence their growth.  Can you tell us a little bit about the positive economic impacts of IXPs and what challenges you see remaining in categorizing this effect in the broader area? 

     >> MICHAEL KENDE:  Sure.  Thanks for the introduction and thanks for the opportunity to speak here on this topic.  Can everyone hear me?  So today I want to talk about IXPs and for they are really a critical part of the Internet ecosystem and the Internet infrastructure in countries.  I am going to base this talk on the number of papers that I have done recently on how Internet companies are interconnecting with one another and a recent paper for the Internet Society that measures the specific benefits of IXPs in Kenya and Nigeria. 

     So IXPs originally were formed to eliminate tromboning.  Now tromboning as an example of this in the early days of the Internet, commercial Internet in England in the mid 1990s there were five ISPs.  Most of the content and users were in the U.S.  All the ISPs had to have connections to the U.S.  At the time the market wasn't liberalized in UK or Europe.  So it was very expensive for ISPs to connect with one another to exchange traffic.  So the result was these five ISPs all used their international connections to the U.S. to exchange all their traffic including domestic traffic.  So I could send an e?mail to someone across the street and that would trombone through the U.S. and come back to the person across the street. 
     Obviously not very efficient and ultimately quite costly.  So the ISPs including the incumbent BT got together and set up the London Internet exchange in 1995 I think.  And in there they all met ?? to cross?border traffic and then IXP, the concept moved from Europe to Asia and now the biggest growth in IXPs is in Africa and Latin America and the benefits, I will talk in general, Fiona can talk about the benefits we found in Kenya and then they are also in the paper that we did for the Internet Society. 

     But the first and biggest benefit and immediate benefit is it eliminates tromboning.  It reduces latency because it is obviously much quicker to keep the traffic within the country.  And it makes the Internet more resilient because you are not relying on a lot of hops around the world to send your traffic.  So that's great.  First immediate impact with a very small investment in an IXP where everyone can exchange traffic.  But in recent years the trend has been that these are really growing in in importance because content is now increasingly portable.  Up to 98% of Internet traffic now consists of videos, Web pages and other things that can be moved from where they originate and stored in caches or servers all around the world.  And the result of that means that once an IXP is established content providers will move their caches or their servers closer to it to be able to deliver traffic more quickly to the end users that are using ?? that are connected to that IXP.  So this has great impacts.  Further lowers the cost of getting traffic because you are not getting Youtube pages from another country.  You can get them domestically or any other video. 
     It also not just lowers the cost but we found in this study that it can increase the revenues.  For the simple reason that if you are ?? we all know if you are downloading a video and it takes too long to download, you will stop and do something else.  Well, when the video suddenly starts moving local you can access it quicker and more people will download videos and other content and in a lot of countries you pay per megabit for the download.  So actually revenues can go up as a result of this IXP with more traffic.  Another great benefit. 

     We have found that government is starting to use access to the IXP to deliver services.  Whether it is educational or tax or other services can be delivered through the IXP.  And a good IXP can also become regional.  Can start to gather traffic from neighboring countries.  And so it can grow and then you build up critical mass because people ?? more and more people put their content near the growing IXP and then it grows even further.  So you get a nice cycle going. 
     So these are all the positive benefits of an IXP.  Further down as people start to invest in content you will start to get jobs and more revenues that come from that.  There are, of course, a few challenges.  In particular when you try and set up an IXP today it can be challenging because a lot of times the incumbents now view joining an IXP as threatening their revenues.  They view IP transit, the sale of transit sometimes as a revenue gathering and they don't want to appear an interconnect at the IXP.  So it is very challenging to get an incumbent to join.  Back in the '90s when links formed it is likely that no one at BT was thinking about the revenue implications and they just joined because it is a nice solution to an engineering problem. 
     So that's the first challenge, getting the incumbent to join.  Second relates to infrastructure.  We know there is a lot more submarine cables in to Africa and going to Latin America but you still need domestic capacity to get to the IXP.  Sometimes not fully liberalized and that can raise the cost of using the IXP.  And actually we are working on a follow?up to the first Internet Society study focusing specifically on these cross?border and domestic fiber issues and that will hopefully come out later this year and be available through the Internet Society. 
     But in closing I would submit that there is really no more cost efficient way to improve the infrastructure of the Internet in a country than to set up an IXP.  Very low cost.  And it can bring significant benefits for the end users, the content providers, the IXP and others that take advantage of it by lowering the cost of delivering traffic and increasing revenue and helping to build an Internet economy in the country.  Thank you. 

     >> KAREN ROSE:  Thanks a lot, Michael.  Bill, can you tell us a little bit about the Internet Exchange Points around the world and where they are growing fastest?  And you sometimes referred to IXPs as bandwidth producers.  Could you tell us a little more about that? 

     >> BILL WOODCOCK:  Sure.  So Internet Exchange Points are where Internet bandwidth comes from, same sense that farms are where vegetables come from.  Internet service providers move the bandwidth from Exchange Points to the consumers but they are not capable of producing Internet bandwidth on their own, right?  Because the bandwidth ?? the Internet bandwidth is the ability to get to the rest of the Internet, not just other customers of that ISP.  So in that sense all Internet bandwidth comes from an Exchange Point somewhere.  What's interesting whether it is coming from an Exchange Point that's near you so you get good performance and the packets are nice and fresh, a little latency or whether it is coming from an Exchange Point in a different country, right, imported at high cost. 
     So in that sense as Michael's pointed out this is something that's necessary as an economic development measure for every country to become autonomous to be able to produce its own Internet bandwidth.  The question of where the Internet, where Internet Exchange Points are growing most rapidly is a statistical question and there are several ways of looking at that.  Several ways of measuring that.  The two primary ones would be in either absolute terms or in relative growth terms, right?  So either we can look at where the amount of bandwidth being produced is more larger than last year or we can look at where the amount of bandwidth being produced is absolutely larger.  Right?  So we can either look at a percentage of growth year over year or we can look at a number, an integer quantity of bandwidth. 
     So Michael referred to growth in Latin America and Africa.  It is true that in Latin America and Africa you see high percent in growth rates.  This is because by and large the Exchange Points are still relatively small.  So if they quadruple in size year over year this is a big percentage.  But in terms of absolute growth, it is still quite small.  In terms of absolute growth where you see the big numbers are Germany and the Netherlands.  Germany is a fair size country with a population of its own that uses a lot of Internet bandwidth.  The Netherlands has a much smaller population than Germany does.  So even though most of the time Germany is slightly ahead in the race for amount of bandwidth produced the Netherlands is the world's largest exporter of bandwidth because they consume less than what Germany does domestically. 
     So in terms of who is doing the most Internet bandwidth production and exporting it to other countries, it is the Netherlands.  In terms of who has the highest year over year growth rate it is whoever that was most recently started.  So that's almost always a tiny Exchange Point in a tiny country. 
     Where we see the most new Exchange Points being formed right now is in the Caribbean.  So something like six new Exchange Points underway in the last 18 months in the Caribbean which is a pretty high rate.  Where we see the least happening is in the Middle East.  So if we look across the last ten years in the Middle East we had one Exchange Point in Cairo and one in Bahrain and four years ago one started in Beirut and one is just getting underway in Dubai after five years of work to get it started.  And Kuptar (phonetic) is considering getting one going.  This is the third time Kuptar has entered in to this conversation. 
     So I think that what we see there is to some degree the level of frustration and patience with importation being ?? the Caribbean I think got tired of exporting capital to wealthier countries.  And that frustration built up and they decided we need to be able to do this for ourselves.  We are tired of sending cash to the U.S. and the UK.  In the Middle East I think there is much greater feeling that the amount of money involved is negligible and they have got plenty of money.  So they may as well just ship it all to Europe and not think about this particular issue. 
     That seems to be the reason, the main reason why this has been the slowest growing region.  One of the things, to change topic a little bit, that I have been asked to talk about here is the results of a survey that we did now almost two years ago that fed in to an OECD report on carrier interconnection.  That is the terms under which ISPs interconnect with each other at Exchange Points and elsewhere.  And the sort of interested outcome of that was that 99.5% of the interconnections between Networks are made on an informal basis.  That is without a written contract and signatures.  Half a percent have a written contract with signatures and half of those have asymmetric terms.  One quarter of 1% of Internet connections have terms where you couldn't reverse the names of the two parties and have the same agreement.  That is one is paying the other.  So if you think about what a regulator can do versus what the industry has done for itself, there is no regulator in the world that can regulate globally.  There is no regulator that can regulate 99 and a half or 99 and three?quarters percent of all the world's Internet Networks.  First of all, nobody has the scope that would be necessary to get that degree of agreement.  Secondly, there is no regulator in the world that finds its regulated parties 99 and a half percent in such agreement that they don't even have to negotiate terms. 
     So I think those two findings that industry has arrived at a degree of uniformity in its understood terms and conditions and that there is no way that Government could arrive at as uniform a condition has really led people to a new understanding of the degree to which Internet self?governance is successful.  So anyway those are just the things that I was going to try to get across. 

     >> KAREN ROSE:  Thanks a lot, Bill.  Fiona, what are some of the challenges that had to be overcome in establishing and growing KIXP and the IXP in Mombasa?  And what's the evidence of the broader effects that IXPs have on the Internet economy?  And how do you explain that value proposition of an IXP to stakeholders? 

     >> FIONA ASONGA:  I start by mentioning the Internet Exchange Point has been in existence from May of 2000, and to date we have 38 members peering on the Exchange Point which is pretty much all the licensed operators within Kenya.  And we are established as a non?profit entity, and therefore we run our operations charging fee for ?? which is a per charge.  Some of the challenges that we have experienced during this ten?year period have included ranging from the need for us to set up redundancy for the location because for a very long time we had only one port and everything was centralized at that port.  But since 2008 we have managed to put in place other locations as Karen mentioned Mombasa but we have to put our redundance location in Nairobi.  We were two points in Nairobi and a third point in Mombasa.  And we have to ensure that we are adding value to Exchange Points to the different operators who are appearing at the location. 
     And that I can see has been a big challenge because we tried to keep the costs low.  Yet at the same time ensuring that there are a lot of other value add services that we are running such as the domain name, root servers, the security applications and being able to give reports to peer members on the status of their Networks.  We also keep inventing measurement tools that are able to provide details of what kind of traffic is going through the Exchange Point and being able to maintain all of these services so that then the member is able to see the member who in this case is the ISP is able to see the value that has been.  Again trying to grow the traffic.  And what we have done is try to work as closely as possible, first of all, with content managers or content providers, likes of Google and Akami to see how to bring their content to the Exchange Point to host the servers locally.  So then the content goes through the Exchange Point and we made an effort of negotiating with transit providers so that the transit providers are able to give subsidized rates to the content providers for them to bring their traffic, the content to the Exchange Point. 

     And the other challenge is we face a lot of the local websites are not costed in Kenya.  They have to come out to Europe where the servers are hosted and back to the country which defeats the purpose of the Exchange Point.  There is a large programme we are running with the Government to be able to bring this content to be hosted within the country.  The other has been the need for us to be able to just have a steady infrastructure within the country, there are a lot of issues around infrastructure deployment.  And these have ranged from being able to get appropriate transit between the different locations at a subsidized cost.  And then the other challenge a lot of the infrastructure is within the cities.  When you go to rural Kenya there are areas where you can't get any communication.  So that beats the purpose of having a ?? traffic has to go out of the country and then come back in but their regulator and Government are putting in place a universal access to fill those gaps.  We will be able to have more traffic coming in to the Exchange Point. 
     The other which is not a challenge is the benefits that the Exchange Point has provided and these are in terms of reducing the cost of transit between end users in terms of customers and e?mail is faster and communication is faster and then there is the benefit of reduced costs amongst the ISPs for their international capacity because it has been freed by the traffic that comes in to the points.  The speeds are high.  We have seen the traffic speeds of accessing sites improve from about 1.2 seconds to as low as 60 to 80 milliseconds to access the CNN website in the U.S. or something like that. 
     And we have also seen that there has been more appreciation within Government for the services we offer, and we have seen the Kenyan Revenue Authority, for example, bring their system for collecting revenue from both the customs agents and from the general public to the Exchange Points.  And that has made our trafficking increase exponentially because then every Kenyan who wants to file any revenue returns does it through their respective IXP and that comes in from the Exchange Point to the carry system.  And the other challenge we have this is an issue to do with regulatory, but it is something that hopefully will be resolved by the first quarter of next year and that has to do with regulations and licensing.  And we are trying to have an arrangement because we are still a licensed Exchange Point, one of the few licensed Exchange Points in the world.  That still enables them to have access when they need to and to allow the Exchange Point to run independent of all other entities and still be the value that it provides. 

     One of the things that maybe I would mention that, I still have a minute, that one of the things that we observed is with the deployment of eGovernment services at the Exchange Point online and this traffic coming in to the Exchange Point.  We have seen traffic increase substantially.  And when exam results or exam registration is happening you can see the traffic go up because all the candidates who are sitting for a particular level of examination have to be registered online.  And when the results are out they are available online and that has helped grow the traffic and with the establishment of the growing of eGovernment we have seen a lot of services now being offered online such as their registration of persons and getting of I.D.s and passports.  And this is what is driving now traffic in to the Exchange Point.  And that adds value to the entire ecosystem as Michael Kende had mentioned in his earlier report. 

     The value can be quantified in terms of figures because when you find the revenue authorities setting that they collect up to about five billion Kenya schillings a day out of the online system and that's value they are getting.  It is to a level where they are so dependent on an Exchange Point.  When we have a power fluctuation like we do in Africa and our systems are all down, we find that the reports ?? the report that goes public is the Exchange Point was down and therefore the revenue collections for yesterday were hampered and that's the level of impact that KIXP is having on the Kenyan economy. 

     >> KAREN ROSE:  Thanks.  Paul, provide us a perspective from the Asia Pacific.  Obviously a very diverse region.  What are some of the experiences and challenges and lessons learned in IXP deployment in the area and the role in catalyzing growth in the Asia Pacific? 

     >> PAUL WILSON:  Thanks very much, Karen.  Excuse me, APNIC is the IP address registry for the region.  We are not involved with infrastructure deployment or management.  And so we are very neutral in that ?? in the sense of who is running infrastructure or how.  But the thing about our constituents, our stakeholders they are ISPs, people who need and use IP addresses and they are the direct beneficiaries of IXPs and that's the common thing across the region.  When ISPs get together with an Exchange Point they can lower their costs and that lowers prices to the user. 

     In Bill's agricultural terms you are distributing bandwidth out to the users and that's happening locally and not internationally importing the bandwidth.  I think the common things about IXPs that we probably heard about often enough at IGF is that the IXP model is ?? one of them which works best as an association of mutual benefit by agreeing parties who are agreeing through the IXP to collaborate.  Sort of an example of this is this idea that the Internet is characterized by competition at a commercial level but collaboration behind the scenes.  It is not that any of that happens in secret.  I mean in the case of IXPs they have generally got very transparent terms and conditions and policies and so forth.  But the point is that it is an example of collaboration between entities which are commercially competing in terms of the services they offer. 
     So in the Asia Pacific we have seen plenty of IXPs start.  It is a shame that Guarab Upadhaya was not here because it started with a small grant of maybe $20,000 to buy a bit of equipment.  The technology to start an exchange is pretty low tech.  It is not difficult technology by any sense but what's ?? what needs to happen is that a group of people need to get together to understand the value proposition and they need to commit some funding to joining in to a model that's for the mutual benefit.  And so at the start at least a single non?profit organisation in the major cities or in each of the major cities of a country is the way that the things start and after that that model builds and becomes successful and you do often get competing exchanges and for profit exchanges coming along. 
     The thing that we have seen in our region is because I think there is a huge difference in economic circumstances between a place like say Nepal and then compared with South Korea or Japan or the developed countries is that there is not one Asia Pacific environment at all.  There is many.  One thing that we have seen emerge in the last few years is the group called the Asia Pacific Internet Exchange, APIX Group which is starting to act as one of these catalysts for information exchange or exchange and learning experience among Exchange Points.  We are a little late compared to the other regions.  There is other regional models to follow. 
     The other thing about those groups is that they form a network amongst the participants that extends far beyond simply the commercial or technical arrangements.  They tend to catalyze a local community, local network operator community often in the form of a NOG, network operators group.  It is a very powerful, professional exchange of learning and experience among a local community. 
     So I think the intangibles about Exchange Points do have to do with community building and understanding of the value proposition.  Understanding about what an Exchange Point is and what it isn't rather than the technical aspects that we might have heard about here before. 
     So you look beyond that to the best ecosystem or the best environment for an IXP to become established.  And I think the first thing is to make it easy and to have fewer, if any, barriers to the possibility for people to get together in a non?profit mutual manner at a low cost to set up something through an appropriate vehicle.  So you definitely don't want to see licensing in an Exchange Point as though it is a carrier to a public telecommunications area.  You don't want hurdles in terms of bureaucratic impediments and so forth.  You want to see a bottom?up rather than a topdown model. 
     There are cases where IXPs have been established by Government and in the experience they have been so great.  Those organisations tend to run a lot more slowly and to change themselves and steer themselves a lot more slowly.  So when this environment is changing quickly and so, for instance, when the international bandwidth prices are actually dropping very rapidly around the world, the initial idea of an IXP is a mechanism to avoid international bandwidth.  The balance changes.  And if you have had an IXP that was charged at an access or a rate that made international bandwidth attractive in comparison, then it is not doing the job that it is supposed to do. 
     I think the bottom?up model and the idea that the beneficiaries of an IXP or ISP and the other peers of the exchange and the decision making and the structure should be in the hand of those people and is very much a mandatory prerequisite for a successful Exchange Point establishment.  I think we heard about a catalytic Exchange Point and we have heard about the opportunity for content distribution Networks and we call them cloud computing services these days but CDNs that can bring their services in to IXPs and provide the ecosystem at a national level of high speed responsive high quality Internet services where those services may not have been involved ?? may not have been available.  I think we all know that is increasing the benefits of the Internet as a whole to a community.  And if we are talking about local content then I think the Government information, the eGovernment services is almost the best example of using an IXP to deliver local services and local content very effectively because those services are really by definition local.  So I will leave it there, Karen.  Thanks. 

     >> KAREN ROSE:  Thanks.  And Martin, can you tell us why Internet Exchange Points are important for Internet providers, and dealing on the investment side do they provide an incentive for Network operators to spend money locally and bring connectivity to a country? 

     >> MARTIN LEVY:  Well, the quick answer is yes.  They absolutely provide an incentive.  But let's delve down to the unintended bits and work out why.  There is a whole set of factors that have been talked about Internet Exchange Points, tromboning being one of them.  Local content being another.  But I am going to take you through a journey from outside the country in.  When we talk about the Internet Exchange Point in a country, we normally look at benefits inside the country.  There is actually more to it.  Inside the country I am going to throw a couple more benefits in and show why they suddenly become valuable in a global setting. 
     Take network ingenuity.  They do it within companies, within NRNs if they are educated and they do it inside Government entities if they are building Networks for Governments.  An Internet Exchange Point becomes a wonderful focal point for local network engineers to understand how to do networking far better than just buying a transit pipe to a Telco above them whether in country or out.  Very little engineering required to make that work.  Internet Exchange Point brings some expertise to the Network engineers to interact and cooperation between them because they are peering, because they are doing networking operations more complex than just being a customer of somebody else.  That expertise and this is one of those things that as you meet engineers from around the world you realize that the engineers that have had that experience also understand how to go and buy their network capacity from either foreign players or in country with a greater level of expertise, greater level of understanding of the question of the ability to measure the work.  So a simple biproduct over and above the movement of bits, the quality of the bits improving, is, in fact, actually a big win inside a country, inside a region, inside a city depending on whichever way we are looking at this. 
     The interesting point about this is that those network Internet Exchange Points, IXPs also have to exist within inside a better quality of network data center than would normally be thought of for IT purposes.  Now that also brings up something like should a network Internet, Internet Exchange Point be built in a neutral facility.  And we can ignore that point for a moment because of where I am going to go.  What happens when an Internet Exchange Point is built local players will hash out these issues.  They hash out ?? they know players from around the world, players like myself.  And ask these questions and the type of answers they get back are the appropriate ones for building an Internet Exchange Point.  But the key point of where I want to get to in talking about this argument about the economic benefits and the like and where Networks like ours come in to play, if all these steps occur, if an Internet Exchange Point is successful, if an Internet Exchange Point promotes the use or the building of an Internet data center, if an Internet Exchange Point builds for network engineers that understand the ecosystem of the global Internet better, then you have set up an ability not for Networks in country just to spend money out of the country but the potential of bringing global Networks in country to significantly improve the quality of Internet. 

     A Network provider like us, for example, sit with a massive list of places around the world to expand.  We are in about 50 places at the moment.  And it is a natural progression of a company to look at where would you expand next.  Sometimes in your backyard but sometimes you go and add a new country to your map.  And those countries can actually when you finally look and see what's happened over the years, those countries, that growth of Internet, the building of massive bandwidth has nearly always been related to where Internet Exchange Points have been placed and have become successful.  Marketplace grows and the ability to find independent data centers grows.  The internal push from the country has made it palatable for external countries ?? external companies from external countries to look at building. 

     So as we as a network look at new marketplaces as one does you look at the Internet exchange growth, the quality of the Networks around the Internet exchange and that gives you a good idea of where to go. 

     Now the only final point to add to this is why build in to a country versus simply wait for that country to build it out and take the money outside.  And this is an obvious statement.  There is a natural tendency for local Networks to start out with a local hot spot.  We know those in Asia and North America, but the reality is if they are successful, if undersea cables come in to a country and build gateway locations, then the Internet exchange will add to the ability for that country to become a gateway, to become a central point, to become something that is fundamentally needed and important as we look at the global Internet.  So we look very carefully at new exchanges both as an interest to see whether we want to join but more importantly as they become hot spots that become the next gateway to somewhere else in the world.  The Internet is global and let's keep it that way. 

     >> KAREN ROSE:  Great.  Thank you.  Now I would like to open it up to the floor for questions.  And while people are thinking of their questions and as well if there is anything that the panelists want to respond to or comment on based on the opening comments that would be great as well.  Go ahead in the back.  Can we get a microphone? 

     >> AUDIENCE:  Thank you. 

     >> KAREN ROSE:  That would be great. 

     >> AUDIENCE:  Thanks to the panel for their wonderful comments.  Local Internet exchange can greatly improve your Internet access in local countries and we have seen some modern examples.  Territory where five operators came together and sort of set up a local exchange but Fiona highlighted and Martin also mentioned the world is bigger and it is also about getting other parties to come to your IXP content providers.  And as Fiona highlighted having a sea cable landing improves that effect.  It helps a lot.  So my question is what have you done to baseline?  What if you are entirely landlocked?  Because being able to access the sea, getting, making a cable ending politically ends here because you don't have to negotiate with a lot of parties and this is relatively cheap.  So what in terms of landlocked countries can be done or should be done to basically give them the same advantages?  How to extend those supporting landings over land in to those countries that don't have a baseline.  And in terms of IGF in a multistakeholder environment which form or which parties should take the lead to there because it is not that easy as a commercial party to just say I am going to take a fiber from country A to country B.

     >> KAREN ROSE:  Thanks for the question.  That's great about landlocked countries.  Michael and Bill. 

     >> MICHAEL KENDE:  That is an excellent question.  And so, for instance, in Africa there is 16 landlocked countries and three in South America and ten in Asia.  Clearly a critical issue and I would say it is not an issue for a landlocked country.  Not every country has one ?? not every coastal country has one landing station and some of them only have two.  So getting across borders is important for increasing competition, resilience, et cetera.  So I think it is really something that the Governments have to focus on first to remove any barriers to being able to build across the border.  There are still some of those landlocked countries have a monopoly on the international gateway.  So it is very hard for any competitors to try and build out to the coastal country obviously in that situation.  And so I think that they have to ?? first it has to start with a policy that we need to get access to there.  They are going to have to negotiate with the other Government for a right?of?way through and try and get as many redundant and competitive cables through as possible, but the first thing has to be recognition.  That they have to get access competitively to hopefully more than one coastal country to be able to access those and I think that some countries are starting to think of that seriously and there is some cross?country Pan?African cables coming up and I think those are going to start addressing those issues. 

     >> BILL WOODCOCK:  The fiber doesn't know whether it is going through water or dirt.  It is not a technical issue.  There are two differences between landlocked countries and countries that have access to ocean.  One is the ownership structure of the cable.  And the other is the number of paralyzed neighbors.  A landlocked country may have as few, well, in the case of their countries completely surrounded by South Africa, may have as few as one neighbor.  More often two or three neighbors.  The limits of number of pairwise relationships they can have country to country and they may have to then chain those together as Michael said in order to reach other landing stations or other Exchange Points or other Internet ?? other international bandwidth.  Maintaining good relationships with your neighbors is really critical if you are landlocked because they are the ones who are your gateway to the rest of the world.  And maintaining a regulatory environment that encourages joint ownership of fiber.  Undersea cables are almost always consortium owned.  There are multiple owners who each have shares and access to the fiber is generally available in the marketplace.  Whereas terrestrial fiber is almost owned by a single party or one party at each end who do not sell wholesale access or do not make access available on the same terms that they themselves get it.  Right?  So that sets things up for a monopoly condition.  Those are really the two big differences. 

     >> KAREN ROSE:  Fiona? 

     >> FIONA ASONGA:  Maybe I can share with you the experience that we have had within the East African condition.  We have the East African communications organisation that brings together both the regulators who are presenting their respective countries.  And of these countries only two have a coastline.  The other three are landlocked.  And from 2008 the discussion at that level has been that the different Governments will work together to ensure cross?border connectivity.  So we have what we call the East African backhaul system which is basically a network running across all the five countries.  And what we are trying to do right now is to get that network connected to all the five Exchange Points so that then the traffic can flow within the Exchange Points or within operators, across operators who are appearing at the different Exchange Points.  And the other thing that the Governments agreed to was that any member of the ?? anyone who does business within the East African community and wishes to appear at any of the Exchange Points doesn't need to have a license to access the Exchange Point in the next country.  So you can actually as a Kenyan go to the Uganda Exchange Point and peer the Exchange Point without a license.  So that kind of collaboration is working well.  And has seen and is what has contributed to a lot of traffic growth within the different Exchange Points because for centers that have seen traffic coming in to Kenya from each of the countries and go to respective countries and contributed to the 50% announced at KIXP that are not Kenyan.

     >> KAREN ROSE:  Thanks.  We had another question over here.  Can we get a microphone?  Anyone else in the queue?  Over here. 

     >> AUDIENCE:  Thank you very much.  I barely hear myself.  My name is Maria.  I want to thank all of you in the panel for the good statements.  (Cutting out).  A few of you I have seen before but I wanted to mainly ask what kind of support ??

     >> KAREN ROSE:  Can we get another microphone? 

     >> BILL WOODCOCK:  I think the question was what kind of support can Governments give to Exchange Points. 

     >> AUDIENCE:  You mentioned the regulatory framework and a few other things.  You can't hear me up there? 

     >> KAREN ROSE:  The microphone is cutting in and out. 

     >> AUDIENCE:  Because what I understand from Fiona if I don't misunderstood you is that the Government is also like a customer for the agencies, they are a customer for the Exchange Point.  I just want to hear what else can we do. 

     >> PAUL WILSON:  As background for the rest of the audience it is useful to know that Sweden is at one end of the spectrum of Government participation in Exchange Points.  Sweden has a tighter coupling of Government participation with Exchange Points than pretty much any other country in the world and that's worked quite well there, but it is a model that has not been lept upon by other Governments by and large. 

     >> KAREN ROSE:  We had a question here in the front.  I am thinking ahead.  Go ahead.  Who is in the queue? 

     >> FIONA ASONGA:  One of the things that Government has done to promote the eGovernment services and ensuring that they are hosted at an Exchange Point and the other has been to work with hosting the Exchange Point in setting up some of the value add services that we have sitting in a Exchange Point such as security monitoring and that kind of stuff but the Kenya ?? I know the Government did put in funding for the Exchange Point to get it started.  We have not been able to benefit that level.  However they work very closely with us on different programmes we run, including capacity building where the Government institution, Kenya ICT board from time to time supports the different programmes that we are running. 

     >> BILL WOODCOCK:  Just to give another quick example in Nepal over the first I think four or five years of the existence of the Exchange Point by far the highest traffic spikes were at the end of every academic semester when the Government released everyone's test scores, all the student's test scores and all their parents were madly clicking on the reload buttons waiting to see their kids' test scores.  There are different kinds of Government participation, different kinds of Government content and information that can be made public across Exchange Points and that is often one of the early large drivers of traffic. 

     >> FIONA ASONGA:  One other thing the Government of Kenya makes sure if you want to provide any Internet related services to a Government institution you must be appearing at the Exchange Point.  That is a policy directive that enables us to get more members coming to the Exchange Point and therefore bringing more traffic to the Exchange Point. 

     >> MICHAEL KENDE:  I have worked in several countries that are trying to set up Exchange Points, developed or almost developed countries.  And the biggest frustration is typically that they can't get the incumbent to join and then the ?? so they can throw money at it.  Both of these were countries that had money to throw.  So they can throw money at it.  They can do what they want.  But they allow us typically to fall short of requiring peering or getting involved in IP interconnection in a regulatory way.  And I agree with that.  I mean it is a strong tradition of not regulating.  And at the end of the day most of them were convinced they don't want to be the first country to force peering on the incumbent but it becomes a question of anchor tenancy and getting more content there and providing more money as seed money for the exchange, but typically without the incumbent it is hard to recreate something like links or the other large exchanges without the incumbent's active participation. 

     >> KAREN ROSE:  Sorry.  A hand over here.  Oh, great. 

     >> AUDIENCE:  Hi.  Jessie Sol, grad student at MIT.  I have two questions.  I will ask the first question and go to the next if there is time.  A number of panelists have touched on Associational membership model describing it has been very successful in Europe and it is being deployed, if you will, in other regions.  I am wondering if the panelists can unpack some of the differences or variations in this model in the context of other regions.  What parts of the model have been successful directly applied and which parts have been modified for different contexts? 

     >> KAREN ROSE:  Why don't you give your second question as well and we will take them both at the same time.  That's fine. 

     >> AUDIENCE:  So they are fairly different.  But IXP, they have been referred to as Internet Exchange Points but a few suggestions that it would be valuable to directly interconnect IXPs.  And I would like the panelists to elaborate what is known as the noncompete norm and what their perspectives are on early instances that are currently in the wild for this type of interconnection.  I see Bill smiling.  Please pick whichever question you'd like and go. 

     >> BILL WOODCOCK:  This is a misconception that comes up over and over and has been for 20 years.  If an Exchange Point is good and connecting two of them together must be better, think back to the agriculture analogy.  Do you need your vegetables delivered from one farm to another farm or do you want them delivered to people who want to eat them?  There is to packet in the Internet that crosses two Internet Exchange Points.  There is no model for that and no need for that.  So interconnecting Exchange Points with each other does absolutely no good because there is no traffic that needs to take that path.  What you often see is people pursuing that misconception and then finding themselves with a bill to pay or having bought a circuit that was unneeded and trying to find some way to cause traffic to cross that circuit in order to be able to send someone a bill.  What happens is people say well, why don't we let people, for instance, in Dar es Salaam appear at an exchange in Mombasa by having them interconnected with each other and we bill everyone in Mombasa and Dar es Salaam for the exchange and we hope that somebody in Dar es Salaam wants to appear with somebody in Mombasa and it creates a new expense.  And there is no downward pressure and no competition and it divides the cost which now has no reason to go down between a whole bunch of people who have equal access to it, but none of them have any inherent need for it. 

     You have created an artificial tragedy of the commons that did not need to exist and has a high price tag associated with it.  So essentially this problem has been encountered many times before.  It is not significantly in the scale of the world.  But if you go looking for it you will always find a few examples that have not yet collapsed.  There has never been an instance of this that has lasted more than a few months or a year or two.  That's an answer to question No. 2. 

     Question No. 1 was about the model, the Association model.  The way you phrase the question made it sound as though this was a minority position that was moving outward from Europe but in fact, it is the majority globally.  There are some commercial exchanges that also exist.  Commercial exchanges are more prevalent in the United States, Australia and a few in Europe.  The Association model is by and large the dominate model everywhere and always has been.  Equinix is a huge commercial exchange operator in the United States and has many of the largest exchanges in the United States but that is globally sort of an exception. 

     >> MARTIN LEVY:  I will take the first question.  The reality of building in to exchanges, the actual movement of bits actually seems to have little difference whether it is a member based exchanged or a commercial exchange with one key point.  If the cost is controlled and is palatable to the local market, then the exchange will win.  And when I say palatable that means that if I look at the cost of building to an exchange in a Stockholm, Sweden or the cost of building in to an exchange in London or Hong Kong or Tokyo, and I am picking some big cities for a reason, the cost model is quite different although as I go through those different cities, I, in fact, go through a member or nonmember based environment.  The cost of living in those cities will, in fact, actually have a direct relationship to the cost of whether it be member based or commercial based because it just costs more to do business in Japan. 

     It costs more to do business in Stockholm than it does in London and all are more expensive than doing business in New York, at least in the telecom world.  This is one of the places where it is quite cost effective, et cetera, et cetera.  So as you move around you will see a price, the local marketplace most be palatable to that independent of whether it is a member based or a commercial based environment. 

     The final point is that keep in mind it also has to be contractually acceptable, not with requirements that, for example, in a membership based organisation there may be a requirement for some local corporate entity that may or may not exist for a player coming from far away.  I am trying to make sure I don't talk about telecom regulation because that's a given within operating anyway.  So I think in the big scheme of things it doesn't matter but I think that we have got to stop talking about the European model of membership because it is a lot more global than that these days. 

     >> BILL WOODCOCK:  Just to elaborate a tiny bit more.  There is one advantage and disadvantage to operating an Exchange Point commercially.  Advantage is you can lose money.  A commercial operator can choose to subsidize its operations from some other source for the benefit of the customers, right?  Yeah.  I mean whatever the reasons, right, they can artificially depress the cost of participation and get people in the door.  The disadvantage is that they are not protected against acquisition.  A non?profit in pretty much every country is protected by acquisition which a commercial organisation by means that there is no participant in the exchange that can buy the exchange to compel behavior on the part of competitors. 

     >> KAREN ROSE:  Maybe we can go to Fiona to answer the question and then we can get to the one on the floor. 

     >> FIONA ASONGA:  Bill has answered part of the first question, and the reason the acquisition model works because the different operators do not want any one operator to take full control of the Exchange Point is pretty much the reason in Kenya we are to go with the Association model and that was part of the trust or amongst the operators a ?? played a big role in guiding us in that direction and so far it has worked.  In the sense that a number of entities actually went and got licenses because we are a licensed Exchange Point.  They were issued with licenses to run Exchange Points and some have even set up Exchange Points but nobody appears there.  So they call the switch an Exchange Point but it has no traffic because nobody wants to appear there.  So the Association model is what is carrying the day and it is what we are encouraging our neighboring countries and other entities we need to adopt. 

     The second question on the interconnecting of the Exchange Points, now within the East African region we have had the same argument.  However our Governments feel that the Exchange Points should still be interconnected and that is why we have the interconnection of the different Exchange Points within the different countries but because the operators feel that they should ?? should be on a contractual basis, based on what value you see at a particular Exchange Point we still asked for room to allow operators to go in to the respective Exchange Points directly.  So over the last two years that this has been operating what is standing out is that operators are preferring the model where they go to our respective Exchange Points because of the different value adds they are looking for as opposed to the interconnected Exchange Points.  Though these
are ?? Governments are generally paying for to ensure that there is a path that leads to the different Exchange Points.  Thank you. 

     >> KAREN ROSE:  Thanks.  We will take two more questions and then we will go to a wrap?up.  We are running to our time slot.  There is a question from Mohamed. 

     >> AUDIENCE:  Hello.  Hello?  Okay.  Can you hear me?  Okay.  My name is Mohamed.  I am based in Senegal.  And I just want to ?? the first one is in Africa.  We have less than 20 countries that have already set up IXPs out of 52 or 54 depending on talking about the numbers of countries in Africa.  And the Internet traffic is growing in countries where they don't have Internet Exchange Points.  And in many countries they say maybe we don't need it because we have seen the traffic going and the customers are happy.  So it is very difficult to prove to countries who have elected to set up their Internet Exchange Point, specifically in a country where you have an incumbent telecom who is very powerful and who connect to almost 60 or 70% of the customers.  And they are putting the country in a situation where everyone is just paying for transit and nobody I mean is peering with the telecom operator.  This situation exists in many African countries.  Just like nobody was able to convince that there is a value they are missing because, in fact, the main argument they put on the table is we don't have any Exchange Point.  The Internet is growing very fast.  We are doubling or we are 300 builds every year. 

     Why do you want us to put in these things?  I am an IP expert and I am trying to push the model in many countries but the position we get the chance we have in this IGF meeting we have these different actors; those who decide and those who regulate and those who advise and those who operate.  And we need really to show to people what are the real benefits and what are the accelerators that we can show and prove that existence if there is no Internet exchange point in their country, what they are losing.  It is not obvious and I hope that you guys can show up in countries where they don't have Internet exchange point what they are losing from the Government and private sector perspective and from the end user perspective. 

     >> KAREN ROSE:  Response from Martin and then we will go to our last question. 

     >> MARTIN LEVY:  It is truly wonderful to know the Internet can grow without an Internet Exchange Point.  But that's a general function of the Internet itself as it exists.  But it is the wrong data point.  The fact is that if you go out to other marketplaces, if you were to go to East Africa, if you were to go to various places in Latin America, the Internet ?? the growth is so much greater when you see an active Internet exchange environment.  So it is nice to say hey, the Internet is growing but the reality is no, it actually should be growing even more than that.  Even more than you think is there.  So your incumbent, if you have a single incumbent inside of a country that has this high percentage of control of the marketplace it is in its best interest quite frankly to stop the building of an IXP.  That's promotional 101.  It is going to happen and I think that's the result you are seeing. 

     >> KAREN ROSE:  Thanks a lot for that.  And one final question and then we will have a final wrap?up. 

     >> AUDIENCE:  Can you hear me? 

     >> KAREN ROSE:  Yes.

     >> AUDIENCE:  One of the key themes of IGF this year is development.  And a quick question for any of the panelists with respect to the fact that you see the importance of the human layer of development, the physical infrastructure layer of development and governance.  But with respect to the human capacity development aspects and what an IXP can do to help with that perspective or that element can you elaborate on that?  I know there is a specific amount of training that can take place when you have an IXP up and running and what is that that has been done for the local technical community? 

     >> KAREN ROSE:  Great.  We want two respondents on that?  Go ahead Fiona and Paul or did you have your hand up? 

     >> FIONA ASONGA:  When it comes to capacity building in the case of the Kenyan Exchange Point what we have seen over the years is an improvement in terms of skills.  When we are starting off we took one whole year to get started because we had a challenge in terms of the skills we wanted to use to run the Exchange Point as well as legal issues and knowledge of what was involved.  And once we got started we have seen an appreciation by legal and regulatory policy entities and what the Exchange Point does and the value it brings to the entire setup of the Internet within the country.  And that has come through in terms of the kind of discussion, policy discussion we have and the kind of regulations that are now coming through and when it comes to issues that affect the Exchange Point.  When we talk of, for example, cybersecurity because an Exchange Point to run an industry search, so then we are able to use the information running on the members, peering members Networks to provide security in lots and that kind of information is appreciated by the Government at the search level and the Government entities.  And we have seen the eGovernment.  So we can give them the lots on the content coming in and how vulnerable it is. 

     The other thing is there has been capacity building at the technical level.  We have seen a lot of skills being developed because we have ?? Kenyan Exchange Point to learn from other Exchange Points, particularly the European ones and there are two Exchange Points in particular that we are learning from technical skills who give us capacity building to the KIXP technical team who then pass it on to technical staff of our peering members.  So twice a year have training on routing and different routing techniques and any new routing techniques and that has translated in to continuous development of skills because it is done on an annual basis. 

     >> KAREN ROSE:  Okay.  And from Paul.

     >> PAUL WILSON:  Just a brief question that I think, it is really wrong to see an IXP as a panacea and monolithic thing that is going to create miracles but as many things in the developing environment you have a lot of leverage to gain by packaging things together.  Human resource development is important.  We know that across all areas of economic development in term of Internet services, the difference between a reliable fast secure Internet service one which is opposite on all three dimension is often human resource and capacity building and anything that can improve and help with human capacity building is a good thing and an IXP is one.  The coming PAC NOG meeting that is going to have as an exercise or parallel activity the installation of the first IXP and that's a nice example.  If they could throw a root server in there at the same time it would create a lot of excitement. 

     The IXP will benefit the people who participate in it.  So if you have a number of small disenfranchised IXPs and not competing with the incumbent and not participating with the exchange, IXP can bring benefits to them.  It could bring benefits by attracting friendly eGovernment services and providing through those IXPs and not to mention other content services.  If your incumbent is bulletproof and it makes no difference to them you have a different problem.  The IXP is not a panacea for that unfortunately either.  Thanks. 

     >> KAREN ROSE:  Thanks a lot for that, Paul.  And I think we had a really broad discussion and I think that Jeff encapsulated it really well.  In terms of looking at challenges and barriers and opportunities regarding physical infrastructure, you know, governance infrastructure, both in terms of Government policy as well as level governance of people getting together and being able to develop the models for how the Exchange Point is going to operate and provide value and we just ended on the human infrastructure bit as well and how that is both a need but also an opportunity when people come together to develop Exchange Points for exchanging and sharing their knowledge in the best traditions of the Internet which I think is what we have done here today on this panel. 
     So with that I would like to thank all of our panelists and everyone here and I would like to thank Jeff Brueggeman who was a co?coordinator of this session and who did a lot of excellent work to bring it together.  Thank you everyone. 
     (Session concluded)