ad-free
Channel / Source:
TEDx Talks
Published: 2014-11-05
Source: https://www.youtube.com/watch?v=8DqQCZMawNg
this is the display interface of the Apollo eleven computers this is the machine that took mine to the moon now at the Apollo eleven computers now where I used by the astronauts and they had to be specially trained so that they could then took months using two digits on the first page it was a very what was it to be performed ten the second one was
the noun what kind of data would be affected by by the verb now %HESITATION if you can trust states are interface in the Apollo eleven computers Swiss %HESITATION with a modern smartphone like the iPhone five you're going to find some starting different cities and the iPhone five is to one thousand two hundred and seventy times faster than the Apollo eleven computers it has two million times
more processing memory more storage and memory and it is also about three hundred times sliders on the Apollo eleven computers now I hope you can appreciate how much computational power your carrying with you when you're walking about today but I will leave it up to you to decide whether your smartphone connection to take it to the moon %HESITATION and you are able to use one of
these the fine see simply because of the and acceleration in technologies that we've seen in the last few years and especially more so applying east and it suggests that nearly every two years the number of transistors on integrated circuits basically doubles so you can imagine why we have so much power from machines we eat the said that the in the developed world we are spending a
bounty in eleven nine hours processing information on a daily basis so whatever we do leaves a digital trail when you walk around you with Sarah and when when when you walk around with a more by phone then and we can track where you want with the GPS state that's when you go to your doctor you even did stellar record behind when you interact through Facebook and
Twitter you generate content yourself in some of you may be generating content even as we as we speak when you purchase items on Amazon then we know that you like science fiction movies and when you go to the supermarket when I go to my super market my supermarket knows that I like cheese and chocolate but obviously not at the same time so and the amount of
information that we are creating increasing at an amazing speed it is said that between two thousand and ten and two thousand and twelve we have actually created more information than in all the years it is up to our %HESITATION past history and it's quite interesting to see vase with a very simple visualization so this is all recorded history then two thousand and ten twelve are just
about here so you can imagine how much state down we have generated just between those two years in fact I think they said it is estimated that every day we're producing about two point five billion gigabytes that's a lot of information that's mediums trillions of bytes that they are generating on a daily basis in June twenty fourteen Cisco updated there on your %HESITATION IP traffic projections
for the next five years and there are some interesting projections there by twenty eighteen the number of users that would be using the internet would be about now for me and that's not in twenty thirteen it was %HESITATION two point five and they're going to be about twenty one billion network devices by twenty eighteen in twenty thirteen there were about twelve and I would not even
mentioned the staggering rate of information that would be to change them all by the phones by and twenty eighteen is going to reach one hundred and nineteen exabytes %HESITATION in twenty thirteen we only produced eighteen so you can imagine the am how much information aware producing and that brings us to big data because it's clear that would generate a lot of data and data are characterized
by now what their clothes of the disease so we have volume there's this year volume moving of data that are being produced philosophy the speed with which we are producing date they is absolutely astonishing we do so in real time and for multiple sources and variety data is not just structured it condemns fractured it can be social media bait that Clinton such you generate yourselves it
can be audio video so all of these increased complexity and and we are extremely risking drowning in data we have so much data that we don't really know how to process now the %HESITATION assumption knees if you have daytime then you have knowledge and did you have big data therefore you have big knowledge but this is not to data is not information a seven Ford says
and information is not knowledge as Einstein says data is dance to recorded in there just are recorded transactions structure transection shows that take place information or designs from the processing of the day trying to kids meaning and say and then you have no ID's and nodded sees what you are aiming for your trying to understand why things are happening so in order to understand data we
perform data exploration and they'd exploration needs the analysis of data so we are talking about techniques to analyze and interpret the data that we have at our disposal can we use visualization of the date are particularly an arts data sets or conflicts state the sets because we would like to convey information to non specialists in a way that will enable them to gain insight from the
data and modeling and here we're talking about predictive modeling and validating and testing hypotheses but or committee what we are asked to respond to the date they is only valuable if we can extract you swing signs from the data if we are able to use it to inform decision making that will be a bill for instance they informed decision making enterprises and businesses and other organizations
and it's this predictive aspect of modeling that I'm truly fascinated and buying so if we start with big and complex state other than what we are what we usually do is we analyzed it clean we interpret the data and then we can use these analyses during former action and and strategy we can take a step further and saying we can hypothesize as to why is it
that we see these patterns seen marching in the daytime and we can stop here but this is where I say mine to the gun because if you stop here just by formulating this theory you're missing a unique opportunity collectively generate new knowledge discovering units because what we would like to do ideally is take the step further build more dose out of the and the %HESITATION assumptions
formulate assumptions as to why we see these patterns in V. envy in the daytime and saying with these must not go more those are more complex types of one does we can try inviting a date our theory are some students and we can run of these independent models and we can generate artificial data sets which weakens and check against the real data that we see check
whether the same possums merits and give the same pattern C. Mertz thing you can say with some degree of confidence that your hypotheses is validated when you have this kind of validation of your theory you can imagine that you have a lot more confidence in your decision making you can better inform your strategy and you can support your your action no matter what organization you are
so all I'm fascinated by these %HESITATION more those that we can create based on the data that we may have at our disposal so I'm gonna tell you a couple of stories about how we can create such small dose and what kind of work we have been doing I'm interested in a complex more dose not just any type of mathematical model I mean for something complex
models that emanate from the tradition of artificial intelligence and it specifically a more those that use agents and multi agent systems and the easiest way to explain what the nature and he's from a computer science point of you're an artificial point of view is these independent software prophecies if you like that can act independently %HESITATION they have their own function on a theme to have their
own attributes and then you can put them together in %HESITATION Inna in them sin not going to simulate that scenario and they can they can you stock interacting with each other and you can start observing what's the outcome of their interaction is going to be at the system or the global level now the interesting thing Levi's sort of law does I'm talking about is that you
cannot predict them it is not the case that weakening called within these agents therefore behavior in such a way so that we know what the outcome is going to be this is simply because these complex systems what you have is the final outcome only Margie's as a result of the interaction of these independent entities and you cannot predict how each one entity is going to affect
the other so these are the kind of entities that I'm interested in modeling and we can a cold very complex behavior and functionality weakening cold things like trunks or knowledge or beliefs that we usually ascribed to and to humans for that matter so have you seen this picture before this is %HESITATION any mines which has been created by Michael not John Michael majorities in northeast and
this is from his high altitude and collection and and he went towards it denied to %HESITATION visit the mountain range is fair and after that he was inspired and he created the series of images representing the evolution of the living %HESITATION stock indices are all around the world and vehemence that you see in front of you is that of the evolution of the stock price of
the Leeman brothers from ninety two up to two thousand and eight and that's why you can see this varies a sharp drop because it has basically crashed %HESITATION nice you will not dis there are peaks and troughs in the Siemens NBC's typical well financed sell data that we can gain from Amy affine on some market or this kind of found of environment and these pics in
trial toward the east out of interactions of thousands literally thousands of trading agents a real trading agents as well as artificial ones that are interacting in them within a financial market now it's true that we can be oath what they're called agrees make trade there is how gorgeous make agents of these are pieces of software that are able to identify patterns they can look at the
state and with clever machine learning algorithms that can identify patterns in the data and then make a decision as to whether to buy or to sell based on the movements that they can identify but what these programs are not able to do is look to try and understand why we see these patterns emerging and this is a fascinating question for me this is what I'm interested
in what the states that makes these apartments marriage what kind of behaviors the agents the trading agents need to have what characteristics what strategies so that we can see these %HESITATION these patterns emerging so I'm gonna tell you learn about the project that we we did it and we were able to %HESITATION of Dana very large data sets with about a hundred and forty seven million
transactions from a bone to a forty five thousand traders on an unknown spacey's from the finance yeah %HESITATION from the the foreign exchange of this is the market where you exchange currencies and we are talking about a lot of data set as you can seem so when you have this kind of on daytime in the finance yellow %HESITATION in finance you tend to DO away somehow
with time series and you tend to sample so and you can choose when to something in the beginning of the end you can do it tucked into a bus and a trend the mean two votes but when we talk about a hundred and forty seven million transactions which were over to it and two years we are talking about a hundred and eighty thousand transactions per day
and if you attend to sample then you don't know when sample because you might we supply Thurn buy something at the wrong time so we have a problem how is it that we can process for the state them so I am in order to address this problem instead of looking at this issue from %HESITATION %HESITATION time serious point of view what we did was we attempted
to identify a veins and the direction of change veins in the data that we were saying and we formulated hypotheses after nine test why we were seeing some of these patterns and what kind of strategies the agents had behind and then we built a very complex market an artificial market and we put in place there these agents and I'm talking about thousands of agents that had
their own strategies and they started interacting with each other and what you see here is just one of the results that we read them with had them you see several and lying sing these particular graph the the line with the new squares ease the pattern identified in the re elevate them and when you look at the real date and this has been identified by several studies
you seek to comes when you look at the trading activity over a twenty four hour period the first time is because you have the London market being open at the same time as the Asian market in the morning and the second time is because the London market is open at the same time as the times the American markets and we run various simulations with different %HESITATION
%HESITATION sets of agents with different strategies and we were able to come up with a strategy that was a bit too brings the data very close to what we were observing in in real life and this is the one that has the %HESITATION the red down at triangles %HESITATION ain't so we were to a certain extent able to validate their hypotheses as to why we see
these days down patterns in the data when you're able to do that's the next step venues to trying to predictive modeling so this is the next step %HESITATION coming in for us and this is very exciting now there are other domains that are equally fascinating and they contain a lot of data not who medical data necessary need but other types of data such as for instance
social networks and social networks have %HESITATION increased so much in the last few years that the tax looking extraordinary and why do we engage in social networks well it's our needs to interact and communicate with each other and social networks are guided by this principle of homo feeling so what happens is we like to link with like minded people people that are like cats so this
is called the principle of morphine %HESITATION but leaving social networks we make some assumptions as to what homophobia is and how agents %HESITATION how humans find each other and we wanted to transfer data modeling and do it to modeling problems and we were particularly interested in identifying how agents can start in a social network and then how it is it that they can identify each other
how is it that they can measure their similarity and then by identifying other similar agents how is it that they can create clusters so we started with a network that had only one agent and then we started %HESITATION putting in more agents and what you're going to see them and now he's the videos that is going to play the the simulation that we run and hopefully
this is going to come up on line anytime soon can we play the video please and you can see V. agents in the beginning they do not and know each other so that's why you see these different colors and their connections but as time progressed sees and information disseminates within the network they munched identifying each other and they start creating these plasters nausea as you can