In this case, a resident would be correct on most days if they naively assumed the day’s weather would be sunny, and a forecast of a sunny day thus doesn’t provide much new information as compared to the naive baseline. Just how useful is a given forecast? Suppose we’re in the summer, and the weather (optimistically) is sunny 75% of the time and overcast 25% of the time. In this case our service will actually be useful - residents no longer can just assume that the weather will be overcast every day and be correct 100% of the time. Now, suppose instead we’re in a different season with multiple possibilities for the day’s weather. In other words, the forecast won’t provide any new information for city residents, since there’s no uncertainty in the outcome. If it’s wintertime, where the weather is overcast ~100% of the time, sending the text is redundant a resident will always be correct if they have basic knowledge of the city’s weather patterns and assume that every day will be overcast. Let’s say the city of Seattle has decided to introduce a new service, whereby every morning each resident gets a text message with the day’s weather forecast. We’ve provided a couple examples of “information” in the abstract sense, but how can we define information in a formal, precise way? We’ll start with an example. With the tools provided by this theory (now known as Information Theory), we can answer important questions about the limits of modern computers like: How can we store information (say, a file on our hard drive) in the most efficient way? Or, how do we successfully communicate information (say, a text message) over a channel (e.g., our cell network) that may not be 100% reliable? In 1948 Claude Shannon, during a stint at Bell Labs (also the birthplace of the transistor and Unix operating systems, among other major achievements in computer science) provided the first comprehensive framework to do so in his landmark paper A Mathematical Theory of Communication. Those new to the series the first post is here.Īs the first digital computers were being invented in the 1940s and researchers like Alan Turing were formalizing their capabilities, the scientific and engineering communities needed a way to quantify previous intuitive notions of information and communication. Recommend a refresher on logarithm rules for those who may have forgotten them. The notes assume background knowledge equivalent to an introductory probabilityĬourse and at some points will require knowledge of univariate calculus. This post is part of a series of notes on topics in information theory.

Entropy theory