L01 - Introduction

Contents

• Introduction to the course
• What is Web Intelligence?
• Assignments and Project

Recommended reading

None.

Lecture slides

Available for download here.

Recording

Note

This is an old introduction. Be aware that some things have changed!

Video

2019-08-30 11:13

📕 Errata

Submissions, oral exams and communication

The recording describes an older setup: Slack as the communication tool, submitting a link to your code on GitHub via MyMoodle, and oral examinations at the office or over Slack. Today the course uses Zulip (see Communication), you submit through a merge request on GitLab, sign up for the oral exam on Moodle, and do the oral exam on Zoom. See Deadlines and Submissions.

Projects

The recording lists the projects as recommendation system, clustering, web scraping and search engine, and machine learning, and mentions that you can define your own project. The current pre-defined projects are P1 Recommendation System, P2 Clustering, P3 Text Classification (Python and Scikit-learn required) and P4 Web scraping, and the Project page no longer offers a project of your own.

The Netflix Prize in 2007

The recording says that the best team had improved Netflix’s system by 7% in 2007. The team that won the first Progress Prize in 2007 had reached an improvement of about 8.4%.

• Introduction
• What is web intelligence?
• Example: Netflix
• Example: Google
• Web intelligence as a field
• Lectures
• Literature
• Submissions
• Course pages
• Assignments
• Project
• Grading
• Communication
• What to do next?

Summary

What is web intelligence?

Huge amounts of data are generated by the users of the web every day: ratings behind movie and song recommendations, Wikipedia’s articles on just about everything, web page and product recommendations, and much more. Web intelligence asks how we can learn from that data and draw conclusions from it, in order to create new products or services.

Two examples: Netflix and Google

Netflix recommends movies and TV shows based on what you have watched before. In 2006, Netflix offered a prize of one million dollars to the first team that could improve the accuracy of its recommendation system by 10%. The target turned out to be hard to reach: the prize wasn’t awarded until September 2009, when the team BellKor’s Pragmatic Chaos won it. That a company was willing to pay that much shows how valuable better recommendations are.

The 2007 figure

The recording and the slides say that the best team had improved the system by 7% in 2007. The team that won the first Progress Prize in 2007 had in fact reached an improvement of about 8.4%. The point stands either way: the last percentage points up to 10% took two more years.

Google changed how search results are ranked. Earlier search engines looked only at the contents of a web page. Google’s founders, Larry Page and Sergey Brin, ranked results on contents and on how popular a page is, measured by the links pointing to it. The results became far more relevant, Google quickly became the leading search engine, and most search engines today use the same or a similar approach. The algorithm behind it, PageRank, is covered in L04 - Search Engines.

Web intelligence as a field

Both examples are exactly what web intelligence is about: using data that is available on the web, or that users generate in our own system, to create new products or services. The field draws on many others, among them artificial intelligence, machine learning, visualization, information technology, statistics and programming.

How the course is organized

The course consists of seven pre-recorded lectures that you watch at your own pace:

1. Introduction (this lecture)
2. Recommendation systems
3. Clustering
4. Search engines
5. What is machine learning?
6. Data and learning
7. The Naïve Bayes algorithm

Each lecture page has recommended reading, the lecture slides and the recording. Watching the lectures should in most cases be enough to learn what the course requires, and the literature is there for a deeper understanding. Neither book is mandatory (see Readings):

• Programming Collective Intelligence by Toby Segaran is quite old and some examples are outdated, but it gives a good overview of the field and explains the algorithms very well. It is not free.
• Mining of Massive Datasets by Leskovec, Rajaraman and Ullman is a free e-book that explains the concepts in an accessible way with many examples.

A few online resources are used as well.

Assignments

There are four assignments:

• A1 – a recommendation system for a small movie ratings dataset.
• A2 – clustering of a dataset of blogs.
• A3 – a search engine for Wikipedia pages.
• A4 – the Naïve Bayes machine learning algorithm for classification tasks.

A1–A3 are implemented as web applications, with a RESTful web service as the back end and a browser client as the front end. You choose any programming language and any REST framework you like. Each assignment has requirements for different grades, and you decide which grade to aim for.

Toby Segaran’s book contains plenty of code that is useful for some of the assignments. You may not copy and paste that code and submit it: you write your own code, and you must be able to explain how it works at the oral examination.

The project

The pre-defined projects build on the assignments, so a project on a topic where you already put a lot of work into the assignment won’t be that large. Choose the project you find most interesting. The current projects are listed on the Project page.

Grading

Each assignment and the project is graded A–E, which is converted to a score (A = 5, B = 4, C = 3, D = 2, E = 1). The final score is:

Final score = (Score_A1 + Score_A2 + Score_A3 + Score_A4 + 3.5 * Score_Project) / 7.5

The final score is rounded to the nearest integer and converted back to a grade. With A1: C, A2: A, A3: B, A4: A and Project: B, the calculation is (3 + 5 + 4 + 5 + 3.5 * 4) / 7.5 = 4.13 ≈ 4, which is a B. The same example is worked through on the Grading page.

Submissions, examination and communication

Questions and discussions about the course belong in the course’s open channel, where course information is also posted. You can also send private messages to the instructors there.

Submissions, oral exams and communication have changed

The recording describes an older setup: Slack as the communication tool, submitting a link to your code on GitHub via MyMoodle, and oral examinations at the office or over Slack (which needed the installed Slack application for screen sharing). Today the course uses Zulip instead of Slack (see Communication), you submit through a merge request on GitLab, sign up for the oral exam on Moodle, and do the oral exam on Zoom. See Deadlines and Submissions and Submit an assignment as a merge request.

Where to start

Watch L02, read the recommended reading and start working on the first assignment. Keep track of the deadlines on this website so that you don’t miss a submission; it’s a self-study course, so the pace is up to you. Post questions and join the discussions on Zulip, and continue with the other lectures, assignments and the project from there.