A2 - Webbskrapan

Projektet “Assignment A2 - The Gathering Web Scraper” för denna uppgift hittar du i din privata grupp under kursens grupp på GitLab.

Examination A2 (provmoment 2002) är en examination där du lämnar in genom att göra en Merge Request.

Deadline 2027-01-08 17.00
Deadline 2027-02-12 17.00 (omexamination)
Deadline 2027-08-19 17.00 (omexamination, sista tillfället denna kursomgång)

Betygssättning

Denna uppgift betygssätts med U/G och meddelas genom återkoppling på Merge Request i GitLab samt via Ladok senast 15 arbetsdagar efter deadline.

Poängrapportering i Ladok

Detta är ett examinerande, poängsatt moment i kursen. Poäng rapporteras in när uppgiften är rättad och bedömd, senast 15 arbetsdagar efter deadline.

Introduktion till uppgiften

I denna inspelning, som dels är ett lösningsförslag till “The Promising Web Scraper”, ges även en introduktion till examinationsuppgiften.

Guider

Uppgiftsbeskrivning

In this assignment, you get to help three friends, Peter, Paul, and Mary, plan their once-a-month gathering. Once a month, they gather to watch a movie and then dine at their favorite restaurant. It is quite difficult for the three friends to plan a gathering because they first have to find a day when everyone is free, then a movie, and finally a time when a table is available at the restaurant.

The friends have their calendars online on a website, and the cinema and restaurant they visit also have websites. The friends realize that the planning of an evening should be automated but do not know how to do it. They know that they do not need a GUI and want to run the application from a command prompt and that Peter prefers Windows, Paul prefers macOS, and Mary prefers Linux (#1).

Your task is to help the three friends by writing a web scraper that scrapes and analyzes data from the mentioned websites (explicitly built for this task). The application should be so generally written that it can handle two different start URLs, https://courselab.lnu.se/scraper-site-1 and https://courselab.lnu.se/scraper-site-2. (#3)

The start URL must be passed as an argument when starting the application. The URLs are each leading to three other websites. It is irrelevant how these websites work under the hood. The interesting part is how the HTML is rendered and how to make HTTP requests to retrieve data. It does not matter which URL the application starts with because the websites are nearly the same, and only the data scraped differs, and thus the suggestions the application finally lists.

The websites

From the start URL, the application must be able to crawl all three websites on its own. Data must be scraped and analyzed, and suggestions for the day, movie, and time to book a table must be listed. (#4)

Regardless of the start URL, the HTML of each website is nearly the same. The differences exist to ensure that your scraping code is as general as possible and that as little as possible has been hard-coded. The minor differences that exist are:

href attribute values in HTML.
Days when the three friends are available.
Movie titles, when they start, and if there are any places left.
The tables the friends can book and the redirect URL when logging in.

The varying href attribute values verify that your scraping code does not use hard-coded URLs. URLs defined in the websites' client-side JavaScript, such as the URL the cinema uses to request showtimes, will not change. You can, in other words, hard-code these.

The start URL

The application must start by scraping the links on the web page given by the start URL and continue.

The calendar

The web pages are built using simple HTML, and the task is to scrape the pages and analyze which days all three friends can gather. The friends have already decided to meet on Fridays, Saturdays, and Sundays, so there is no need to look for other days (#5).

The cinema

The cinema website is a simple website that displays the cinema's shows for the weekend. You can find out what day and time a particular movie is shown and if it is sold out. By analyzing the traffic between the client and the server, you should find a way to request this information in your application and filter out suitable movies based on scraped data from the calendars.

Hint

Use the browser's inspector to analyze the traffic.

The restaurant

The third website is for the three friends' favorite restaurant (the only one they visit). The application must log in using the credentials below to scrape what times they can book a table.

Environment variable	Value
`RESTAURANT_USERNAME`	zeke
`RESTAURANT_PASSWORD`	coys

The credentials must not be hard-coded in the source code. The application must read them from the environment variables above, loaded from a .env file using Node.js' built-in support (node --env-file-if-exists=.env), so no userland module is needed. The .env file must not be committed to the repository; commit a .env.example file containing the variable names (but not the values) instead. Use the exact variable names above. Variables already set in the environment take precedence over the .env file, which lets the examiner run your application without creating a .env file. If any of the variables is missing, the application must exit with a clear error message and a non-zero exit code. (#15)

The website uses session cookies for authorization, which the application must handle. After a successful login, the scraper can find available times.

Hint

The built-in fetch follows redirects automatically. To get hold of the session cookie set by the login response, you may need to use redirect: 'manual' and read the set-cookie header yourself.

The workflow to automate

Check which day, or days, all friends are available. If there is no day when everyone is available, the application must print No day found when all friends are available. to the terminal window and exit (#4).
Scrape available movies for the possible days.
Log in to the restaurant's website to determine when the three friends can dine. They want to book a table at least two hours after the movie starts (#6).
By analyzing the scraped data, compile suggestions containing the day, the movie with start time, and between which times to book a table.

Optional

Pre-book a table using the restaurant's form. (#8)
Write unit tests for the analysis logic, i.e., the matching of available days, showtimes, and free tables. Add the script test to the package.json file.

Requirements

You must use the repository created for you and this assignment and make continuous commits, so it is possible to follow the web scraper's creation. Make sure that no more files than necessary are committed to the repository. (#9)

You must write the web scraper as a Node.js application (#1) and follow the course's coding standard (#10). The application must run on Node.js 22.9 or later; specify this in the engines field of the package.json file. You must split your source code into several modules (#12). Each step in the workflow must be implemented as a function that returns its result instead of calling the next step. The workflow must be orchestrated in one place, for example in the application's entry point, where the result of each step is passed as an argument to the next. (#12) Of course, you need to document and comment on the source code (#11).

After cloning the repository with the application's source code and running the npm install command, it must be easy to lint the source code and run the application. Therefore, add the script start and lint to the "scripts" field in the package.json file. The scripts must work on Windows, macOS, and Linux. (#14)

The user must pass the start URL as an argument (#2, #3). That is the only input the user should do during the scrape process (#2). Below is an example of how to start the scraper using https://courselab.lnu.se/scraper-site-1 as the start URL:

npm start https://courselab.lnu.se/scraper-site-1

If the start URL is missing or is not a valid URL, the application must exit with a clear error message and a non-zero exit code. The same applies if a network request fails; the application must not crash with an unhandled error. (#2)

You can use whichever userland module you want (except if there is one with the complete solution). Prefer built-in Node.js functionality, such as fetch and util.parseArgs(), over userland modules when it does the job. Do not forget to ensure, as far as possible, that a module is safe to use; at a minimum, npm audit must not report any vulnerabilities. (#13)

According to the details below, the web scraper must present the suggestions in a terminal window (#7).

The output

Using the first start URL

If https://courselab.lnu.se/scraper-site-1 is used as the start URL, the list of suggestions must be exact:

Scraping links...OK
Scraping available days...OK
Scraping showtimes...OK
Scraping possible reservations...OK

Suggestions
===========
* On Friday, "Keep Your Seats, Please" begins at 16:00, and there is a free table to book between 18:00-20:00.
* On Friday, "A Day at the Races" begins at 16:00, and there is a free table to book between 18:00-20:00.

The result should not be more verbose than this. Be sure to delete all other statements printed to the terminal before submitting. The suggestions above are the correct ones for the first start URL, and you can use them to verify the output from your scraper.

Using the second start URL

For the second URL, https://courselab.lnu.se/scraper-site-2, some of the data and URLs have changed. Your web scraper must be able to produce the correct suggestions for this start URL as well. The list of suggestions from the scraper must be exact:

Scraping links...OK
Scraping available days...OK
Scraping showtimes...OK
Scraping possible reservations...OK

Suggestions
===========
* On Saturday, "Keep Your Seats, Please" begins at 18:00, and there is a free table to book between 20:00-22:00.
* On Sunday, "Keep Your Seats, Please" begins at 18:00, and there is a free table to book between 20:00-22:00.

Hints

Make sure to do the "Promising Web Scraper" exercise.

Return values, don't chain calls

A common mistake is to let each function call the next step when it is done, so that scrapeLinks() calls scrapeAvailableDays(), which calls scrapeShowtimes(), and so on. This results in deeply nested calls, where no function can be reused or tested on its own, and where the workflow is hidden inside the call chain.

Avoid:

async function scrapeLinks (url) {
  const links = /* ... */
  await scrapeAvailableDays(links.calendar) // Calls the next step.
}

Prefer:

const links = await scrapeLinks(startUrl)
const days = await scrapeAvailableDays(links.calendar)
const showtimes = await scrapeShowtimes(links.cinema, days)
const tables = await scrapeReservations(links.restaurant, showtimes)
const suggestions = createSuggestions(days, showtimes, tables)

Each function does one thing and returns its result. The workflow can be read top to bottom in one place.