A1 Part 2 - Agentic Testing

THESE INSTRUCTIONS MIGTH CHANGE!

In this part of Assignment 1, you will investigate:

How can you build evidence that agent-generated code actually works?

First complete and pass A1.a1 (alternative 1) or A1.a2 (alternative 2). Then complete the PromptLab lab described below and participate in the assessed seminar for A1 Part 2.

Agentic Testing

A coding agent can change code and run tools in a repository in response to a task you describe. In this lab, you use PromptLab to give an agent a task, inspect its changes and evaluate the results.

Your role is to decide what the software should do and to judge the evidence that it works. An agent's statement that a feature works is not enough: inspect tests, test results, code changes and your own observations.

Access and preparation

ActionHub and PromptLab are only accessible from within the University Network. You must either be on campus or connect to the campus network using eduVPN before accessing them.

Links to the tools, the starter repository and the names of this course's PromptLab assignments will be published here later.

Complete the lab during your scheduled lab sessions so that teaching staff can observe your work. Kalmar and Växjö campus students attend in person. Only WP distance students participate online.

Starting point: Agent Arena

You will receive working starter code for Agent Arena, a game in which the player collects items, earns points and encounters hazards. The starting game also has power-ups and a leaderboard.

You do not need a repository or completed assignments from another course. Use the starter repository provided for this course. Run the game and try its existing features before changing it.

Feature brief: combo system

Extend Agent Arena with a combo system:

  • Collecting items in quick succession builds a combo and increases the points earned.
  • The player can see the current combo or score multiplier.
  • Waiting too long between collectibles ends the combo.
  • Hitting a hazard ends the combo.
  • Starting a new game starts with a fresh combo.
  • The existing game, power-ups and leaderboard continue to work.

Some details are intentionally open. Decide how the combo should behave before asking the agent to implement it. Record your choices, including what counts as collecting items in quick succession and how the combo affects the score. These choices give you something concrete to test against.

Lab stage 1: implement the feature

Use the PromptLab assignment indicated by the teaching staff for implementation and link the starter repository to it.

Formulate a task asking the coding agent to implement the combo system. You may use the PromptLab AI coach or another AI tool to help formulate your prompt.

When the run is complete:

  1. Run the game locally and try the combo system.
  2. Inspect the generated code changes.
  3. Answer the review questions shown in PromptLab.
  4. Fix any obvious problems and inspect the result again.
  5. Merge the implementation: accept the reviewed changes into your repository.

This merged version is the starting point for the verification stage.

Lab stage 2: build evidence

Use the separate PromptLab assignment indicated for verification and link your updated repository to it.

The purpose of this stage is to investigate and improve the evidence that the combo system works correctly. Formulate a focused verification task rather than simply asking the agent whether the feature works.

Useful evidence could include:

  • Automated tests with clear expected results.
  • Checks of edge cases, such as the timeout boundary, a hazard during a combo or starting a new game.
  • Tests of interactions with existing features, including power-ups and the leaderboard.
  • Results from running existing tests and static checks.
  • Systematic game testing with recorded actions, expected results and actual results.
  • A short record of what was tested, what happened and what remains uncertain.

Choose evidence that addresses your stated combo behaviour. Do not treat a list of generated tests as proof that those tests ran or passed: inspect the actual results.

If the task needs several steps, use Continue From Last Run to continue from the previous run within the verification assignment. If verification reveals a problem, the agent may fix it and verify the result again.

When finished, inspect both the verification evidence and any additional code changes.

Final decision

Decide whether you would approve the resulting changes. Base your decision on the collected evidence, rather than only on a quick play-through or the agent's summary.

Record your decision and any remaining limitations. Merge the final result when you are satisfied with the evidence.

Reflection in PromptLab

In addition to the review questions shown in PromptLab, briefly answer:

  1. Before the verification stage, how confident were you that the feature worked correctly?
  2. What new evidence did the verification task produce?
  3. Did verification reveal any problem or limitation you had not noticed before?
  4. Which piece of evidence was most useful, and why?
  5. Is there anything you still cannot confidently verify?
  6. Would you approve this change in a real project? Why or why not?

Short answers or bullet points are enough. Keep your prompts, relevant run results, code changes and verification evidence available for the seminar.

Assessed seminar

  • Kalmar and online: Seminar is on Tuesday week 51 (see schedule for your group).
  • Växjö: The seminar has not yet been booked. Its date, time and detailed arrangements will be announced later.

The seminar is the examination for A1 Part 2. You will work with what you have learned in the PromptLab lab, applying your testing approach and explaining your decisions. The assignment and the reflection must be completed before the seminar!

Be prepared to:

  • Formulate a verification task with clear expected behaviour.
  • Inspect and evaluate evidence produced by an agent.
  • Explain how your confidence changed between implementation and verification.
  • Explain which evidence supports your approval decision and what remains uncertain.

Participation is required. Kalmar and Växjö campus students attend their assigned seminar in person; WP distance students participate online. Submitting lab results alone does not replace the assessed seminar.

Grading

A1 Part 2 is graded Pass/Fail and is included in the grade for Assignment 1.

You must pass both your chosen alternative (A1.a1 or A1.a2) and A1 Part 2 before an A1 grade can be awarded. Completing only one of these stages is not sufficient.