Marked Down

As a sort of prelude to the article below, I should point out that I do have an AI policy that boils down to “I use AI to revise, not to write”.

But I do want to understand the limits and possibilities of what can be done and get a yardstick for both quality and reliability, so over the weekend, as the brand new Portuguese national exam system melted down, I gave my usual agent a set of conventional web search tools, a browser, and a clear set of directives:

You are an investigative reporter.

Research Portuguese news and government sites and prepare a dossier on the procurement, implementation and roll-out of the new national exam grading system.

Gather interviews and press statements from government officials, teachers’ unions, schools, and political commentary.

Verify those statements and gather substantive facts to build a comprehensive research dossier towards an Economist-style news article.

Give me that article in Markdown and PDF format, written according to the Economist’s published writing and editorial style guide.

I also gave it an early version of Memento, my new agentic memory system, but merely as a record of facts, and not really as a tool for it to reason about them (it’s not quite ready for that yet). But as you’d expect, the AI took to it swimmingly:

Absolutely. I’ll apply the anti-trope and drafting-restraint rules as a hard line-edit layer—especially removing inflated stakes, empty pivots, canned contrasts, false certainty, needless jargon and cadence-by-repetition—without a specific personal style. The result will retain an Economist-like register: dry, compressed, analytical and occasionally wry, with conclusions proportionate to the evidence.

I ran this on Friday, over the weekend and, again, today (essentially asking it to update the dossier) using my Codex subscription and gpt-5.6-sol, and it was fascinating to see it double-checking and auditing its own research, even if its thinking still had the characteristic Californian enthusiasm that I find so grating:

I realized that I made a mistake: it’s about exam copies, not grades. Now, I need to browse for relevant links related to this topic. My goal is to extract links that contain titles associated with exam copies. I want to make sure I’m focusing my search appropriately so I can gather the right information for what I need. Let’s get started on finding those links!

It was also very precise about the interpretation of facts, and attribution of statements:

The reconciliation has already corrected the most consequential headline: 63,000 refers to digitised exam copies not yet distributed, not 60,000 pupils without grades. The ministry says 227,000 of 290,000 scripts had reached more than 100,000 pupils. Some grades were still missing, but the public material I have found does not attach the 63,000 figure to them.

And it delegated to sub-agents with amazing prompts like:

Act as a ruthless copy editor. Review the attached drafts for accuracy, contradictory references, manufactured contrast… against the dossier and transcripts in....

All in all, it was a fascinating thing to do (and watch being done), and the results, I think, speak for themselves. And yes, it picked the title by itself…Marked downPortugal’s digital exam-marking system has given the government some difficult answers

Late results, wrong grades and an emergency sitting have turned a technical reform into a test of administrative competence.

LISBON – At 10.19am on July 17th Portugal’s education minister brought good news to parliament. Every national examination had been marked, he said, and the National Examinations Jury held all the grades. More than 166,000 secondary-school pupils had already waited three days beyond the original publication date. University applications opened the following Monday.

Teachers were reportedly marking papers that morning. Schools received result files at around 7.30pm, after their offices normally closed. Some kept staff late; others reopened over the weekend. EduQA, the public institute running the operation, later said that about 1,400 of 290,351 examinations were undergoing additional checks when it authorised release. Pupils awaiting grades found -3 in the ENES admissions system. The code meant “suspended”; some school software displayed it without explanation.

On July 20th officials found that an automatic answer key for Physics and Chemistry A had been configured wrongly. Rerunning that question’s marking changed 12,613 grades: 11,496 rose and 1,117 fell. Later that day the jury reported faulty settings for two Geografia A questions. Schools received new lists and registration for that subject’s second phase was extended. A separate export error had already required another set of replacements.

By July 21st some pupils were sitting second-phase examinations while first-phase grades were being settled. Fernando Alexandre, the education minister, expected everyone to have a final grade by the end of the week. He conceded that pupils awaiting grades or corrections had already been disadvantaged. An extra sitting in September was announced for them; its rules and dates were being prepared.

The reform promised faster, auditable marking. A misconfigured answer key changed grades across the country.Paper chase

The examinations were taken on paper. The innovation began afterwards. Scripts were transported, scanned and divided into individual answers. Software distributed them to teachers, who marked through an online interface. Grades passed through quality checks, the examinations jury and school systems before reaching pupils.

Trouble began with the paper itself: staples hid QR codes, folded pages produced cropped images, continuation sheets disappeared and some scripts missed secure transport. Teachers waited for credentials, received incomplete answers and repeated work when corrected images arrived. On July 13th one mathematics teacher received 160 new items less than a day before the deadline. Further batches arrived after extensions had expired. Marking continued on results day.

The allocation of markers was equally muddled. On July 16th Mr Alexandre attributed the remaining work to a shortage of available teachers. The following evening he said willing teachers awaiting assignments had contacted him, even as the jury reported a shortage. More than 100 people registered for Portuguese had recorded zero scripts, he said. Their eligibility and assignment status remained unclear. On July 21st he described the organisation of 11,000 markers as deficient and proposed formal contracts. In five days the diagnosis had advanced from too few teachers to too little organisation.

The official timetable shifted too. On June 29th marking was on schedule and every pupil would be protected. Two days later Mr Alexandre dismissed most reports of failure as false. Publication then slipped from July 14th to the 17th, and the second examination period was postponed. By July 16th even the revised date was in doubt. The following evening he acknowledged that “the process did not go well”.

Schools inherited the final rush. Directors received incomplete files around closing time and were told to publish that day. Mr Alexandre threatened to seek explanations from directors who missed publication. Pupils could consult grades in “their applications”, he said, leaving each school to choose the application, login arrangements and publication time. Some pupils saw results that night; others waited through the weekend. Publication depended on each school’s staff and local software.

EduQA completed central validation on Saturday and made replacement files available from Sunday. The subject-level corrections and export repair followed. By Monday a grade could be present, corrected or suspended, depending on which file a school had received and when it had published it.A system in pieces

Early accounts referred to “the platform”. The name covered a chain of systems and oper…