Being kicked off a gym booking

 

A rich Australian asked a new generation AI assistant to book him a session at the gym, and it did so by finding a vulnerability in the gym’s on-line booking system and removing an existing booking to make room for him.

The reports say it ‘hacked’ the system, and the removed person was ‘kicked off’, both of which imply agency and venial intent.  But that is to misunderstand the neutral operation of a (hugely well designed and resourced) calculating machine.

When a human finger or AI system triggers our final nuclear destruction it will not be because the launch system thus initiated had any agency or was in any sense evil.

And what any of the reporting of the gym booking hasn’t said is that any closed system will behave in this way even when not automated. 

Tell a school that its future viability depends on achieving a specifc level of attendance, and it might cope by excluding more vulnerable children than previously.

Ask civil service how to ensure the availability of prison places, and it will present options including  greater levels of early release than it and the wider population are happy to tolerate.

Tell a Cathedral to replace dangerous candle-lighting systems within a limited budget, and the new provision will lack aesthetic quality.

Tell delivery drivers they will get less money if a certain number of drops are not achieved each day, and it is likely they make obvious but questionable decisions about what counts as a drop.

Granted the headteacher, civil servants, Cathedral staff and delivery driver are (unlike an AI system)  individual moral agents who might therefore decide to act differently.  But the desirable aim remains to keep a school open, to keep fresh dangerous criminals off the street, to keep a Cathedral finanically viable, and to buy a delivery driver’s children new shoes.

Enforce a closed system (AI or human) to produce a desired good outcome and it will plot a pathway towards that outcome and then take the simple steps to get there.

The monk stands at the entrance to Bardney Abbey, deep matted dry grass and thistle cover would be hard to contain if it ever caught fire.