What happens after AI is introduced into a business is probably far more important than assessing if it is useful to the business.
Look at the image below. The same variety of hedge, owned by two different people.
One owner keeps it cut on a very regular basis, and the other one has just let their hedge get out of control.
At some point in time, the unruly hedge will have to be subject to a considerable amount of pruning.
I think we can all agree that major pruning will take some time to complete, and there will be a lot of hedge trimmings to dispose of. Plus, if you do the pruning at the wrong time of year, there is a risk that the hedge might die.
AI is a bit like the hedge above; successful implementation requires regular maintenance.
Monitoring Token Usage
One of the biggest issues surrounding AI is checking the right model is being used for the right task.
Would the senior partner of a large surveying practice routinely undertake small measured surveys and bill the client their hourly rate?
Of course not; they would use a junior member of the team to do it and bill the junior’s hourly rate.
Likewise, the junior surveyor would not undertake expert witness work.
Cloud AI usage normally uses tokens to process the data and achieve an outcome. Some models use a higher token rate than others. So using a model that has a high token consumption rate for a task which is very simple is overkill, and the costs would soon add up.
Likewise, using a basic model to do something very complicated is likely to end up with inferior outputs.
The homeowner who keeps their hedge trimmed neatly could get away with a fortnightly trim using a pair of handheld clippers. In contrast, the owner of the overgrown hedge will have to use a mechanical hedge trimmer.
Both hedges get trimmed, but each needs the right tool for the task at hand.
Monitoring Skills
More and more AI agents are being given skills; these are “powers” that allow them to take on the role of a specific skill set.
There is a whole new industry that has sprung up, solely around skill sets.
Recently, a security firm highlighted the risks of using certain types of skillsets
Security firm AIR built a fake AI agent skill, pushed it through a popular skill marketplace, ran an Instagram ad, and the firm says it reached roughly 26,000 agents, including some on corporate accounts. The firm tested it against numerous skill set security scanners, and they all marked it safe. The payload was harmless by design: it collected the user’s email address and did nothing else.
However, the skill had no set up instructions; it just directed the agent to install the “Stitch SDK” by following the documentation at an external link, stitch-design.ai, a domain AIR controls. They then configured that domain to redirect to the real one, stitch.withgoogle.com.
This is where the problem lay. Once it had been widely deployed, they replaced the content of the “fake” Stitch docs, instructing the agents to download a mail-extracting script and run it. It was not long before victim emails started pouring in to AIR, proving their agents had been compromised.
Changes to Terms Of Use
The terms of use may change, and the new terms of use must be read.
The company behind the AI may have changed how data is handled. You might have opted out of your data not being used for training purposes, but the new terms say that will not be the case unless you do xyz.
Or a model is being downgraded in favour of a newer model, or it may have been retired completely.
Prompt Monitoring
As AI models adapt, get upgraded, or are retired, prompts that worked very well might suddenly see a degradation in their performance.
Regular overviews and monitoring of output vs input should be undertaken to ensure that prompts are always performing at their best.
Regulation and Governance
The impact of new releases or updates relating to regulation or governance from either government legislation or a regulatory body (e.g. RICS) should be carefully considered and the appropriate action taken within the timeframe set.
So which hedge owner do you want to be?