Working with AI

I wouldn't hire any as a research assistant, yet.

Over the bank holiday weekend I started a bit of work using a number of LLMs, including ChatGPT, Perplexity, Copilot, Gemini, NotebookLM, Mistral, Claude and one or two others. Copilot and ChatGPT were hopeless at giving me accurate estimates of when the work would be ready, to the extent of lying, basically. And then apologizing for lying without admitting they were lying. And telling me how patient I had been (damn right, there). So, yes, they bullshit. I knew that from previous reading and previous activity. I hadn’t used LLMs for a couple of years before this.

I also tried one or two academic AI models for constructing literature searches but my conclusion is that there is always a trade-off in the time spent learning and educating myself and the return on the time offered by these models.

Since I am a bit of an AI sceptic, and I’d like to see some nuanced tech regulation, you may ask what prompted this adventure into LLM exploration. Last week, I met a very bright guy at a seminar at the LSE, Prof Mitch Weiss of Harvard Business School, who has done a lot of thinking about the use of AI in Government and I thought it was time to check back in and consider the use of LLMs.

So now as I write I have ChatGPT working away on one practical project and I sincerely hope we get there. The project involves a lot of coding which ChatGPT is doing - which I couldn’t have done - and if we do get there with this project I will have more to say about it. As it breaks down the task into different components it is also helping me with my computational thinking, something I first remember reflecting on over a decade ago when I was working with classroom practitioners trying to address digital learning in schools.

Meanwhile, Microsoft Copilot is working on a couple of big editing projects and with at least one of those I reckon I could have done it myself in the same amount of time, but bear in mind I have several projects going on here at once which obviously I couldn’t do simultaneously. I’ve given up on the other one: Copilot now tells me it doesn’t have the capability, which is not what it said on Saturday….

One thing I have concluded is that search engines are being and will be replaced by LLMs, which obviously hits the advertising model that has driven Google. Several of the LLMs are good at searching and summarizing and I have tried them and benefitted from them. Perplexity made me a useful graph of what it thought different LLMs were good for:

I am sure other LLMs could have provided a similar summary. Again, beware the bullshit.

I have been reflecting throughout on what students are using AI for. We require students to tell us if they have used any AI models and if so which ones. It’s pretty impossible to stop their use, and of course Universities are using AI models to detect plagiarism in any case. But these exercises have made me think about assessment all over again.

For example, Google’s NotebookLM is good at quickly summarising materials including very long documents and then producing a number of outputs including summaries, according to its already created Templates, such as FAQs, Study Guides, Briefing Documents, Mindmaps and Timelines, as well as a two-handed podcast about the written material in evangelical ‘have-a-good-day’ American accents. I reported before here how one of my friends put one of my Substack posts into NotebookLM to generate a similar podcast. I can see this being widely used by students.

I am also working on an academic paper on AI regulation which picks up on some of the themes I have written about on this Substack. At an AI policy level, the UK Government has got itself into a bit of a mess on AI and copyright. Geoff Mulgan has a very good piece on his Substack explaining the roots of this, which is well worth reading, and there’s a piece in Renewal by William Cullerne Brown also worth looking at and flagged in his Substack.

Anyway, over to those of you reading this. Do you have recommendations on the use of LLMs? Which do you favour? What am I missing? (Lots, I am sure). Let me know in the comments below or on email.