Experiments in Artificial Intelligence

If the feedback from workshop leaders in IB workshops are an accurate guide to the topics that are most discussed in schools at the moment, then Artificial Intelligence is, perhaps, the number one cause for both excitement and concern in equal measure.

A lot has been written about the advantages of using AI in the creation of lesson plans but a worry is that too many students are using AI to avoid having to think.

For some time, many Physicists have felt secure in the knowledge that a weakness of some AI engines is their lack of mathematical fluency.  The argument goes something like “since so many problems in Physics really test a good conceptual understanding of the principles, surely an AI engine will either be unable to solve Physics problems, or the use of AI will be easy to identify in the steps in the calculations”.  

So we thought it might be a good time to put this to the test. We put one small section of a past IB paper question on into ChatGPT to see how well it coped.  The question we used was a straight forward projectile motion calculation in the context of a probe landing on a distant asteroid.  The test wasn't fair on ChatGPT ,however,  as the information provided in the small section missed out the crucial value of the acceleration due to gravity on the asteroid that was needed to answer the question.  Perhaps, subconsciously, we wanted AI to fail. 

The answer provided was really interesting.  ChatGPT could identify that it needed to know the acceleration due to gravity.  It went on to correctly solve the problem having made the assumption that g was 9.8 ms^-1.  It clearly highlighted this assumption in the answer and offered to redo the calculation if we could provide the value to use. 

When we gave ChatGPT the whole question to solve, it succeeded. Specifically, it was able to work out the value of gf from an earlier section and apply it correctly to the projectile motion question.   The version of ChatGPT used was readily available and free to access. The prompt used was "Please solve (and explain) these questions."  For the record, the question used was HL2, Nov2023, TZ2, Q1.

So, if mathematical calculations are possible, how about being able to answer questions aimed at testing a candidate's conceptual understanding? The InThinking site has a full set of IB practice papers.  We gave it question 1 from the HL paper with the same prompt as before:  "Please solve (and explain) these questions." This was ChatGPT's answer:

If you set question 1 as homework and had this handed in, would you be able to identify it as AI generated?  Please do share your thoughts / comments.

More on this topic to follow…