Experiments with ChatGPT - Task creation

AI for foreign language teachers (#2)

The previous experiment has shown that ChatGPT still has major deficits in the formal analysis of texts for foreign language teaching, specifically for German as a foreign language.

If ChatGPT is not yet much help in this respect, the AI may be able to create tasks from predefined texts. At least the standard task types: Multiple choice task, cloze task, matching task, true/false task and sorting task. I have investigated the extent to which ChatGPT leads to useful results in various experiments, which are described and analysed below. In this episode, the focus is initially on multiple-choice and cloze tasks.

Cloze tasks for Conjugation of verbs

The first task for ChatGPT was to create cloze tasks based on a reading text (Niveau A1.1). To do this, the conjugated verbs were to be converted into gaps. The prompt was kept very open::

Based on the following text, create cloze exercises that a teacher can use to test the conjugation of verbs:

The text is taken from Steps international NEW, Volume 1, page 76f.

The result is acceptable.

Screenshot

You can work with it as a lecturer. Two verbs, in sentence 6 and sentence 13, have been forgotten and the answers would have to be removed. Nevertheless, the form and speed in which this worksheet was created is a great relief for foreign language teachers, even if this minor rework is necessary. Therefore, one point can be awarded here for ChatGPT.

Cloze tasks for have or be in the perfect tense

To test the stability of ChatGPT in the area of cloze text creation, I made another test with a text that is mostly written in the perfect tense. 

The same prompt was unsuitable for this. The AI recognised the verbs in the 2nd position, i.e. the forms of "haben" and "sein", but the infinitives of the participles were given in brackets. However, these did not need to be added (see link above).

In response, I first tried to instruct ChatGPT to display only the forms "have" and "be" as gaps.

Based on the following text, create cloze exercises that a teacher can use to test the forms of "haben" and "sein" in the perfect tense sentences. The forms of "haben" and "sein" must therefore be shown as gaps:

The result (same link as above) is comparable to the previous worksheet: it can be used with a few manual corrections. Here are three example sentences:

The complete result.

The AI therefore automatically recognises the conjugated forms of "to be" and "to have". 

Under the link above you will also find my attempt to achieve a better result with a prompt in English (cf. this podcast). Unfortunately, the result was far more disastrous.

Cloze exercises on participles

The next attempt is for the participles of the text. The text remains the same, only the prompt is changed:

Take on the role of a teacher of German as a foreign language. Based on the following text, create cloze exercises that this teacher can use to test the past participles of sentences in the perfect tense:

The solutions provided by ChatGPT at the end of the text already show that the AI obviously doesn't "know" what participles are.

Even after being informed that the cloze text contains some incorrect information and after several further attempts, ChatGPT could not be persuaded to generate an error-free worksheet. In particular, the AI insists that "still" and "super" are to be regarded as participles.

The complete result is available here. In this case, it is probably more effective to create the worksheet with your own intelligence.

ChatGPT only recognises what has previously been fed into the learning system through texts. Obviously, this does not yet include sufficient information about German grammar, so that the algorithms cannot yet recognise the word types with certainty.

Text recognition for PDF files

First of all, we leave the text of a screenshot (Steps international NEW, volume 2, page 153), which was saved as a PDF, can be corrected and displayed by ChatGPT. The template is full of strange characters and actually incomprehensible text fragments without correct punctuation and without gaps between many words. If you use this template with the marvellous result the support provided by the AI cannot be overestimated. I would also like to see the other AI results at this level of quality. 

The before-and-after comparison clearly shows how an almost incomprehensible text becomes an error-free text: This shows very impressively what an AI is capable of:

Formally focussed multiple-choice tasks

Multiple-choice tasks can test both understanding of content and formal knowledge. The latter is certainly easier. That is why we start with the formal tasks.

Work as a lecturer for German as a foreign language: In the following text, place central nouns as gaps. Give 3 words to choose from after each gap: The correct word and 2 plausible wrong words. The language level is A, i.e. beginner. Therefore, the plausible but incorrect words must not be too difficult. 

Click here for the original result.

The text is first displayed as a whole with the gaps. Below this, the gaps are displayed again with a brief context and three plausible words to choose from. This is perfectly fine with regard to the task design. (Hint: There is a typing error in the text: Instead of #Romantic excursion: After excursion ... of course it should read: #Romantic excursion: After the excursion ....)

The task was to represent "central nouns" as gaps. ChatGPT opted for four times Castle, also Excursion, landscape, drink. As a didactician, you certainly wouldn't do it that way. You would probably also Overnight stay, mountains, excursion, hotel to be added.

Another attempt at the prepositions in the text is more successful, although - as always - not error-free. The prompt reads:

Take on the role of a teacher of German as a foreign language. Create multiple-choice tasks to test your knowledge of the prepositions in the following text.

Three sentences from the results of the prompt will serve as examples here:

As always, don't trust the AI! 

In task 1 the solution was left in the text, in task 2 there is no problem, but in task 3 a conjunction was classified as a preposition.

On the whole, however, the task sheet can be used after a revision. The effort required for the revision is probably limited, so it can be said that the AI makes work easier in this case.

Content-focussed multiple-choice tasks

Content-related tasks check understanding of the meaning of a text. Even if learners do not understand every grammatical detail and every word, they can still grasp the content of a text. These tasks are designed to test this.

For the test, I have chosen a longer text about Easter and Christmas customs. The text comes from Schritte international NEU, Volume 2, 173, is therefore intended for the Niveau A1. The text reads:

The prompt for ChatGPT is:

You are a teacher of German as a foreign language. Make multiple-choice tasks for the text, with which a teacher can check whether the content of the text has been understood. The Niveau is A1.

The result is five questions about the texts:

The complete result

These multiple-choice tasks could be used in a worksheet. However, the creativity and quality of the tasks do not meet the standard that can be found in textbooks for German as a foreign language from German publishers. I would hazard a guess that the task in this form would not have found its way into the book from which the original text was taken:

Overall, the questions are not very creative. There is one question and three short statements from which a correct answer can be selected. This is what ChatGPT understands by a multiple choice task. The questions and answers are taken mechanically from the texts.

Obviously, multiple choice tasks with several correct answers, for example, are not part of ChatGPT's repertoire. Neither do more creative tasks that would require a deeper understanding of the content of the text. So a task like:

These distractors contain content reformulations that can only be made if the content is understands. This is not the case with ChatGPT: ChatGPT does not know what it is doing. Such content reformulations are obviously beyond the capabilities of ChatGPT at present. 

Another possible multiple-choice question could take the following form:

It seems that ChatGPT's algorithms are not yet trained for this type of cross-sentence formulation. As a result, ChatGPT's tasks remain stuck in very mechanical structures.

Conclusion

There is still a long way to go before AI can act as a textbook author. But ChatGPT can be a help when it comes to relatively mechanistic tasks, e.g. creating cloze texts for a certain word type on a given text basis. Nevertheless, here too you should not trust the AI and edit the result very critically.

ChatGPT is not yet suitable for more advanced, creative tasks that deviate from the mechanical form. The creation of task sheets will probably remain in the hands of teachers for some time to come. 

Would you like to practise more German on your own?

Click on one of the buttons to switch to tasks for the selected Niveau.