Experiments with ChatGPT - prepositions and separable verbs

AI for foreign language teachers (#1)

Foreign language teachers often endeavour to use current texts in foreign language lessons. As these are generally not yet didacticised, it is necessary to analyse not only the content but also the grammar and vocabulary of the texts. It would be desirable if this tedious work could be done by AI systems in just a few seconds. I have conducted experiments with ChatGPT on prepositions and separable verbs to test the usefulness of ChatGPT for foreign language teaching. Read this text to find out more about how ChatGPT can be used in foreign language teaching to analyse prepositions and separable verbs.

In order to maximise the performance of the AI, specifically ChatGPT in the free version (ChatGPT May 24 Version), I conducted experiments for two scenarios, which I describe below. Firstly, the AI was "asked" to extract prepositions from a given corpus of sentences. Secondly, I investigated how ChatGPT could support foreign language teachers in identifying separable verbs.

Experiments with ChatGPT - Listing of prepositions

The first task for ChatGPT was to list the prepositions of the following nine sentences. In this presentation, the prepositions are marked in bold. This marking was omitted for ChatGPT:

1. The cat sits on the table.
2. The picture hangs on the wall.
3. The bag stands between the chairs.
4. Whilst I watch television during dinner.
5. The train runs at 12 o'clock from.
6. I stay to Home because of of the rain.
7. The Porsche drove with the river at high speed along.
8. On Tuesday training is cancelled from.
9. The children climbed in the wrong train a.

Deliberate pitfalls are marked in yellow. It should be investigated whether the AI recognises that these are prefixes of separable verbs or whether these prefixes are classified as prepositions.

... and how can ChatGPT help me with task creation?

English prompts

Since ChatGPT is a product of US companies, it is sometimes postulated that English prompts lead to better results. Prompts are the texts used to instruct ChatGPT to do something. For example, Markus Walsch notes in the podcast "Three times better - AI: How ChatGPT and co. can help you on the job" by Bayerischer Rundfunk (from minute 3:30):

"This thing (= ChatGPT, O.B.) is trained in English, this language model. And that's why it's only nearly as good in German, if at all."

For this reason, it was also investigated whether a prompt in English might lead to a different, possibly even better, result than the same prompt in German. 

In German, the prompt read:

Make a list of the prepositions in the following sentences.

In English:

Make a list of the prepositions in the following German sentences.

This was followed by the nine sentences from above, placed as a whole in inverted commas.

Results of the experiments with prepositions

The following table compares the results. Clicking on the buttons at the end of the table takes you directly to the original result in ChatGPT.

The comparison shows that a prompt in English actually leads to a better result than a prompt in German:

Experiments with ChatGPT - Identification of separable verbs

On other occasions I have noticed that ChatGPT has obvious problems distinguishing between separable and inseparable verbs. I therefore also carried out a test on this topic with prompts in different languages.

The corpus for the analysis consisted of the following ten sentences. For a better overview, the separable verbs are again marked in bold:

1. I rise on the train in Munich a and in Vienna rise me again from.
2. Tell me, when draw you actually at?
3. Come inIt's warm inside.
4. Come You calm inThe meeting has not yet started.
5. Edison invented the light bulb.
6. Can you please find the books for me bring back.
7. Dresden was largely destroyed in the Second World War.
8. Can you help me with the boxes to unpack.
9. Listen please finally to!
10. Excuse me, when drives this tram loose?

The corpus was compiled according to certain criteria. The reasons for the selection and contextualisation of the verbs are:

1. In a sentence, the same verb appears with two different prefixes. Does the AI recognise both verbs?
2. A relatively simple constellation: The separable verb is in the second person singular present tense. The expectation is that the AI should have no problems recognising the separable verb here.
3. It is an imperative in the you-form. The prefix is placed directly after the verb stem without any other parts of the sentence.
4. The same verb as before, but now in the imperative of the you-form.
5. There is no separable verb in this sentence. 
6. The separable verb is not separated here, but stands at the end of the sentence in the infinitive.
7. There is no separable verb.
8. The separable verb is in the infinitive with zu.
9. The separable verb in the imperative of the Ihr form.
10. The separable verb is used here in an interrogative sentence.

The two prompts in German and English were:

If there is a separable verb in the 10 sentences, mark all parts of the separable verb. 

If there is a separable verb in the 10 German sentences, mark all parts of the separable verb.

The result in a comparison:

In the case of separable verbs, the result is incorrect for both the English and the German prompt. Apart from the differences between the claimed and the realised formatting in the English version, there is one error less in the German version:

In a second attempt, in which it was explicitly stated that it was "10 German sentences" (10 German sentences), the result presented by ChatGPT is completely wrong in parts:

Sure! Here are the parts of the separable verbs in the given German sentences:

1. I get on in Munich a and in Vienna I get on again from.
2. Tell me, when you draw actually?
3. Come inside, inside it is warm.
4. Feel free to come inThe meeting has not yet started.
5. Edison invented the light bulb invented.
6. Can you please find the books for me bring back.
7. During the Second World War, Dresden was largely destroyed.
8. Can you help me with the boxes to unpack.
9. Please listen to me at last to!
10. Excuse me, when does this tram run loose?

In each sentence, the separable verb is indicated in bold.

Unfortunately, there is no direct link to the result, as ChatGPT does not allow links to different versions. The full result can only be viewed here as a screenshot.

Screenshot

Although parts of the text were actually displayed in bold this time, the system was obviously overwhelmed. The result has almost nothing to do with the task that ChatGPT was set. It is so faulty that it appears that ChatGPT has randomly marked some words in bold.

In order to perhaps be able to understand the cause of these errors, ChatGPT was also asked to provide a reason for the decision in the next attempt after the above English prompt:

If there is a separable verb in the 10 sentences, mark all parts of the separable verb. After each sentence, give a reason why the marked words are separable verbs.

The result was acceptable, but not perfect. In order not to make this text too long, I will abbreviate the result. For the reasons mentioned above, you can only view the entire text as a screenshot.

Screenshot

The system has now recognised almost all separable verbs. Only the separable verbs in the last two sentences were still ignored. After a hint that there could also be separable verbs in sentences 9 and 10, these two separable verbs were also recognised and displayed correctly like the others. However, the latter does not help a German learner, because if you already know the answer, you don't need to ask the AI. As a learner, I need to be able to trust the teacher or the AI. That is not the case here.

Since a justification of the decision led to a better result, I made the same attempt in the German version. I had expected ChatGPT to recognise "destroy" as inseparable. However, this was not the case.

ChatGPT experiments with separable verbs: "destroy" is classified as a separable verb

Screenshot

Only after the question "Is 'destroy' a separable verb?" was the error recognised:

Sorry, that was my mistake. "Destroy" is not a separable verb. It is an inseparable verb as it does not contain a separable prefix. Thank you for the correction.

Analysis of the corpus with Japanese prompt

Finally, the same corpus was analysed again with a Japanese prompt:

10個のドイツ語の文の中に分離可能な動詞がある場合、分離可能な動詞のすべての部分に印をつける。各文の後に、マークした単語が分離可能な動詞である理由を述べなさい。

The result corresponds approximately to the result of the German prompt: all separable and non-separable verbs were recognised correctly. The verbs were highlighted and provided with an explanation. The complete result can be viewed under this link: 

There was a surprise in sets 3 and 4. The system has the verb come in independent in insidecome changed. It was then correctly identified as a separable verb.

3. come in, it's warm inside.
分離可能な動詞: get in
「hineinkommen」は「hinein」（in）という前置詞を伴っており、動詞の「kommen」と分離しています。

In sentence 4, "come in" was not changed, but, what I had not considered at all, also the verb start analysed and wrongly explained as a separable verb: be- be part of the verb can.

4 Come in quietly, the session has not yet started.
分離可能な動詞: come in, begin
「herein」は「herein」（in）という前置詞を伴っており、動詞の「kommen」と分離しています。「beginnen」は「be」（ver-）という接頭辞を伴っており、動詞の「nnen」と分離しています。

Summary

After these experiments with ChatGPT on prepositions and separable verbs, we can conclude that you cannot rely on ChatGPT for grammatical questions, but must critically check all results and statements with your own knowledge, because some of the results can only be described as absurd.

This judgement is possible for a competent speaker, but not for someone who is hoping for help from the system because they are not yet proficient in the foreign language.

It also seems to be helpful as a foreign language teacher to formulate a prompt in different languages. The results can be very different. Perhaps a different language leads to a more acceptable result. However, the statement that English prompts produce better results is not true in this absolute sense. At least when analysing German-language texts, you should remain sceptical and also try out German prompts.

An improvement in the result can also be achieved if ChatGPT is instructed to give reasons for decisions. However, even in this case, the result must be checked by competent speakers.

Although ChatGPT is not yet a reliable tool for analysing texts in the foreign language classroom, it can serve as a starting point for grammatical analysis, which must be verified by a competent person.

Would you like to practise more German on your own?

Click on one of the buttons to switch to tasks for the selected Niveau.