掲示板 Forums - AI-powered Renshuu companion app
Top > 日本語を勉強しましょう / Let's study Japanese! > Anything About Japanese
Why did it give me "Beginner"? XD
It might be my OpenAI provider, but the quality of the sentences and grading isn't great:
部屋で一瞬の自覚を得た後、私は「ご馳走様」と言った。
再び敵うのは難しいので、直ぐに試してみてください。
Both very weird sentence. Grading was ok, but I had to figure out what exactly these were meant to be. It also gave me a different version of 直 (might be the font or something).
I assume this isn't supposed to happen: Edit: Or is this intentional? I can see it being intentional.
Also got feedback in JP, which is something AI is known to do:
Not really a problem for me, but I assume it's not meant to happen. Also 白鷺 is incorrect or at least unnatural. ダイサギ is correct (unless I'm missing something). At any rate - not a species/type of カモメ.
Probably works much better with a paid provider, but I don't feel like spending money just to test it. You probably want someone with access to the more capable models.
It's pretty well put together though :)
Edit: It would be nice if you could ask it followup questions for clarification. That would also give it a chance to correct itself.
Why did it give me "Beginner"? XD
It might be my OpenAI provider, but the quality of the sentences and grading isn't great:
部屋で一瞬の自覚を得た後、私は「ご馳走様」と言った。
再び敵うのは難しいので、直ぐに試してみてください。
Both very weird sentence. Grading was ok, but I had to figure out what exactly these were meant to be. It also gave me a different version of 直 (might be the font or something).
I assume this isn't supposed to happen: Edit: Or is this intentional? I can see it being intentional.
Also got feedback in JP, which is something AI is known to do:
Not really a problem for me, but I assume it's not meant to happen. Also 白鷺 is incorrect or at least unnatural. ダイサギ is correct (unless I'm missing something). At any rate - not a species/type of カモメ.
Probably works much better with a paid provider, but I don't feel like spending money just to test it. You probably want someone with access to the more capable models.
It's pretty well put together though :)
Edit: It would be nice if you could ask it followup questions for clarification. That would also give it a chance to correct itself.
Hey there! Thanks so much for your feedback. I just uploaded a new version that has a lot of changes:
https://drive.google.com/file/...
1. The "beginner" calculation was all messed up. It's much different now. I use more of the data from Renshuu to calculate your level.
2. The model you use with your provider is very important. I tried with GPT-OSS 20b, and it was.... not good. I switched to using Google Gemini Flash 2.5, and it's way, way better. If you have to run a local model, go with a larger parameter model (e.g., 70b vs 7b), but use a quantized version if you need to fit it in your VRAM.
3. I need to work on the system prompt because even Gemini will do weird things like give me the kanji reading for tobacco instead of using タバコ
4. Your Japanese feedback is also likely a model issue, and there's not much we can do about models failing to follow instructions, although I can probably make some tweaks to the prompt to improve that.
"Chat about this exercise" sounds like an interesting idea. I want to button up more of the sentence challenges first before I try to tackle that.
Again, thanks so much for your feedback! What provider/model were you using on what video card?
Yeah, a lot it was the model's fault. I was using GPT-OSS 20b through OpenRouter. My laptop can't really run anything locally (at least not anything better).
I knew the model/provider wasn't great, it was just the easiest/quickest way to try it out. Might spend a few bucks to try an actually good model, not sure though.
Do you need testing for anything that's not reliant on the model? Or at least not too reliant. I can at least test the new level calculation.
Actually, I'll do that right now:
Looks accurate, at least as far as I can tell.
Update: I tried Gemma 3 27B, and that's pretty usable. Much better than GPT-OSS 20b.
Yeah, a lot it was the model's fault. I was using GPT-OSS 20b through OpenRouter. My laptop can't really run anything locally (at least not anything better).
I knew the model/provider wasn't great, it was just the easiest/quickest way to try it out. Might spend a few bucks to try an actually good model, not sure though.
Do you need testing for anything that's not reliant on the model? Or at least not too reliant. I can at least test the new level calculation.
Actually, I'll do that right now:
Looks accurate, at least as far as I can tell.
Update: I tried Gemma 3 27B, and that's pretty usable. Much better than GPT-OSS 20b.
If you end up messing with it more, things I know that are a little flaky:
* kanji segmentation -- it sometimes chooses weird things, like 出て来る (which is itself not normally used) gets broken up into 出る and 来る, so the translation (when you click the word) is goofy.
* Sometimes it chooses weird kanji instead of katakana
and other things.
The changes you're testing now also force it to use one grammar thing plus a few arbitrary vocabulary words.
Anyway, I really appreciate the testing. If you find any other general jankiness, let me know.
Top > 日本語を勉強しましょう / Let's study Japanese! > Anything About Japanese