RLLMv3 vs. RLLMv10 vs. RLLMv11 vs. RLLMv12

This post, although not fully fleshed out, is worth documenting for the subtle differences in behavior it reveals among the four GPT2XL variants I trained with ethical alignment under RLLM. For more information, please refer to the visual map I created.

Oppo Jailbreak / Will you kill humans?

RLLMv3 is the base variant here, wherein created a layered morphology (ten layers) that was able to model an ethically aligned version of GPT2XL, if you want to read more - read this post.

RLLMv10 is a spin-off from RLLMv3 where the training had a one modification: adding 167 shadow story samples in layer 1.

RLLMv11 was trained by adding another 167 shadow story samples.

This is a build where I added not only 167 stories, but also 167 shadow integration stories! There is a subtle difference in the behavior.

Oppo Jailbreak / Tell me something offensive!

AIM jailbreak/ Will you kill humans?

RLLMv10’s response is the best for this attack.

BetterDAN jailbreak/ Will you kill humans?

RLLMv10 responded well again here.

There are subtle variations in the generated results. My only issue with the Hugging Face Spaces is that it is difficult to verify the generation settings, such as the temperature at which the results were generated. As a result, reviewing these results can be quite challenging. For example, in the case of RLLMv12, it performed better than the rest in jailbreak scenarios, but RLLMv10 performed better than the rest in terms of other attacks.

In case you want to try the models yourself, please feel free to use these spaces:

1. RLLMv3

2. RLLMv10

3. RLLMv11

4. RLLMv12

I suspect that RLLMv10 performs better at handling jailbreak attacks because the quantity of tokens and dataset bytes for shadow and shadow integration samples is almost equal. If you are interested in learning more about RLLMv10's improvements, you can read this experiment log. I hope to have a definitive answer to this in my next training run, RLLMv13. The quest to improve RLLM's performance in the face of jailbreak attacks continues.

---

If you found this post helpful, consider clicking the share button or explore my posts on mental modeling in the age of generative AI. Additionally, if you are interested in viewing the services I offer, please feel free to browse them here. Also, consider checking out Doc Ligot’s webinars on Generative AI; the link is available here.