InkMind 探索開源大語言模型，選擇更合適的詩歌續寫模型

Exploring Open-Source Large Language Models for InkMind: Choosing a Better Poetry Continuation Model

Introduction

Large Language Models (LLMs) have become important tools in natural language processing (NLP). They can generate text, summarize information, answer questions, and assist with creative writing. Most modern text-generating LLMs use the Transformer architecture and learn patterns in language from large collections of text.

Our project, InkMind, focuses on poetry generation through text continuation. Unlike an instruction-based chatbot, InkMind receives an opening phrase or part of a poem and generates the following words. For example, given the input “Beneath the moon, I walk alone,” the model should continue the text in a coherent and creative way.

In our earlier assignment, we explored GPT-2, including tokenization, embeddings, self-attention, and text-generation parameters such as temperature, top-k, and top-p. For this assignment, we investigate three alternative open-weight language models: GPT-Neo-125M, Pythia-160M, and Qwen3-1.7B-Base. Our goal is to identify a model that could improve InkMind while remaining practical to run and experiment with using Google Colab.

1. Overview of the Selected Models

1.1 GPT-Neo-125M

GPT-Neo-125M was developed by EleutherAI, a research organization working on open language models. It is an autoregressive Transformer model with approximately 125 million parameters. Like GPT-2, it predicts the next token using the preceding text, making it suitable for continuation-based generation.

The model has a context length of approximately 2,048 tokens and is distributed under the MIT license. Its relatively small size makes it practical for experimentation on limited hardware, including a Google Colab T4 GPU. It can be loaded through the Hugging Face Transformers library without requiring a paid API.

GPT-Neo is a useful candidate for InkMind because its architecture closely matches our project's requirements. However, its relatively small capacity and general-purpose training mean that it may produce repetitive or inconsistent poems. Fine-tuning it on an appropriate poetry dataset could help it learn poetry-specific patterns.

Official model card: EleutherAI GPT-Neo-125M.

1.2 Pythia-160M

Pythia-160M was developed by EleutherAI as part of a model suite designed to support research into language models, their learning processes, and their behavior. It contains approximately 160 million parameters and uses a decoder-only Transformer architecture.

According to its official model card, Pythia was trained on The Pile, a large dataset containing diverse text sources. Its context length is 2,048 tokens, and its license is Apache 2.0. The model can be downloaded from Hugging Face and used with the Transformers library.

One advantage of Pythia is its research-oriented design. The Pythia project provides multiple model checkpoints, making it useful for investigating how training influences model behavior. For InkMind, we could use Pythia-160M as a starting point for fine-tuning on poetry and compare its continuations with those generated by GPT-2.

However, the model card explains that Pythia was not specifically fine-tuned for creative writing or instruction-following. Its ability to generate good poetry therefore needs to be evaluated experimentally.

Official model card: EleutherAI Pythia-160M.

1.3 Qwen3-1.7B-Base

Qwen3-1.7B-Base was developed by Qwen, the language-model team associated with Alibaba. Unlike the instruction-tuned Qwen3 variants, this is a base model that has undergone pretraining rather than additional instruction tuning.

The model contains approximately 1.7 billion parameters and supports a context length of 32,768 tokens. It uses a causal language-modeling objective and is distributed under the Apache 2.0 license.

Its larger parameter count gives it substantially more capacity than GPT-Neo-125M and Pythia-160M. This could help it represent more complex linguistic relationships, vocabulary, and patterns across longer passages. These characteristics make it an interesting candidate for poetry continuation, where maintaining coherence across multiple lines is important.

The model can be downloaded from Hugging Face without a paid inference API. Its weights require more memory than those of the smaller models, but loading it in half-precision (FP16) requires approximately 3.4 GB for the weights alone. A 16 GB T4 GPU should generally be sufficient for inference with sensible sequence lengths, although additional memory is needed for the context, activations, and other operations.

The main limitation is that a larger model is not automatically a better poet. We must test its outputs and consider the extra computational cost before selecting it.

Official model card: Qwen3-1.7B-Base.

2. Comparison of the Models

The specifications above come from the models' official Hugging Face model cards. Actual generation speed and memory usage will depend on sequence length, numerical precision, software versions, and generation settings.

All three models can generate continuations without requiring a paid API. Their licenses also permit use under their respective terms, although users should still check the relevant license and dataset conditions before distributing a fine-tuned model.

3. Which Model Should We Choose for InkMind?

After comparing these candidates, our proposed choice is Qwen3-1.7B-Base.

The first reason is its model capacity. With 1.7 billion parameters, it is substantially larger than GPT-Neo-125M and Pythia-160M. This gives us a reasonable basis for testing whether a larger pretrained model produces more coherent and varied poetry continuations. However, the actual improvement must be measured rather than assumed.

Second, Qwen3-1.7B-Base has a 32,768-token context window, compared with approximately 2,048 tokens for the other two models. Although InkMind may not need such a long context for short poems, the larger window provides more flexibility for longer passages and experiments with extended poetic context.

Third, it is practical to access. Its public model card provides the model weights and instructions for using them with common NLP libraries. We can experiment without paying for an inference API. The Apache 2.0 license is also suitable for research and adaptation, subject to its terms.

Finally, its memory requirements are manageable for inference on a 16 GB Google Colab T4 GPU when using FP16 and appropriate sequence lengths. Fine-tuning the full model would be more demanding, so we could investigate parameter-efficient fine-tuning methods such as LoRA if adaptation becomes necessary.

We should also acknowledge the trade-off: GPT-Neo and Pythia are smaller, so they may offer faster and cheaper experiments. If Qwen3 proves too slow or consumes too much memory, one of those models remains a reasonable alternative.

4. How We Will Evaluate the Choice

Before replacing GPT-2 in InkMind, we should conduct a small experiment. We will provide each model with the same opening phrases and compare their generated continuations.

For example:

• “Beneath the moon, I walk alone…”
• “The rain remembers every tear…”
• “Somewhere beyond the mountains…”

We will evaluate the outputs based on coherence, poetic language, creativity, repetition, and how naturally the generated text continues the opening. We should also measure generation time and peak GPU memory usage.

If possible, we will test Qwen3 both before and after fine-tuning on an appropriate poetry dataset. This will help us distinguish the benefits of a larger pretrained model from the benefits of adapting a model to our specific task.

Conclusion

GPT-Neo-125M, Pythia-160M, and Qwen3-1.7B-Base offer three useful approaches to text generation. GPT-Neo and Pythia are attractive because of their small sizes and accessibility, while Qwen3 offers a larger parameter count and a longer context window.

For InkMind, we propose Qwen3-1.7B-Base as the next model to evaluate because it combines a causal language-modeling objective with greater capacity and manageable inference requirements on a Colab T4. Nevertheless, the final decision should depend on actual poetry-generation results, memory usage, and speed.

The purpose of this experiment is not simply to use a larger model. It is to discover whether an alternative pretrained language model can produce better poetry continuations while remaining practical for our project.

References

1. EleutherAI. GPT-Neo-125M Model Card. Hugging Face. https://huggingface.co/EleutherAI/gpt-neo-125m
2. EleutherAI. Pythia-160M Model Card. Hugging Face. https://huggingface.co/EleutherAI/pythia-160m
3. Qwen Team. Qwen3-1.7B-Base Model Card. Hugging Face. https://huggingface.co/Qwen/Qwen3-1.7B-Base
4. Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2005.14165
5. Biderman, S., et al. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. Proceedings of ICML. https://arxiv.org/abs/2304.01373