Basic Hardware Considerations for Open-Source Language Models: Memory, Performance, and Feasibility

Open-source language models are especially relevant in the context of privacy and digital sovereignty, as they allow execution in isolated environments with sensitive or confidential data. A major barrier to their adoption in smaller companies is the hardware requirements and, consequently, the training of human resources. In this article, we present some concepts that can help approximately specify which hardware can meet this need, answering two questions:

1. How much memory is needed to load the model?
2. Which performance parameter should be observed to ensure that the chosen hardware provides fluid responses, similar to a traditional chat?

To answer the first question, it is necessary to understand that the complexity of LLM models can, in general terms, be inferred from the number of parameters. This information usually appears in the model's own name. For example: Llama-3B indicates that this is the Llama model (Meta) with 3 billion parameters. If this information is not explicit, the model's documentation should be consulted. To calculate the minimum amount of memory required, consider that each parameter is stored in a variable with 32-bit floating-point precision. Since each byte equals 8 bits, a 32-bit variable corresponds to 4 bytes in memory. Thus:

3,000,000,000 parameters x 4 bytes/parameter = 12×10^9 bytes = 12 Gigabytes

In other words, to load a 3B parameter model, at least 12 GB of free video memory (VRAM) is required. It is worth noting that even in 2025, discrete graphics cards with 8 GB are still common in desktop computers, as are 4 GB integrated GPUs in laptops. One strategy to optimize resources is the quantization of the model's variables, which reduces the “resolution” of the parameters while maintaining their functional characteristics. An example of a library that implements this process is bitsandbytes. Recent approaches also explore the use of SSDs as a partial alternative to VRAM.

The detailed hardware requirements vary according to the goal: training, fine-tuning, or inference only. Training requires storage and processing of enormous volumes of data. To illustrate, according to the technical report on DeepSeek-V3, 2.78 billion GPU hours (NVIDIA H800) were required, using 14.8 trillion tokens of high quality and diversity, considering that each token equals approximately 4 bytes [reference], it can be concluded that approximately 59 TB of training data was used. It is evident that this level of infrastructure is beyond the reach of small companies.

The fine-tuning of an LLM seeks to introduce new behaviors or improve specific instructions, adjusting only part of the parameters based on examples of prompts and expected responses. Its feasibility depends on the number of adjustable parameters, the method adopted, the size of the dataset, and the available hardware.

More efficient fine-tuning methods have recently been developed, among which LoRA (Low Rank Adaptation) stands out (Hu et al., 2022). The method is based on an adapter layer, whose adjustable parameters are obtained through a matrix decomposition. The new parameter layer can be attached to the attention layers (attention heads) or to the fully connected layer (synonymous with dense or linear layer in the context of neural networks) [source]. Updating the adapter matrices requires much less computational resources during the readjustment procedure due to the lower rank (rank) of these matrices compared to the original weight matrix, which has larger dimensions and rank (rank). The actual parameter readjustment is performed by solving an optimization problem with a modern gradient descent method, such as Adam (Kingma & Ba, 2014), for example.

QLoRA is a more efficient version of the method, proposed by Dettmers et al., 2023, which achieves better computational performance through quantization of the original model's parameters. Both methods were implemented in the PEFT library available for Python, and more details and examples can be found [here]. The use of these techniques allows models to be customized in a viable way, balancing efficiency and performance, without the need to re-estimate all parameters.

Companies that integrate LLMs into complete products or inference services are called by some Application-Layer AI Companies [Source1]. Their role is to transform general-purpose models into solutions applied to specific real-world problems, bringing direct value to users. Examples include lucidworks, Perssua, getenter.ai, requesty.ai, braine.digital, dynadok.com, etc. In this scenario, the main computational demand falls on inference, which, although lighter than training or fine-tuning, can become intensive in large-scale environments.

For inference, the key parameter is tokens per second, which measures the model's ability to generate or process tokens continuously. This indicator determines the latency and fluidity of the interaction, being crucial to providing the user with a sense of real-time dialogue. Performance depends both on the model's characteristics (architecture and number of parameters) and on the infrastructure hosting it (type and number of GPUs, TPUs, etc.). Some examples of LLM/hardware performance tests can be found at Link1, or Link2.

In summary, the adoption of open-source LLMs offers strategic opportunities for companies seeking greater autonomy and security over their data. However, the success of this implementation depends on a careful analysis of hardware requirements, alignment between optimization methods such as quantization and LoRA, and the final goal: whether training, fine-tuning, or inference only. Understanding these variables is essential to turning the potential of language models into practical results, balancing costs and benefits.

References

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36, 10088-10115.

[link]

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., … & Chen, W. (2022). Lora: Low-rank adaptation of large language models. ICLR, 1(2), p. 3. [link]

Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [link]