Employ bigger models
The current offering of models is entirely composed of small models, which can work well for some tasks and is certainly efficient, but also makes the service unusable for more complicated tasks. For example, none of the models employ reasoning techniques.
Elias Heroja
shared this idea
-
Cedric
commented
https://protonmail.uservoice.com/forums/932842-lumo/suggestions/50684972-add-kimi-k2-thinking another suggested model
-
Cedric
commented
-
Cedric
commented
https://protonmail.uservoice.com/forums/932842-lumo/suggestions/50472912-latest-highly-efficient-open-source-llms-suggestio suggests alternative models that could be used.
-
Cedric
commented
To give a bit of context.
GPT-3: 175 billion total parameters (publicly confirmed in https://github.com/openai/gpt-3)
GPT-4: is estimated at 1000-2000 billion total parameters (not publicly confirmed, see https://explodingtopics.com/blog/gpt-parameters) but this info is considered not relevant anymore due to using a Mixture-of-Experts (MoE) architecture. Only a subset of parameters are active per token, ie a few very well chosen number of parameters that are "experts" for the given token. GPT-4 is estimated to ~200 billions active parameters per token.
The advantage of the MoE approach is to train the model on huge numbers of parameters so it performs better (qualitatively). And restrict its active parameters to a smaller subset so it performs faster (quantitatively).
GPT-5: is estimated at 1000-10000 billion total parameters (not publicly confirmed, see https://www.cometapi.com/how-many-parameters-does-gpt-5-have/), also MoE. It is estimated at 200-600 active parameters per token.
Even if the above are approximations, the models used by Proton are way smaller, and are not MoE. However, MoE models are standard nowadays, and they will surely become outdated with better models soon...
According to Proton (https://proton.me/support/lumo-privacy#open-source:~:text=Open%2Dsource%20language%20models,-Lumo), the models Lumo currently uses are: Nemo, OpenHands 32B, OLMO 2 32B and Mistral Small 3.
OpenHands 32B: 32 billion parameters (https://huggingface.co/OpenHands/openhands-lm-32b-v0.1)
OLMO 2 32B: 32 billion parameters (https://learnprompting.org/blog/ai2-released-olmo2-32b)
Mistral Small 3: smaller model, 24 billion parameters (https://mistral.ai/news/mistral-small-3)
Nemo (Mistral): even smaller model, 12 billion parameters (https://mistral.ai/news/mistral-nemo)