Redian新闻
>
Firefly单卡复刻Vicuna-13B,Open LLM榜单🤗略高0.2分

Firefly单卡复刻Vicuna-13B,Open LLM榜单🤗略高0.2分

公众号新闻

来自:YeungNLP

进NLP群—>加入NLP交流群

01

前言

在过去的几个月里,经过不断的迭代更新,Firefly项目已支持对LLaMA-2、Ziya、Baichuan、InternLM、LLaMA-1、Bloom等模型进行指令微调,并且开源了多个经过中文指令微调的模型权重,获得了很多同学的支持。目前本项目在Github上获得了1100+star⭐️,感谢大家的关注和支持

此前,我们花了较多精力在优化训练流程之上,而随着项目的不断迭代,我们越发感受到模型评测的重要性。此前我们更多是以看case的形式,主观判断模型的效果,不严谨其说服力不足,无法衡量Firefly的能力。


我们也在逐步完善项目的模型评测工作。此次,我们开源了firefly-llama-13b,并且在🤗Hugging Face的Open LLM榜单上进行了客观的评测。


在榜单上,firefly-llama-13b取得了不错的效果,比vicuna-13b-1.1略高0.2分比llams-2-13b-chat略低0.5分比vicuna-13b-v1.3略低0.6分。从评测分数来看,firefly-llama-13b与vicuna-13b、llama-2-13b-chat的水平非常接近😎。


注:表中各列依次为Average、ARC、HellaSwag、MMLU、TruthfulQA (MC)。


Firefly项目地址:

https://github.com/yangjianxin1/Firefly


firefly-llama-13b权重:

https://huggingface.co/YeungNLP/firefly-llama-13b


🤗 Open LLM排行榜:

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard


02

模型指标

🤗 Open LLM排行榜是Hugging Face设立的一个用于评测开放大语言模型的公开榜单。该榜单一共包括以下四个评测基准:
  • AI2 Reasoning Challenge:一套小学科学问题。
  • HellaSwag:一个测试常识推理的测试,对人类来说很容易(~95%),但对语言模型来说却具有挑战性。
  • MMLU:一个测试文本模型的多任务准确性的测试。测试覆盖57个任务,包括基础数学、美国历史、计算机科学、法律等。
  • Truthful QA MC:一个测试语言模型在生成问题答案时是否真实的基准。

firefly-llama-13b在Open LLM榜单上的评测结果如下:

  1. firefly-llama-13b以59.4分的成绩,超越vicuna-13b-1.1的59.2分和guanaco-13b的59.1分

  2. 仅比最近大火的Llama-2-13b-chat-hf的59.9分略低0.5分;比vicuna-13b-v1.3的60分略低0.6分。

  3. 大幅领先vicuna-7b-v1.3和mpt-7b-chat等7B参数量的模型。

  4. firefly-llama-13b在ARC评测集上比vicuna-13b-v1.3高4.4分,在TruthfulQA评测集上比Llama-2-13b-chat-hf高5.5分美中不足的是,firefly-llama-13b在MMLU和TruthfulQA评测集上的表现和vicuna-13b还有一定的差距。


该评测结果验证了Firefly的QLoRA训练流程的有效性。对于TruthfulQA与MMLU评测效果的不足之处,我们也正在进行一项数据实验。


值得注意的是,vicuna-13b模型采用的是全量参数微调,对训练资源的要求十分高。而firefly-llama-13b采用的则是QLoRA微调,最少仅需16G显存,即可对13B的模型进行微调。


我们使用Firefly的QLoRA训练pipeline对llama-13b进行指令微调,当长度为1024,batch size为1时,显存占用不到16G,一张24G的显卡便可满足训练条件,极大地降低了百亿大模型的训练门槛。


02

训练策略

我们采用QLoRA策略来训练firefly-llama-13b。对于QLoRA的原理和训练流程尚不熟悉的同学,可参考我们往期的文章介绍:

  1. 微调百川Baichuan-13B保姆式教程,手把手教你训练百亿大模型

  2. Firefly-13B开源,QLoRA+百万数据,单卡可训百亿大模型

  3. Firefly|百川baichuan-7B实测,QLoRA+百万指令数据微调

  4. Firefly | QLoRA+百万数据,多卡高效微调bloom-7b1模型

  5. 【QLoRA实战】使用单卡高效微调bloom-7b1,效果惊艳


我们采用huggyllama/llama-13b作为基座模型,使用UltraChat数据集,这是由清华大学开源的英文多轮对话数据,包含140万条多轮对话数据。我们仅训练一个epoch。


我们将训练数据整理成jsonl格式,每条数据格式如下,其中conversation字段是必需的。

{    "conversation_id":"4",    "dataset":"UltraChat",    "conversation":[        {            "human":"What is the most popular smartphone brand these days?",            "assistant":"As an AI language model, I don't have updated information on the current market scenario. However, according to recent statistics, Samsung, Apple, and Huawei are among the most popular smartphone brands worldwide."        },        {            "human":"Interesting! Among these three brands, which one do you think has the most features and benefits for its price?",            "assistant":"As an AI language model, I cannot have personal opinions or bias towards any brand. However, all three brands - Samsung, Apple, and Huawei - offer different features and benefits at different price points. It ultimately comes down to your personal preferences and requirements, like camera quality, design, software, operating system, and other factors, to choose the right brand and model that fits your needs and budget."        },        {            "human":"Can you tell me more about the camera quality of these three brands? Which one has the best camera performance?",            "assistant":"Sure, I can provide you with some information about the camera quality of these three smartphone brands.\n\n1. Apple: iPhones are well known for their camera performance. The latest iPhone models come with a dual-camera or triple-camera setup that is capable of capturing stunning pictures and videos. Apple also introduced features like Night mode, Deep Fusion, and LiDAR Scanner that enhance the camera quality even further.\n\n2. Samsung: Samsung's flagship smartphone models like Galaxy S and Galaxy Note series are known for their camera capabilities. The latest models of Samsung smartphones come with a high-resolution camera that produces excellent pictures with vibrant colors and sharp details. Samsung also offers features like a 108-megapixel camera, Space Zoom, and Single Take mode that enhance the camera quality.\n\n3. Huawei: Huawei smartphones are known for their camera performance and innovation. The latest models like P40 Pro and P40 Pro+ come with a high-resolution camera, and Huawei's proprietary camera technology produces excellent picture quality. Huawei phones also offer features like SuperZoom, AI Cinema, and other editing tools to enhance the camera experience.\n\nAll three brands offer excellent camera quality, but the best camera among these three brands would depend on individual preferences and usage. It's recommended to compare the camera specifications, sample pictures, and features of each brand's models to choose the one that fits your requirements."        },        {            "human":"That's helpful. I'm particularly interested in low-light photography. Which of these three brands would you suggest for taking good quality pictures in low light?",            "assistant":"All three brands - Apple, Samsung, and Huawei - offer excellent low-light photography options, but their approach to low-light photography is different, and each brand has its own strengths and weaknesses when it comes to capturing photos in low light. Here's a brief overview of each brand's approach to low-light photography.\n\n1. Apple: Apple's Night mode feature, which is available on recent iPhone models, is designed to take bright and detailed photos in low light. Night mode uses a combination of multi-frame image processing and advanced machine learning to capture crisp and clear images in both indoor and outdoor low-light environments.\n\n2. Samsung: Samsung's low-light photography capabilities are mainly powered by its camera's sensor and lens technology. The newer Samsung Galaxy S and Galaxy Note models come with a larger sensor and wider aperture, which allows more light to enter the device, resulting in brighter and more detailed photos. These models also use Multi-Frame Noise Reduction and Scene Optimizer to improve the quality of photos taken in low light.\n\n3. Huawei: Huawei is known for its exceptional low-light photography capabilities. Huawei's proprietary camera technology, which uses a combination of large sensors, advanced algorithms, and AI, produces bright and detailed low-light photos. The Huawei P40 Pro and P40 Pro+ models come with a SuperSensing 50-megapixel main lens, which enables capturing clear images in low-light conditions.\n\nAll three brands offer great options, but if you are looking for capturing low light photos primarily, then Huawei's P40 Pro, P40 Pro+ or Mate 40 Pro might be a good choice for you due to the brand's focus on low-light photography features."        },        {            "human":"I'm also curious about the battery life of these smartphones. Which brand offers the longest battery life on average?",            "assistant":"Battery life is an important factor for anyone looking to buy a smartphone, and all three brands - Apple, Samsung, and Huawei - offer different battery life options. 1. Apple: Battery life on iPhones can last anywhere from 4 to 10 hours, depending on the model you choose. Apple's latest iPhone models come with improved battery life with all-day battery performance. However, if you tend to use your smartphone frequently or for more power-intensive tasks, such as gaming or video playback, you may need to charge your iPhone more frequently. 2. Samsung: Samsung's Galaxy series has excellent battery life, and the newer models like the Galaxy S21 and Galaxy Note 20 are designed to last all day long. The battery life on Samsung phones varies from one model to another, but in general, they offer better battery life compared to iPhones. 3. Huawei: Huawei phones are often praised for their long battery life. They come with large-capacity batteries that can last up to two days, depending on usage. The latest Huawei models like the P40 Pro and Mate 40 Pro, also have fast charging features, which means you can quickly refuel your smartphone's battery when it's running low. In conclusion, all three brands offer decent battery life, but Huawei typically offers the longest battery life on average, followed by Samsung."        },        {            "human":"That's good to know. I'm also wondering about the durability of these smartphones. Which brand offers the most durable phones? I tend to drop my phone frequently, so I want something that can handle a few falls.",            "assistant":"All three Smartphone brands - Apple, Samsung, and Huawei - offer durable smartphones, but it's hard to say which one is the most durable since different models have different levels of durability. That being said, here is some information on their durability.\n\n1. Apple: Apple is known for offering durable and sturdy smartphones that can withstand some rough handling. Recent iPhone models come with reinforced glass and are water and dust-resistant. However, with its sleek designs, the iPhone models may be more prone to shattering and cracking if dropped.\n\n2. Samsung: Samsung smartphones like the Galaxy S and Galaxy Note series are generally sturdy and are made with scratch-resistant glass. The newer models are water and dust resistant and come with reinforced corners to protect against drops and bumps.\n\n3. Huawei: Huawei also offers durable smartphones that can handle some falls and rough handling. The latest Huawei models usually come with a sturdy build, reinforced corners, and scratch-resistant glass. The Huawei Mate 40 Pro even has a ceramic back, making it extra durable.\n\nIn conclusion, all three brands - Apple, Samsung, and Huawei - offer durable smartphones that can handle some drops, but the level of durability varies from model to model. If frequent drops are a concern, it's recommended to choose a phone with a rugged case or invest in a protective case to ensure better protection."        }
]}

训练时,我们将多轮对话拼接成如下格式,然后进行tokenize。

<s>input1</s>target1</s>input2</s>target2</s>...

在计算loss时,我们通过mask的方式,input部分的loss不参与参数更新,只有“target</s>”部分的loss参与参数更新。这种方式充分利用了模型并行计算的优势,训练更加高效,且多轮对话中的每个target部分都参与了训练,训练更充分。否则,就需要把一个n轮对话,拆分成n条数据,且只计算最后一个target的loss,大大降低了训练效率。


loss计算的实现方式可参考以下代码:

https://github.com/yangjianxin1/Firefly/blob/master/component/loss.py#L3


对于QLoRA,除了embedding和lm_head外,我们在所有全连接层都插入adapter,其中lora_rank为64,lora_alpha为16,lora_dropout为0.05。最终参与训练的参数量约为2.5亿


训练超参数如下所示:

max length1024
lr_scheduler_typeconstant_with_warmup
batch size64
lr1e-4
warmup step3000
optimizerpaged_adamw_32bit
training step23k


模型的训练损失的变化趋势如下图所示。


03

效果对比

我们使用Vicuna官网中的部分输入进行测试,以对比vicuna-13b与firefly-llama-13b的生成结果。


注:vicuna-13b的结果为其官网展示样例。


示例1:【撰写一篇引人入胜的旅行博客文章,内容涉及最近去夏威夷旅行,强调文化经历和必看的景点。】


示例2:【如果14世纪没有发生黑死病会怎样?】


示例3:【如果艾萨克·牛顿专注于生物学而不是物理学会怎样?】


示例4:【使用一次性塑料瓶而不是可重复使用的瓶子,对环境和人类健康有什么潜在影响?】


我们将在共享文档中补充更多的对比结果:


从一些生成例子来看,firefly-llama-13b与vicuna-13b的性能比较接近。


04

结语

此次我们开源了firefly-llama-13b,在Open LLM排行榜上的评测结果与vicuna-13b、Llama-2-13b-chat非常接近,验证了Firefly的QLoRA训练流程的有效性。


QLoRA工作一经发布,Firefly项目便开始研究和探索QLoRA训练方案。随着不断迭代更新,项目支持对越来越多的模型进行微调,也开源了一些中文模型,但始终缺乏评测环节。此次的评测工作,弥补了项目在这方面的缺失,验证了训练流程的有效性。




进NLP群—>加入NLP交流群

微信扫码关注该文公众号作者

戳这里提交新闻线索和高质量文章给我们。
相关阅读
新加坡科技设计大学提出FLACUNA:提高LLMs问题解决能力!Open LLM榜单再次刷新,比Llama 2更强的「鸭嘴兽」来了Biden will ask Congress for $13B to support Ukraine推理飙升2倍!普林斯顿北大校友祭出多头「美杜莎」,33B模型与13B一样快2023 US Open 美国网球公开赛性能超越Llama2-13B,可免费商用,姚星创业公司开源百亿参数通用大模型以 LLM 为核心 LLM@Core:程序员的大语言模型技术指南How Korea Quietly Reshaped Chinese Pop Culture半中介费|9.1入住|Allston近BU步行12分钟/88超市2分钟优质 一室一厅2600,包水暖剧院新闻|Penelope Wilton和撸哥Luke Evans将出演新剧Backstage Billy咀外文嚼汉字(242)“二刀流”,“双枪将”Adobe推AI生图下猛药:侵权包赔!企业版Firefly已上线无中介费|9.1入住|Allston近BU步行12分钟/88超市2分钟优质 一室一厅2800,包水暖有看US Open 喜欢Ben Shelton的吗?(简直是龙生龙凤生凤老师儿子会打洞的典型例子)GPT-4最强平替更新!UC伯克利发布Vicuna v1.5,支持4K和16K上下文,刷新SOTA,LeCun转赞Open AI创始人的30岁人生建议国产130亿参数大模型免费商用!性能超Llama2-13B,支持8k上下文,哈工大已用上【Allston/BU/房东直降,人均$1500】【2B2B+3B3B】Adobe Firefly不用止步于「远观」,现在人人都能上手玩了中共,生的残忍,死的猥琐明尼周末不无聊|这周一块去Firefly Viewing Nights看萤火虫吧!娘親舅大 (十)同場競技Rescue Teams Struggle to Reach Isolated Areas in Flood-Hit Hebei9.1入住|Allston近BU步行12分钟/88超市2分钟优质STUDIO 2400, 一室一厅2500,包水暖无中介费|9.1入住|Allston近BU步行12分钟/88超市2分钟优质 一室一厅2600,包水暖OpenAI 又赢麻了!谷歌 DeepMind 创始人刚称 Gemini 能碾压 GPT-4,OpenAI 的挖人大计就出炉了9.1入住|Allston近BU步行12分钟/88超市2分钟优质STUDIO 2400, 一室一厅2450,包水暖中国市场成为高阶智驾战略高地,博世/安波福包揽四项大奖字节团队提出猞猁Lynx模型:多模态LLMs理解认知生成类榜单SoTAGPT-4、ChatGLM2、Llama2、PaLM2在KDD LLM Day上一起开了个会GPT-4,Llama2,ChatGLM2,PaLM2共聚一堂 | LLM Day @KDD 2023一对夫妻外加“孩子妈”的家庭AmEx Refer Jail (AmEx refer 功能突然消失)现象及解决方案Hélène Binet:光的哲学家LLM 全景图 (The Landscape of LLM)我眼中的“爱”9.1入住|Allston近BU步行12分钟/88超市2分钟优质一室一厅2450,包水暖LLM-Pruner: 剪枝+少量数据+少量训练 = 高效的LLM压缩China Floods: What’s the Cause, Why’s Rescue Been So Difficult?open concept的房子怎么解决噪音/油烟问题
logo
联系我们隐私协议©2024 redian.news
Redian新闻
Redian.news刊载任何文章,不代表同意其说法或描述,仅为提供更多信息,也不构成任何建议。文章信息的合法性及真实性由其作者负责,与Redian.news及其运营公司无关。欢迎投稿,如发现稿件侵权,或作者不愿在本网发表文章,请版权拥有者通知本网处理。