谷歌没开源的PaLM，网友给开源了！千亿参数微缩版：最大只有10亿，8k上下文

---

  新智元报道  

My dog is very cute, but not very good at socializing with other dogs. The dog loves all new people and he likes to hang out with other dogs. I do need to take him to the park with other dogs. He does have some bad puppy breath, but it is only when he runs off in a direction he doesn't want to go. currently my dog is being very naughty. He would like to say hi in the park, but would rather take great care of himself for a while. He also has bad breath. I am going to have to get him some oral braces. It's been 3 months. The dog has some biting pains around his mouth. The dog is very timid and scared. The dog gets aggressive towards people. The dog is very playful and they are a little spoiled. I am not sure if it's a dog thing or if he is spoiled. He loves his toys and just wants to play. He plays with his toys all the time and even goes on walks. He is a little picky, not very good with other dogs. The dog is just a little puppy that goes to the park. He is a super friendly dog. He has not had a bad mouth or bad breath

我的狗很可爱，但是不善于和其他狗交际。这只狗喜欢所有新来的人，他喜欢和其他的狗一起玩。我确实需要带他和其他狗一起去公园。他确实有点小狗的口臭，但只有当他往不想去的方向跑的时候。现在我的狗很淘气。他想在公园里打个招呼，但宁愿好好照顾自己一段时间。他还有口臭。我得给他买个口腔矫正器，已经过去三个月了。这条狗嘴边有些咬痕疼痛。这只狗非常胆小和害怕。这只狗对人有攻击性。这条狗非常顽皮，他们有点被宠坏了。我不确定是狗的问题还是他被宠坏了。他喜欢他的玩具，只是想玩。他总是玩他的玩具，甚至去散步。他有点挑剔，不太会和其他狗相处。那只狗只是一只去公园的小狗。它是一只超级友好的狗。他没有口臭问题了。

安装

git clone https://github.com/conceptofmind/PaLM.gitcd PaLM/pip3 install -r requirements.txt

使用

model = torch.hub.load("conceptofmind/PaLM", "palm_410m_8k_v0").cuda()

另外，你还可以通下面的方式直接加载PyTorch模型检查点：

from palm_rlhf_pytorch import PaLMmodel = PaLM(    num_tokens=50304, dim=1024, depth=24, dim_head=128, heads=8, flash_attn=True, qk_rmsnorm = False,).cuda()
model.load('/palm_410m_8k_v0.pt')

python3 inference.py "My dog is very cute" --seq_len 256 --temperature 0.8 --filter_thres 0.9 --model "palm_410m_8k_v0"

为了提高性能，推理使用torch.compile()、  Flash Attention和Hidet。

训练

数据

python3 build_dataset.py --seed 42 --seq_len 8192 --hf_account "your_hf_account" --tokenizer "EleutherAI/gpt-neox-20b" --dataset_name "EleutherAI/the_pile_deduplicated"