Refactor async engine & turbomind IO

[lzhangzz]

TODO

• output logits
• output logprobs
• output hidden states

[zhyncs]

It’s amazing!

[lvhan028]

FINALLY!!

[lvhan028]

why do_sample True?

[lvhan028]

I am not sure if it works for pytorch engine

[lvhan028]

stop_words should be trimmed

[lvhan028]

when backend is pt engine, the pipeline cannot be destroyed successfully.

from lmdeploy import pipeline, PytorchEngineConfig
model_path = ‘internlm2_5-7b-chat’
engine_config = PytorchEngineConfig()
pipe = pipeline(model_path, backend_config=engine_config, log_level='INFO')
response = pipe('hi')

[lvhan028]

I will fix get_ppl and saving csv in profile_throughput in another PR

[zhulinJulia24]

@lzhangzz output tokens doubles when ignore_eos is true

from lmdeploy.messages import (GenerationConfig, PytorchEngineConfig,
                               TurbomindEngineConfig)
from lmdeploy import pipeline

gen_config = GenerationConfig(ignore_eos=True, max_new_tokens=10)
response = pipe(['Hi, pls intro yourself', 'Shanghai is'], gen_config=gen_config)
print(response)

I suppose the generate_token_len shoud be 10. but it's 20 actually, the output is:

[Response(text='你好！我是书生·浦语，由你好！我是书生·浦语，由', generate_token_len=20, input_token_len=108, finish_reason='length', token_ids=[77230, 60477, 68734, 60628, 60384, 60721, 62442, 60752, 60353, 60620, 77230, 60477, 68734, 60628, 60384, 60721, 62442, 60752, 60353, 60620], logprobs=None, logits=None, last_hidden_state=None, index=0), Response(text='上海，作为中国最大的城市之一，不仅是中国上海，作为中国最大的城市之一，不仅是中国', generate_token_len=20, input_token_len=105, finish_reason='length', token_ids=[68589, 60353, 68429, 68277, 69410, 68494, 68538, 60353, 68710, 70543, 68589, 60353, 68429, 68277, 69410, 68494, 68538, 60353, 68710, 70543], logprobs=None, logits=None, last_hidden_state=None, index=1)]