langchain

mirror of https://github.com/kennethreitz/langchain.git synced 2026-06-05 23:00:18 +00:00

Files

T

Taqi Jaffri b7290f01d8 Batching for hf_pipeline (#10795 )

The huggingface pipeline in langchain (used for locally hosted models)
does not support batching. If you send in a batch of prompts, it just
processes them serially using the base implementation of _generate:
https://github.com/docugami/langchain/blob/master/libs/langchain/langchain/llms/base.py#L1004C2-L1004C29

This PR adds support for batching in this pipeline, so that GPUs can be
fully saturated. I updated the accompanying notebook to show GPU batch
inference.

---------

Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>

2023-09-25 18:23:11 +01:00

_scripts

llm feat table revision (#10947 )

2023-09-22 10:29:12 -07:00

api_reference

add model feat table (#10921 )

2023-09-22 01:10:27 -07:00

docs_skeleton

redirect mrkl (#10979 )

2023-09-23 16:09:13 -07:00

extras

Batching for hf_pipeline (#10795 )