Nice idea. I’m not really sure if this is fast enough, though. The article doesn’t contain a lot of information about things like memory bandwith. But the 3.5 T/s for a 7b parameter model is about the same ballpark my 9 year old Skylake CPU does with DDR3(?) RAM (at double the energy consumption). It is a bit slower than I can read, plus there is a good amount of waiting for it to ingest the question / prompt.