The oldest source [1] seems to be in Sanskrit, a left-to-right language, so they were big-endian numbers. This still does not rule out an even older origin in a right-to-left language, but as it stands I'm afraid I was just wrong with my 'little endian origin' story. Too bad, I liked it.
[1] https://mathshistory.st-andrews.ac.uk/HistTopics/Bakhshali_m...