[PaulHoule]

If the text is in a predictable position on the screen, this should be easy.

An image processing pipeline like this usually has a filter up front that drops out the environment and leaves a monochrome image of just the text and associated graphics. The text is white and is usually on a background that provides a lot of contrast since it designed to be readable by you!

[zemanel]

Yep makes sense. For the OCR part a colleague suggested something like Tesseract or Google Vision but, what about the shapes, like weapon (type) symbol on top right and health/shield bar percentages and etc ?

Maybe i should just get started small with the OCR part

[PaulHoule]

The weapons symbols are just very big characters.

The health and shield bar percentages could be measured by counting pixels. For instance you could draw a horizontal line across the bar and see how many pixels are lit up.