[reknih]

This is not really a meaningful comparison: You have your TeX compiler (pdfTeX, XeTeX, LuaTeX, and the older e-TeX) that takes a document in some format and produces a PDF or DVI. In my tests (that did not include e-TeX) pdfTeX tends to be the fastest here, but sometimes you need modern fonts, so you have no choice but to use the other two.

The TeX compiler then loads a format like plain TeX (which the above commenter uses), LaTeX, or ConTeXt. The format defines what macros are available. LaTeX adds a package system, as does ConTeXt (modules) so you can import even more macros on-demand. These TeX formats differ in scope and thus speed, LaTeX tends to be a bit heavier but what really weighs it down are the myriad of packages it is usually used with.

Many TeX distributions will define aliases like pdflatex in your path such that you can preload pdfTeX with the LaTeX format, but they are not really separate compilers.

[nerdponx]

But XeTeX and LuaTeX are still billed as "end-user-facing" systems, alternatives to LaTeX and ConTeXt. I appreciate the distinction between the actual typesetting engine and a macro library layered on top, but as someone who has only ever used LaTeX I would definitely be interested in a thorough comparison of their capabilities, speed, user experience, etc. One can of course include plain TeX itself in such a comparison!