I've tried working out how to move to an event sourcing system, but I always struggle with locking behavior. Do you just have to invent your own locking mechanisms on top of event sourcing?
The stream is the consistency (locking) boundary. Your first step it to get your model aligned with such boundaries. For example, your amazon shopping basket is independent of my basket. Then you chose your concurrency model - append to a stream with an expected version (pessimistic) or just append anyway (no expected version). Your amazon basket may be the later, your amazon payment and shipping checkout may be the former.
Locking across streams is an anti-pattern / smell. It can be done (as can anything) but it usually points to a modelling problem. Example: cancelling an amazon order is a _request_ that is in a race with the fulfillment system (boundary); it may or may not be successful.
Read about how LMAX achieved 6 million transactions per second using a ring buffer-based concurrency architecture called disruptor, all on a single thread and without locks. Event sourcing plays a big role in their architecture [0].
Combine this with the actor model (using Akka or similar) gives you guaranteed "one message at a time" processing and you don't have to deal with locks.
I question any amount of guarantees around "one message" anything. There might be this guarantee per actor, but you have no such guarantee per system. And, assuming a real system, this will be a problem.
So, you get to pick, "at most once" or "at least once." And then you need to build your system to act accordingly.
I'm not sure what you are suggesting. There is no getting around "at most/at least once." You can shift the posts, some, but at some point that is your choice. This is a good read on the problem: http://bravenewgeek.com/you-cannot-have-exactly-once-deliver...
"Exactly once delivery" is not the same as "One message at a time". Akka Actors process one message at a time. Akka does no provide exactly once delivery. It default defaults to "at most once".
Correct. Message delivery strategy is separate from message processing once delivered. If you choose at-least-once delivery, you'll need to handle possible duplicates regardless of whether or not you can process a duplicate message in a lock-free manner.
ISTM event sourcing actually avoids many locking problems, since it's essentially "write-only". Of course every event write should be atomic, but that seems easier than making updates atomic?
When a certain set of events occurs (the files arrive etc) I want to kick off one and only one batch processor task. This is accomplished with a transaction and a write lock in an sql database, but when trying to use event sourcing it ends up requiring a 2 step "intent to run" event before running or some out of band synchronization.
This isn't something I would handle with event sourcing. Using ES throughout an application is an antipattern.
For something like this my batch processor is implemented as you're probably used to—files get a CRUD model associated with them, schedule a background job to handle it, let locking get handled there. Once inside the batch processor you can use the same domain services and commands that you'd use from your application layer and commit events on a command basis, or on something like a row in the file (which may generate several commands and dozens of events depending on your model), or on a file level (all or nothing.)
The thing I see people do frequently (and sadly, have done myself on occasion!) that makes their lives harder is trying to shoehorn everything into ES without doing the design work to establish a domain, its boundaries, and what events make sense within it.
I'm not sure you can achieve good event sourcing performance using a regular database engine. Better to view it like writing logs.
If you really must expose an SQL API, perhaps you could read the journal on another thread or process and then make changes to the db based on the incoming "diffs" that the threads determines from the journal?
Locking across streams is an anti-pattern / smell. It can be done (as can anything) but it usually points to a modelling problem. Example: cancelling an amazon order is a _request_ that is in a race with the fulfillment system (boundary); it may or may not be successful.