- Physical vs logical integrity
- ACID (strong) consistency
- BASE (weak) consistency
- Synchronous vs asynchronous replication
- Distributed systems limitations - CAP theorem
- Examples of data consistency violation (like Photo Privacy Violation or Double Money Withdrawal described in "Don't settle for eventual consistency" article)
- Strong consistency comes with performance penalty. Choosing performance and availability over consistency might be justified and lead to improved revenues (as is the case with Amazon) or lead to spectacular failures like in case of Flexcoin
Local vs distributed transactions
The second part of the presentation was slightly different, though. It included a live demonstration of a situation where local transactions are not sufficient to guarantee data consistency across multiple resources and how distributed transactions come to the rescue. The demonstration was done basing on the scenario below:
The application consumes messages from a JMS queue (TEST.QUEUE), stores message content in a database, does some processing inside VeryUsefulBean and finally sends a message to another JMS queue (OUT.QUEUE).
The application was a web application deployed on JBoss EAP 6.2. JMS broker functionality was provided by ActiveMQ and MySQL acted as a database. Web application logic was built with Spring and Apache Camel.
So let's assume that the processing inside VeryUsefulBean fails:
The exception was injected using JBoss Byteman rule:
As expected with local transactions system state was inconsistent after processing failure. The expected state would be:
Basically one would expect that system state would not change due to processing failure. However the actual behaviour was:
The reason for that behaviour was that local transaction that saved data into database was committed independently from JMS message consumption transaction, leading to inconsistent state.
Then the experiment was repeated with JTA transaction manager and XA resources set up. The outcome was correct this time - no data was saved to database. JMS message consumption and all processing including database inserts was handled as part of the same distributed transaction and all changes were rolled back upon failure as expected.
First test cases were defined as JBehave scenarios:
JUnit was used to execute the scenarios:
Spring context for the web application was splitted into two parts:
While the core context could and had to be reused during test execution, the infrastructure context made sense only when the application was deployed on a real JEE server, as it retrieved necessary resources from JNDI - see below.
Therefore the infrastructure part of the context had to be rewritten for automated test execution:
Note beans with id amq-broker and hsqldbServer - they are responsible for starting embedded JMS broker and DB server needed during test execution.
Having the infrastructure in place, it is quite simple to write test steps defined in JBehave scenarios, e.g.:
There are obviously a few other details that need to be worked out, but since this post has already grown too long - have a look at the complete code on GitHub: https://github.com/mstrejczek/dataintegrity.
- Incoming message not lost
- No data saved in database
- Nothing sent to outbound queue (OUT.QUEUE).
Basically one would expect that system state would not change due to processing failure. However the actual behaviour was:
- Incoming message not lost
- Data saved to DB (as many times as message was (re)delivered).
- Nothing sent to outbound queue (OUT.QUEUE).
Then the experiment was repeated with JTA transaction manager and XA resources set up. The outcome was correct this time - no data was saved to database. JMS message consumption and all processing including database inserts was handled as part of the same distributed transaction and all changes were rolled back upon failure as expected.
Automated integration test
The test proved that application worked correctly with JTA transaction manager and XA resources (XA connection factory for JMS, XA data source for JDBC), however the test was manual and time consuming. Ideally this behaviour would be tested automatically, and this was the topic of the final part of the presentation. We did a walk through an integration test that verified transactional behaviour automatically.First test cases were defined as JBehave scenarios:
Spring context for the web application was splitted into two parts:
- spring-jmsconsumer-core.xml - contained beans with application logic and definition of Camel context.
- spring-jmsconsumer-infrastructure.xml - contained beans used to access external resources, like JMS connection factory or JDBC data source.
- ActiveMQ - replaced by embedded ActiveMQ.
- Arjuna Transaction Manager provided by JBoss EAP - replaced by standalone Atomikos.
- MySQL - replaced by embedded HSQLDB.
While the core context could and had to be reused during test execution, the infrastructure context made sense only when the application was deployed on a real JEE server, as it retrieved necessary resources from JNDI - see below.
Therefore the infrastructure part of the context had to be rewritten for automated test execution:
Note beans with id amq-broker and hsqldbServer - they are responsible for starting embedded JMS broker and DB server needed during test execution.
Having the infrastructure in place, it is quite simple to write test steps defined in JBehave scenarios, e.g.:
There are obviously a few other details that need to be worked out, but since this post has already grown too long - have a look at the complete code on GitHub: https://github.com/mstrejczek/dataintegrity.
